This extends perceptrons \cite{books/daglib/0066902} but uses what is known as the Hinge loss (aka SVM loss):

$$J_i(w) = max(0,\gamma -y\_i f(x\_i))$$

Where $\gamma$ is the margin. $J_i(w)$ is the error given some weight $w$ parameters. $x_i$ and $y_i$ are a training example and correct label. $f(x_i)$ is the perceptron function we are trying learn the best weights for.

Problem
--------------
Refine synthetically simulated images to look real

https://machinelearning.apple.com/images/journals/gan/real_synt_refined_gaze.png

Approach
--------------
* Generative adversarial networks

Contributions
----------
1. **Refiner** FCN that improves simulated image to realistically looking image
2. **Adversarial + Self regularization loss**
* **Adversarial loss** term = CNN that Classifies whether the image is refined or real
* **Self regularization** term = L1 distance of refiner produced image from simulated image. The distance can be either in pixel space or in feature space (to preserve gaze direction for example).


https://i.imgur.com/I4KxCzT.png

Datasets
------------
* grayscale eye images
* depth sensor hand images 


Technical Contributions
-------------------------------
1. **Local adversarial loss** - The discriminator is applied on image patches thus creating multiple "realness" metrices
https://machinelearning.apple.com/images/journals/gan/local-d.png

2. **Discriminator with history** - to avoid the refiner from going back to previously used refined images.

https://machinelearning.apple.com/images/journals/gan/history.gif

This paper apply temporal convolutional neural network on character input to learn abstract text concepts. Depending on application, the model can output the category of text or review sentiment. The model is trained from character level and do not require knowledge of syntax or semantic structure. Therefore, the model can work for various language including English and Chinese with little prior knowledge of languages.

There are 2 implementations for the paper:

 1. [Reference Implementation of Deep Linear Discriminant Analysis (DeepLDA)](https://github.com/CPJKU/deep_lda).
 2. [VahidooX/DeepLDA](https://github.com/VahidooX/DeepLDA).

[It seems something is wrong with the cost function implemented](https://github.com/VahidooX/DeepLDA/issues/1#issuecomment-392261355).
Also, while they derive the Gradient they didn't verify it and in the implementation use Theano's Auto Grad (While other Auto Grad can't work it out).

The weights at each layer $W$ are initialized based on the number of connections they have. Each $w \in W$  is drawn from a Gaussian distribution with mean $\mu = 0$ with the variance as follows. 

$$\text{Var}(W) = \frac{2}{n_\text{in}+ n_\text{out}}$$

Where $n_\text{in}$ is the number of neurons in the previous layer from the feedforward direction and $n_\text{out}$ is the number of neurons from the previous layer from the backprop direction.
Reference: [Andy Jones's Blog](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization)