GANs for images have made impressive progress in recent years, reaching ever-higher levels of subjective realism. It’s also interesting to think about domains where the GAN architecture is less of a good fit. An example of one such domain is natural language. 

As opposed to images, which are made of continuous pixel values, sentences are fundamentally sequences of discrete values: that is, words. In a GAN, when the discriminator makes its assessment of the realness of the image, the gradient for that assessment can be backpropagated through to the pixel level. The discriminator can say “move that pixel just a bit, and this other pixel just a bit, and then I’ll find the image more realistic”. However, there is no smoothly flowing continuous space of words, and, even if you use continuous embeddings of words, it’s still the case that if you tried to apply a small change to a embedding vector, you almost certainly wouldn’t end up with another word, you’d just be somewhere in the middle of nowhere in word space. In short: the discrete nature of language sequences doesn’t allow for gradient flow to propagate backwards through to the generator. 

The authors of this paper propose a solution: instead of trying to treat their GAN as one big differentiable system, they framed the problem of “generate a sequence that will seem realistic to the discriminator” as a reinforcement learning problem? After all, this property - of your reward just being generated *somewhere* in the environment, not something analytic, not something you can backprop through -  is one of the key constraints of reinforcement learning. Here, the more real the discriminator finds your sequence, the higher the reward. One approach to RL, and the one this paper uses, is that of a policy network, where your parametrized network produces a distribution over actions. You can’t update your model to deterministically increase reward, but you can shift around probability in your policy such that your expected reward of following that policy is higher. 

This key kernel of an idea - GANs for language, but using a policy network framework to get around not having backprop-able loss/reward- gets you most of the way to understanding what these authors did, but it’s still useful to mechanically walk through specifics. 

https://i.imgur.com/CIFuGCG.png

At each step, the “state” is the existing words in the sequence, and the agent’s “action” the choosing of its next word
- The Discriminator can only be applied to completed sequences, since it's difficult to determine whether an incoherent half-sentence is realistic language. So, when the agent is trying to calculate the reward of an action at a state, it uses Monte Carlo Tree Search: randomly “rolling out” many possible futures by randomly sampling from the policy, and then taking the average Discriminator judgment of all those futures resulting from each action as being its expected reward
- The Generator is a LSTM that produces a softmax over words, which can be interpreted as a policy if it’s sampled from randomly 
- One of the nice benefits of this approach is that it can work well for cases where we don't have a hand-crafted quality assessment metric, the way we have BLEU score for translation 

### Read-Again
Two options:
* GRU: run a pass of regular GRU on the input text $x_1,\ldots,x_n$. Use its hidden states $h_1,\ldots,h_n$ to compute weights vector for every step $i$ :
$\alpha_i = \tanh \left( W_e h_i  + U_e h_n + V_e x_i\right)$ and then runs a second GRU pass on the same input text. In the second pass the weights $\alpha_i$, from the first pass, are multiplied with the internal $z_i$ GRU gatting (controlling if hidden state is directly copied) of the second pass.
* LSTM: concatenate the hidden states from the first pass with the input text
$\left[ x_i, h_i, h_n \right]$ and run a second pass on this new input.

In case of multiple sentences the above passes are done per sentence. In addition the $h^s_n$ of each sentence $s$ is concatenated with the $h^{s'}_n$ of the other sentences or with $\tanh \left( \sum_s V_s h_s  + v\right)$

### Decoder with copy mechanism
LSTM with hidden state $s_t$. Input is previously generated word $y_{t-1}$ and context computed with attention mechanism: $c_t = \sum_i^n \beta_{it} h_i$. Here $h_i$ are the hidden states of the 2nd pass of the encoder. The weights are $\beta_{it} = \text{softmax}  \left(  v_a^T \tanh \left( W_a s_{t-1}  + U_a h_i\right) \right)$

The decoder vocabulary $Y$ used is small. If $y_{t-1}$ does not appear in $Y$ but does appear in the input at $x_i$ then its embedding is replaced with $p_t = \tanh \left( W_c h_i + b_c\right)$ and <UNK> otherwise.

$p_t$ is also used to copy the input to the output (details not given)

### Experiments
abstractive summarization [DUC2003 and DUC2004 competitions](http://www-nlpir.nist.gov/projects/duc/data.html).

Imagine you make a neural network mapping a scalar to a scalar. After you initialise this network in the traditional way, randomly with some given variance, you could take the gradient of the input with respect to the output for all reasonable values (between about -3 and 3 because networks typically assume standardised inputs). As the value increases, different rectified linear units in the network will randomly switch on, drawing a random walk in the gradients; another name for which is brown noise.

![](http://i.imgur.com/KMzfzMZ.png)

However, do the same thing for deep networks, and any traditional initialisation you choose, and you'll see the random walk start to look like white noise. One intuition given in the paper is that as different rectifiers in the network switch off and on the input is taking a number of different paths though the network. The number of possible paths grows exponentially with the depth of the network, so as the input varies, the gradients become increasingly chaotic. **The explanations and derivations given in the paper are much better reasoned and thorough, please read those if you are interested**.

Why should we care about this? Because the authors take the recent nonlinearity [CReLU][] (output is concatenation of `relu(x)` and `relu(-x)`) and develop an initialisation that will avoid problems with gradient shattering. The initialisation is just to take your standard initialised weight matrix $\mathbf{W}$ and set the right half to be the negative of the left half ($\mathbf{W}_{\text{left}}$). As long as the input to the layer is also concatenated, the left half will be multiplied by `relu(x)` and the right by `relu(-x)`. Then:

$$
\mathbf{W}.\text{CReLU}(\mathbf{x}) = \begin{cases} \mathbf{W}_{\text{left}}\mathbf{x} & \text{ if } x > 0 \\ \mathbf{W}_{\text{left}}\mathbf{x} & \text{ if } x \leq 0\end{cases}
$$

Doing this allows them to train deep networks without skip connections, and they show results on CIFAR-10 with depths of up to 200 exceeding (slightly) a similar resnet.

[crelu]: https://arxiv.org/abs/1603.05201

A key question in the analysis of hippocampal memory relates to how attention modulates the encoding and long-term retrieval of spatial and nonspatial representations in this region. To address this question, we recorded from single cells over a period of 5 days in the CA1 region of the dorsal hippocampus while mice acquired one of two goal-oriented tasks. These tasks required the animals to find a hidden food reward by attending to either the visuospatial environment or a particular odor presented in shifting spatial locations. Attention to the visuospatial environment increased the stability of visuospatial representations and phase locking to gamma oscillations—a form of neuronal synchronization thought to underlie the attentional mechanism necessary for processing task-relevant information. Attention to a spatially shifting olfactory cue compromised the stability of place fields and increased the stability of reward-associated odor representations, which were most consistently retrieved during periods of sniffing and digging when animals were restricted to the cup locations. Together, these results suggest that attention selectively modulates the encoding and retrieval of hippocampal representations by enhancing physiological responses to task-relevant information.

Attention modulates the encoding and retrieval of memories, but the physiological basis of this interaction has largely been unexplored. The formation of memories which depend on the hippocampus involves the conscious recall of events that occur in specific spatial contexts, a form of memory known as episodic. To investigate the physiological consequences of the interaction between attention and memory in the hippocampus, we recorded single-cell activity and local field potentials — the local rhythmic oscillatory activity of neurons — from the same cells over several days while animals learned one of two goal-oriented tasks. In the visuospatial version of the task, mice had to associate a specific spatial location with a reward, independent of an odor cue. In the nonspatial, olfactory version, mice had to associate a specific odor with the food reward, independent of spatial location. We found that, during periods of navigation, only neurons in the visuospatially trained animals displayed long-term stable representations of space, and neuronal synchronization to so-called gamma oscillations, a mechanism of signal amplification that has been proposed to underlie attentional processes. Conversely, when animals were sniffing the odors in fixed spatial locations, only neurons in the olfactory-trained group displayed a stable increase in firing rate in response to the reward-associated odor. Our data suggest that attention modulates what is encoded and retrieved by hippocampal cells and that neuronal synchronization to gamma oscillations may underlie the mechanism whereby attention leads to stable spatial memory retrieval during navigation.

This paper introduces triangle-GAN ($\triangle$-GAN) that aims at cross-domain joint distribution matching: The model is shown below

https://i.imgur.com/boIDOMu.png

Having two domains of data $x$ and $y$, there are two generators:

1- $G_x(y)$ which takes $y$ and generates $\tilde{x}$

2- $G_y(x)$ which takes $x$ and generates $\tilde{y}$

There are two discriminators in the model:

1- $D_1 (x,y)$ a discriminator that distinguishes between $(x, y)$ and either of $(x, \tilde{y})$ or $(\tilde{x}, y)$.

2- $D_2 (x,y)$ a discriminator that distinguishes between $(x, \tilde{y})$ and $(\tilde{x}, y)$.

The second discriminator is ALI and can be used on un-paired sets of data.
The first discriminator is equivalent to a conditional discriminator where the true paired data $(x, y)$ is compared to either $(x, \tilde{y})$ or $(\tilde{x}, y)$, where one element in the pair is sampled. This discriminator needs paired $(x, y)$ data for training. 

This model can be used for semi-supervised settings, where a small set of paired data is provided. In this paper it is used for:

-  semi-supervised image classification, where a small subset of CIFAR10 is labelled. $x$ and $y$ are images and class labels here.

- image to image translation on edge2shoes dataset, where only a subset of dataset is paired.

- attribute conditional image generation where $x$ and $y$ domains are image and attributes. CelebA and COCO datasets are used here. In one experiment test-set images are projected to attributes and then given those attributes new images are generated:

On celebA:
https://i.imgur.com/EX5tDZ0.png

On COCO:
https://i.imgur.com/GRpvjGx.png

In another experiment some attributes are chosen (as samples shown below in the first row with different noise) and then another feature is added (using the same noise) to generate the samples in the second row:
 https://i.imgur.com/KeHL8Ye.png

The triangle gan demonstrates improved performance compared to triple gan in the experiments shown in the paper. It has been also compared with Disco gan (a model that can be trained on un-paired data) and shows improved performance when some percentage of paired data is provided. In an experiment they pair each MNIST digit with its transposed (as $x$, $y$ pairs). Disco-GAN cannot learn correct mapping between them, while triangle-GAN can learn correct mapping since it leverages paired data.

https://i.imgur.com/Vz9Zfhu.png

In general this model is a useful approach for semi-supervised cross-domain matching and can leverage un-paired data (using ALI) as well as paired data (using conditional discriminator).