- *issue:* RL on real systems -> sparse and slow data sampling;
- *solution:* pre-train the agent with the EGAN;
- *performance:* ~20% improvement of training time in the beginning of learning compared to no pre-training; ~5% improvement and smaller variations compared to GAN pre-training.

## Introduction

5G telecom systems -> fufill ultra-low latency, high robustness, quick response to changed capacity needs, and dynamic allocation of functionality.

*Problems:*

1. exploration has an impact on the service quality in real-time service systems;
2. sparse and slow data sampling -> extended training duration.

## Enhanced GAN

**Fomulas**

the training data for RL tasks:

$$x = [x_1, x_2] = [(s_t,a),(s_{t+1},r)]$$

the generated data:

$$G(z) = [G_1(z), G_2(z)] =  [(s'_t,a'),(s'_{t+1},r')] $$

the value function for GAN:

$$V(D,G) = \mathbb{E}_{z \sim p_z(z)}[\log(1-D(G(z)))] + \lambda D_{KL}(P||Q)$$

where the regularization term $D_{KL}$ has the following form:

$$D_{KL}(P||Q) = \sum_i P(i) \log \frac{P(i)}{Q(i)}$$

**EGAN structure**

https://i.imgur.com/FhPxamJ.png

**Algorithm**

https://i.imgur.com/RzOGmNy.png

The enhancer is fed with training data *D\_r(s\_t, a)* and *D\_r(s\_{t+1}, r)*, and trained by supervised learning. After GAN generates synthetic data *D\_t(s\_t, a, s\_{t+1}, r)*, the enhancer could enhance the dependency between *D\_t(s\_t, a)* and *D\_t(s\_{t+1}, r)* and update the weights of GAN.

## Results

two lines of experiments on CartPole environment involved with PG agents: 

1. one for comparing the learning curves of agents with no pre-training, GAN pre-training and EGAN pre-training. => Result: EGAN > GAN > no pre-training

2. one for comparing the learning curves of agents with EGAN pre-training for various episodes (500, 2000, 5000). => Result: 5000 > 2000 ~= 500

During the development of neural circuitry, neurons of different kinds establish specific synaptic connections by selecting appropriate targets from large numbers of alternatives. The range of alternative targets is reduced by well organised patterns of growth, termination, and branching that deliver the terminals of appropriate pre- and postsynaptic partners to restricted volumes of the developing nervous system. We use the axons of embryonic Drosophila sensory neurons as a model system in which to study the way in which growing neurons are guided to terminate in specific volumes of the developing nervous system. The mediolateral positions of sensory arbors are controlled by the response of Robo receptors to a Slit gradient. Here we make a genetic analysis of factors regulating position in the dorso-ventral axis. We find that dorso-ventral layers of neuropile contain different levels and combinations of Semaphorins. We demonstrate the existence of a central to dorsal and central to ventral gradient of Sema 2a, perpendicular to the Slit gradient. We show that a combination of Plexin A (Plex A) and Plexin B (Plex B) receptors specifies the ventral projection of sensory neurons by responding to high concentrations of Semaphorin 1a (Sema 1a) and Semaphorin 2a (Sema 2a). Together our findings support the idea that axons are delivered to particular regions of the neuropile by their responses to systems of positional cues in each dimension.

"The SE module can learn some nonlinear global interactions already known to be useful, such as spatial normalization. The channel wise weights make it somewhat more powerful than divisive normalization as it can learn feature-specific inhibitions (ie: if we see a lot of flower parts, the probability of boat features should be diminished). It also has some similarity to bio inhibitory circuits." By jcannell on reddit

Slides: http://image-net.org/challenges/talks_2017/SENet.pdf

Summary by the author Jie Hu:

Our motivation is to explicitly model the interdependence between feature channels. In addition, we do not intend to introduce a new spatial dimension for the integration of feature channels, but rather a new "feature re-calibration" strategy. Specifically, it is through learning the way to automatically obtain the importance of each feature channel, and then in accordance with this importance to enhance the useful features and inhibit the current task is not useful features.

https://i.imgur.com/vXyBg4j.png

The above figure is a schematic diagram of our proposed SE module. Given an input $x$, the number of characteristic channels is $c_1$, and the characteristic number of a characteristic channel is $c_2$ by a series of convolution and other general transformations. Unlike traditional CNNs, we then re-calibrate the features we received in the next three operations.

The first is the Squeeze operation, we carry out the feature compression along the spatial dimension, and turn each two-dimensional feature channel into a real number. The real number has a global sense of the wild, and the output dimension and the number of input channels Match. It characterizes the global distribution of responses on the feature channel, and makes it possible to obtain a global sense of the field near the input, which is very useful in many tasks.

Followed by the Excitation operation, which is a mechanism similar to the door in a circular neural network. The weight is generated for each feature channel by the parameter $w$, where the parameter w is learned to explicitly model the correlation between the feature channels.

Reddit thread: https://www.reddit.com/r/MachineLearning/comments/6pt99z/r_squeezeandexcitation_networks_ilsvrc_2017/

They propose neural models for dialogue state tracking, making a binary decision for each possible slot-value pair, based on the latest context from the user and the system. The context utterances and the slot-value option are encoded into vectors, either by summing word representations or using a convnet. These vectors are then further combined to produce a binary output. The systems are evaluated on two dialogue datasets and show improvement over baselines that use hand-constructed lexicons.

https://i.imgur.com/G4rm954.png

This is Adagrad. Adagrad is an adaptive learning rate method. Some sample code from  [[Stanford CS231n]](https://cs231n.github.io/neural-networks-3/#ada) is:

```python
# Assume the gradient dx and parameter vector x
cache += dx**2
x += - learning_rate * dx / (np.sqrt(cache) + eps)
```