This paper introduces a modification to the ResNets architecture with multi-level shortcut connections (shortcut from input to pre-final layer as level 1, shortcut over each residual block group as level 2, etc) as opposed to single-level shortcut connections in prior work on ResNets. The authors perform experiments with multi-level shortcut connections on regular ResNets, ResNets with pre-activations and Wide ResNets. Combined with drop-path regularization via stochastic depth and exploration over optimal shortcut level number and optimal depth/width ratio to avoid vanishing gradients and overfitting, this architecture achieves state-of-the-art error rates on CIFAR-10 (3.77%), CIFAR-100 (19.73%) and SVHN (1.59%).

## Strengths

- Fairly exhaustive set of experiments over
    - Shortcut level numbers.
    - Identity mapping types: 1) zero-padding shortcuts, 2) 1x1 convolutions for projections and others identity, and 3) all 1x1 convolutions.
    - Residual block size (2 or 3 3x3 convolutional layers).
    - Depths (110, 164, 182, 218) and widths for both ResNets and Pre-ResNets.

[Code](https://github.com/ashwinkalyan/dbs), [Live Demo](http://dbs.cloudcv.org/) ([code for demo site]( https://github.com/Cloud-CV/diverse-beam-search))

Diverse Beam Search (DBS) is an alternative to Beam Search (BS). Decodes diverse lists by dividing the beam budget $B$ (e.g. 6) into groups $G$ (e.g. 3) and enforcing diversity between groups of beams.

For every time step $t$ iterate over all groups. In 1st group find $B'=B/G$ (e.g. 2) partial beams $Y^1_{[t]} = \{y^1_{b,t} : b \in [B']\}$ using BS with NLL. In 2nd group find partial beams $y^2_{b,t}$ using BS with partial beam score taken to be the sum of NLL and the distance between the partial beam and the partial beams in 1st group. The distance is multiplied by a factor $\lambda_t$. For group $g$ the distance is measured between the partial beam $y^g_{b,t}$ and all the partial beams in all groups that were already optimized for current time step. $\Delta(Y^1_{[t]},\ldots,Y^{g-1}_{[t]})[y^g_{b,t}]$

Evaluation Metrics:
* Oracle Accuracy: maximum value of the metric (BLEU) over a list of final beams
* Diversity Statistics: number of distinct n-grams in all final beams
* Human preference

Parameters:
* $G=B$ allows for the maximum exploration and  found to improve oracle accuracy.
* $\lambda \in [0.2-0.8]$

Distance between partial beam and all other groups is broken to a sum of the distances with each group:
$$\Delta(Y^1_{[t]},\ldots,Y^{g-1}_{[t]}) = \sum^{g-1}_{h=1}\Delta(Y^h_{[t]})$$
individual $\Delta(Y^h_{[t]})[y^g_{b,t}]$ is taken to be one of:
* Hamming (gives best oracle performance): proportional to the number of times latest token in $y^g_{b,t}$ was selected as latest token in beams in $Y^h_{[t]}$.
* Cumulative: cancels out Hamming: $\exp\{-(\sum_{\tau \in t} \sum_{b \in B'} \mathbb{1}_{[y^h_{b,\tau} \neq y^g_{b,\tau}]})/\Gamma\}$
* n-gram: number of times each n-gram in a candidate occurred in previous groups
* Neural-embedding: in all previous methods replace hamming similarity with cosine of word2vec of token (or sum of word2vec of n-gram tokens)


My 2 cents:
* Once a beam reaches EOS you need to stop comparing it with other groups
* Using DBS cause results to be longer. Perhaps too much. You can reduce length by adding a penalty to length

TLDR; The authors augment the A3C (Asynchronous Actor Critic) algorithm with auxiliary tasks. These tasks share some of the network parameters but value functions for them are learned off-policy using n-step Q-Learning. The auxiliary tasks only used to learn a better representation and don't directly influence the main policy control. The technique, called UNREAL (Unsupervised Reinforcement and Auxiliary Learning), outperforms A3C on both the Atari and Labyrinth domains in terms of performance and training efficiency.


#### Key Points

- Environments contain a wide variety of possible training signals, not just cumulative reward
- Base A3C agent uses CNN + RNN
- Auxiliary Control and Prediction tasks share the convolutional and LSTM network for the "base agent". This forces the agent to balance improvement and base and aux. tasks.
- Auxiliary Tasks
  - Use off-policy RL algorithms (e.g. n-step Q-Learning) so that the same stream of experience from the base agent can be used for maximizing all tasks. Experience is sampled from a replay buffer.
  - Pixel Changes (Auxiliary Control): Learn a policy for maximally changing the pixels in a grid of cells overlaid over the images
  - Network Features (Auxiliary Control): Learn a policy for maximally activating units in a specific hidden layer
  - Reward Prediction (Auxiliary Reward): Predict the next reward given some historical context. Crucially, because rewards tend to be sparse, histories are sampled in a skewed manner from the replay buffer so that P(r!=0) = 0.5. Convolutional features are shared with the base agent.
  - Value Function Replay: Value function regression for the base agent with varying window for n-step returns.
- UNREAL
  - Base agent is optimized on-policy (A3C) and aux. tasks are optimized off-policy.
- Experiments
  - Agent is trained with 20-step returns and aux. tasks are performed every 20 steps.
  - Replay buffer stores the most recent 2k observations, actions and rewards
  - UNREAL tends to be more robust to hyperparameter settings than A3C
  - Labyrinth
    - 38% -> 83% human-normalized score. Each aux. tasks independently adds to the performance.
    - Significantly faster learning, 11x across all levels
    - Compared to input reconstruction technique: Input reconstruction hurts final performance b/c it puts too much focus on reconstructing relevant parts.
  - Atari
    - Not all experiments are completed yet, but UNREAL already surpasses state of the art agents and is more robust.

#### Thoughts

- I want an algorithm box please :)

Single cells behave in complex and sometimes seemingly random ways. We have applied Generative Adversarial Networks (GANs), a form of artificial intelligence, to make sense of the way genes are controlled in skin cells. A GAN involves two separate neural networks, a ‘generator’ and a ‘discriminator’. The generator simulates cells and the discriminator tries to tell the difference between the fake data created by the simulator, and data from real cells. As they compete against each other they improve at their tasks and provide new insights into the way cells behave.   

Our neural network approach allows us to understand the relationship between different genes and how this contributes to cell behaviour. One of the networks, the generator, is responsible for simulating cells and we use this to predict how genes are controlled under different conditions, effectively simulating what would previously have been laborious and painstaking experiments. GANs also make it possible to compare data from multiple labs produced in different conditions, opening up the opportunity to answer new questions.

This paper implements a parameter expansion technique with Markov Chain Monte Carlo (MCMC) for the multivariate probit model (MVP). Since the covariance matrix is not identifiable from multivariate ordinal data, a correlation matrix restriction is appropriate with MVP. This is a problem for Bayesian inference, as the structural constraints on correlation matrices make the selection of a good prior and the estimation of a posterior difficult.   

The main advantage of parameter expansion with this model is that we can draw the covariance matrix in the expanded parameter space from an inverse Wishart distribution with a simple Gibbs sampler. At each iteration, all parameters are drawn from the expanded space and rescaled to their original form. Since the parameter draws are only dependent on the latest latent variable, the dependence between iterations is reduced. This technique improves the mixing and convergence rate of the MCMC.

Parameter expansion for the MVP model is also easy to implement for stochastic search variable selection (SSVS). An indicator variable $\lambda_i$ is included as a parameter, where each 1 in $\lambda_i$ is associated to a covariate selected for the model.

Gibbs sampler algorithm:

Let $q$-variate vector $Z_i \sim MVN( \beta X_i, R )$ be a latent variable, truncated in all dimensions according to the corresponding cutpoints, where $R$ is a $q \times q$ correlation matrix.

Let $\beta$ be a $q \times p$ regression coefficient matrix, $\gamma$ be the set of cutpoints where $\gamma_0 = -\infty$, $\gamma_1 = 0$ and $\gamma_k = \infty$, and $k$ be the number of categories for the response.


Map $Z $ onto $W$:

Cycle through draws of the univariate truncated $Z_{i,j} \sim N(\mu_{i,j} - \frac{1}{R^{-1}_{j,j}}R^{-1}_{j,-j}(Z_{i,-j}-\mu_{i,-j}),  \frac{1}{R^{-1}_{j,j}})$, where $\mu_i = \beta X_i$, to get the truncated multivariate normal distribution of $W_i$.

$R \rightarrow \Sigma$:

Draw $\Sigma | Z \sim IW(n-p+q-1, WW^T - WX^T(XX^T)^{-1}XW^T)$

$\beta \rightarrow \alpha$:

Draw $\alpha | W, \Sigma \sim N_{q,p}(WX^T(XX^T)^{-1}, \Sigma \otimes (XX^T)^{-1})$

where $N_{q,p}$ is a matrix variate normal distribution.

$\gamma \rightarrow \theta$:

Draw $\theta_{j,k+1} | W, Y \sim Unif(max_i(\{W_{ij}|Y_{ij}=c\}), min_i(\{W_{ij}|Y_{ij} = c + 1\})) $ for all free cutpoints.

Rescale the parameters back to the original space:

$Z_{i,j} = \dfrac{W_{i,j}}{\sqrt{\Sigma_{j,j}}}$

$R_{i,j} = \dfrac{\Sigma_{i,j}}{\sqrt{\Sigma_{i,i}\cdot\Sigma_{j,j}}}$

$\beta_{k,j} = \dfrac{\alpha_{k,j}}{\sqrt{\Sigma_{j,j}}}$

$\gamma_{j,k+1} = \dfrac{\theta_{j,k+1}}{\sqrt{\Sigma_{j,j}}}$

Repeat.