The regulation of filopodia plays a crucial role during neuronal development and synaptogenesis. Axonal filopodia, which are known to originate presynaptic specializations, are regulated in response to neurotrophic factors. The structural components of filopodia are actin filaments, whose dynamics and organization are controlled by ensembles of actin-binding proteins. How neurotrophic factors regulate these latter proteins remains, however, poorly defined. Here, using a combination of mouse genetic, biochemical, and cell biological assays, we show that genetic removal of Eps8, an actin-binding and regulatory protein enriched in the growth cones and developing processes of neurons, significantly augments the number and density of vasodilator-stimulated phosphoprotein (VASP)-dependent axonal filopodia. The reintroduction of Eps8 wild type (WT), but not an Eps8 capping-defective mutant, into primary hippocampal neurons restored axonal filopodia to WT levels. We further show that the actin barbed-end capping activity of Eps8 is inhibited by brain-derived neurotrophic factor (BDNF) treatment through MAPK-dependent phosphorylation of Eps8 residues S624 and T628. Additionally, an Eps8 mutant, impaired in the MAPK target sites (S624A/T628A), displays increased association to actin-rich structures, is resistant to BDNF-mediated release from microfilaments, and inhibits BDNF-induced filopodia. The opposite is observed for a phosphomimetic Eps8 (S624E/T628E) mutant. Thus, collectively, our data identify Eps8 as a critical capping protein in the regulation of axonal filopodia and delineate a molecular pathway by which BDNF, through MAPK-dependent phosphorylation of Eps8, stimulates axonal filopodia formation, a process with crucial impacts on neuronal development and synapse formation.

Neurons communicate with each other via specialized cell-cell junctions called synapses. The proper formation of synapses ("synaptogenesis") is crucial to the development of the nervous system, but the molecular pathways that regulate this process are not fully understood. External cues, such as brain-derived neurotrophic factor (BDNF), trigger synaptogenesis by promoting the formation of axonal filopodia, thin extensions projecting outward from a growing axon. Filopodia are formed by elongation of actin filaments, a process that is regulated by a complex set of actin-binding proteins. Here, we reveal a novel molecular circuit underlying BDNF-stimulated filopodia formation through the regulated inhibition of actin-capping factor activity. We show that the actin-capping protein Eps8 down-regulates axonal filopodia formation in neurons in the absence of neurotrophic factors. In contrast, in the presence of BDNF, the kinase MAPK becomes activated and phosphorylates Eps8, leading to inhibition of its actin-capping function and stimulation of filopodia formation. Our study, therefore, identifies actin-capping factor inhibition as a critical step in axonal filopodia formation and likely in new synapse formation.

Normal RL agents in multi-agent scenarios treat their opponents as a static part of the environment, not taking into account the fact that other agents are learning as well. This paper proposes LOLA, a learning rule that should take the agency and learning of opponents into account by optimizing "return under one step look-ahead of opponent learning"

So instead of optimizing under the current parameters of agent 1 and 2 
$$V^1(\theta_i^1, \theta_i^2)$$

LOLA proposes to optimize taking into account one step of opponent (agent 2) learning
$$V^1(\theta_i^1, \theta_i^2 + \Delta \theta^2_i)$$

where we assume the opponent's naive learning update $\Delta \theta^2_i = \nabla_{\theta^2} V^2(\theta^1, \theta^2) \cdot \eta$ and we add a second-order correction term

on top of this, the authors propose
- a learning rule with policy gradients in the case that the agent does not have access to exact gradients
- a way to estimate the parameters of the opponent, $\theta^2$, from its trajectories using maximum likelihood in the case you can't access them directly
$$\hat \theta^2 = \text{argmax}_{\theta^2} \sum_t \log \pi_{\theta^2}(u_t^2|s_t)$$

LOLA is tested on iterated prisoner's dilemma and converges to a tit-for-tat strategy more frequently than the naive RL learning algorithm, and outperforms it. LOLA is tested on iterated matching pennies (similar to prisoner's dilemma) and stably converges to the Nash equilibrium whereas the naive learners do not. In testing on coin game (a higher dimensional version of prisoner's dilemma) they find that naive learners generally choose the defect option whereas LOLA agents have a mostly-cooperative strategy.

As well, the authors show that LOLA is a dominant learning rule in IPD, where both agents always do better if either is using LOLA (and even better if both are using LOLA).

Finally, the authors also propose second order LOLA, which instead of assuming the opponent is a naive learner, assumes the opponent uses a LOLA learning rule. They show that second order LOLA does not lead to improved performance so there is no need to have a $n$th order LOLA arms race.

This paper introduces a deep universal word embedding based on using a bidirectional LM (in this case, biLSTM). First words are embedded with a CNN-based, character-level, context-free, token embedding into $x_k^{LM}$ and then each sentence is parsed using a biLSTM, maximizing the log-likelihood of a word given it's forward and backward context (much like a normal language model). 

The innovation is in taking the output of each layer of the LSTM ($h_{k,j}^{LM}$ being the output at layer $j$) 
$$
\begin{align}
R_k &= \{x_k^{LM}, \overrightarrow{h}_{k,j}^{LM}, \overleftarrow{h}_{k,j}^{LM} | j = 1 \ldots L \} \\
&= \{h_{k,j}^{LM} | j = 0 \ldots L \}
\end{align}
$$
and allowing the user to learn a their own task-specific weighted sum of these hidden states as the embedding:
$$
ELMo_k^{task} = \gamma^{task} \sum_{j=0}^L s_j^{task} h_{k,j}^{LM}
$$

The authors show that this weighted sum is better than taking only the top LSTM output (as in their previous work or in CoVe) because it allows capturing syntactic information in the lower layer of the LSTM and semantic information in the higher level. Table below shows that the second layer is more useful for the semantic task of word sense disambiguation, and the first layer is more useful for the syntactic task of POS tagging.

https://i.imgur.com/dKnyvAa.png

On other benchmarks, they show it is also better than taking the average of the layers (which could be done by setting $\gamma = 1$)
https://i.imgur.com/f78gmKu.png

To add the embeddings to your supervised model,  ELMo is concatenated with your context-free embeddings, $\[ x_k; ELMo_k^{task} \]$. It can also be concatenated with the output of your RNN model $\[ h_k; ELMo_k^{task} \]$ which can show improvements on the same benchmarks

https://i.imgur.com/eBqLe8G.png

Finally, they show that adding ELMo to a competitive but simple baseline gets SOTA (at the time) on very many NLP benchmarks

https://i.imgur.com/PFUlgh3.png

It's all open-source and there's a tutorial [here](https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md)

How can humans help an agent perform at a task that has no clear reward? Imitation, demonstration, and preferences. This paper asks which combinations of imitation, demonstration, and preferences will best guide an agent in Atari games.

For example an agent that is playing Pong on the Atari, but can't access the score. You might help it by demonstrating your play style for a few hours. To help the agent further you are shown two short clips of it playing and you are asked to indicate which one, if any, you prefer.

To avoid spending many hours rating videos the authors sometimes used an automated approach where the game's score decides which clip is preferred, but they also compared this approach to human preferences. It turns out that human preferences are often worse because of reward traps. These happen, for example, when the human tries to encourage the agent to explore ladders, resulting in the agent obsessing about ladders instead of continuing the game.

They also observed that the agent often misunderstood the preferences it was given, causing unexpected behavior called reward hacking. The only solution they mention was to have someone keep an eye on it and continue giving it preferences, but this isn't always feasible. This is the alignment problem which is a hard problem in AGI research.

Results: adding merely a few thousand preferences can help in most games, unless they have sparse rewards. Demonstrations, on the other hand, tend to help those games with sparse rewards but only if the demonstrator is good at the game.

This paper tackles a challenging task of hand shape and continuous Sign Language Recognition (SLR) directly from images obtained from a common RGB camera (rather than utilizing motion sensors like Kinect). The basic idea is to create a network that is end-to-end trainable with input (i.e. images) and output (i.e. hand shape labels, word labels) sequences. The network is composed of three parts:
 - CNN as a feature extractor
 - Bidirectional LSTMs for temporal modeling
 - Connectionist Temporal Classification as a loss layer 
![Network structure](https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/3269d3541f0eec006aee6ce086db2665b7ded92d/1-Figure1-1.png)

Results:
 - Observed state-of-art results (at the time of publishing) on "One-Million Hands" and "RWTH-PHOENIX-Weather-2014" datasets.
 - Utilizing full images rather than hand patches provides better performance for continuous SLR. 
 - A network that recognizes hand shape and a network that recognizes word sequence can be combined and trained together to recognize word sequences. Finetuning combined system from for all layers works better than fixing "feature extraction" layers.
 - Combination of two networks where each network trained on separate task performs slightly better than training each network on word sequences.
 - Marginal difference in performance observed for different decoding and post-processing techniques during sequence-to-sequence predictions.