They created a really nice trick to optimize the $ {L}_{0} $ Pseudo Norm - Regularization on the sorted (By magnitude) values of the optimization variable.

Their code is available at - [The Trimmed Lasso: Sparsity and Robustness](https://github.com/copenhaver/trimmedlasso).

They extend skip-grams for word embeddings to use character n-grams. Each word is represented as a bag of character n-grams, 3-6 characters long, plus the word itself. Each of these has their own embedding which gets optimised to predict the surrounding context words using skip-gram optimisation. They evaluate on word similarity and analogy tasks, in different languages, and show improvement on most benchmarks.

A joint model for training bilingual embeddings without supervision. As background information they point out that embeddings in different languages tend to be linear transforms of each other. Then they create a new network and objective that builds on CBOW training, but has two terms in the loss function: The traditional loss for the CBOW language model, and an additional loss over the mean and variance of each dimension of the embedding space. The hypothesis, I guess, is that constraining each dimension of each languages embeddings to be similar (mean and variance) will put them in the same space from the start (i.e., no transformation needed). During training, they then sample a word randomly from the source (or target) language, feed forward, and compute the distribution-based gradient using the target (or source) means as the gold, and add it to the standard gradient.
They evaluate on English-French and English-Chinese using aligned corpora, and get better, though not good, performance.

There are two issues that I can think of that might affect the (lack of) accuracy. First, the mean/variance similarity constraints might be not constrained enough. Second, the fact that the other languages in-progress embedding statistics are used as gold may cause it to get stuck in local optima? FWIW,  Zhang et al '17 (ACL) also question the distributional assumptions (that each dimension is Gaussian-distributed).

Pérolat et al. propose a game-theoretic variant of adversarial training on universal adversarial perturbations. In particular, in each training iteration, the model is trained for a specific number of iterations on the current training set. Afterwards, a universal perturbation is found (and the corresponding test images) that fools the network. The found adversarial examples are added to the training set. In the next iteration, the network is trained on the new training set which includes adversarial examples. Overall, this leads to a network being trained on a sequence of universal adversarial perturbations corresponding to earlier versions of that network.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).

Active learning (choosing which examples to annotate for training) is proposed as a reinforcement learning problem. The Q-learning network predicts for each sentence whether it should be annotated, and is trained based on the performance improvement from the main task. Evaluation is done on NER, with experiments on transferring the trained Q-learning function to other languages.

https://i.imgur.com/5rXm5vZ.png