#### Introduction

* **Machine Comprehension (MC)** - given a natural language sentence, answer a natural language question.
* **End-To-End MC** - can not use language resources like dependency parsers. The only supervision during training is the correct answer.
* **Query Regression Network (QRN)** - Variant of Recurrent Neural Network (RNN).
* [Link to the paper](http://arxiv.org/abs/1606.04582)

#### Related Work

* Long Short-Term Memory (LSTM) and Gated Recurrence Unit (GRU) are popular choices to model sequential data but perform poorly on end-to-end MC due to long-term dependencies.
* Attention Models with shared external memory focus on single sentences in each layer but the models tend to be insensitive to the time step of the sentence being accessed.
* **Memory Networks (and MemN2N)**
    * Add time-dependent variable to the sentence representation.
    * Summarize the memory in each layer to control attention in the next layer.
* **Dynamic Memory Networks (and DMN+)**
    * Combine RNN and attention mechanism to incorporate time dependency.
    * Uses 2 GRU
        * time-axis GRU - Summarize the memory in each layer.
        * layer-axis GRU - Control the attention in each layer.
* QRN is a much simpler model without any memory summarized node.

#### QRN

* Single recurrent unit that updates its internal state through time and layers.
* Inputs
    * $q_{t}$ - local query vector
    * $x_{t}$ - sentence vector
* Outputs
    * $h_{t}$ - reduced query vector
    * $x_{t}$ - sentence vector without any modifications
* Equations
    * $z_{t} = \alpha (x_{t}, q_{t})$
    * &alpha is the **update gate function** to measure the relevance between input sentence and local query.
    * $h`_{t} = \gamma (x_{t}, q_{t})$
    * &gamma is the **regression function** to transform the local query into regressed query.
    * $h_{t} = z_{t} \* h'_{t} + (1 - z_{t}) \* h_{t-1}$
* To create a multi layer model, output of current layer becomes input to the next layer.

#### Variants

* **Reset gate function** ($r_{t}$) to reset or nullify the regressed query $h`_{t}$ (inspired from GRU).
    * The new equation becomes $h_{t} = z_{t}\*r_{t}\* h`_{t} + (1 - z_{t})\*h_{t-1}$
* **Vector gates** - update and reset gate functions can produce vectors instead of scalar values (for finer control).
* **Bidirectional** - QRN can look at both past and future sentences while regressing the queries.
    * $q_{t}^{k+1} = h_{t}^{k, \text{forward}} + h_{t}^{k, \text{backward}}$.
    * The variables of update and regress functions are shared between the two directions.

#### Parallelization

* Unlike most RNN based models, recurrent updates in QRN can be computed in parallel across time.
* For details and equations, refer the [paper](http://arxiv.org/abs/1606.04582).

#### Module Details

##### Input Modules
    
* A trainable embedding matrix A is used to encode the one-hot vector of each word in the input sentence into a d-dimensional vector.
* Position Encoder is used to obtain the sentence representation from the d-dimensional vectors.
* Question vectors are also obtained in a similar manner.

##### Output Module

* A V-way single-layer softmax classifier is used to map predicted answer vector $y$ to a V-dimensional sparse vector $v$.
* The natural language answer $y is the arg max word in $v$.


#### Results

* [bAbI QA](https://gist.github.com/shagunsodhani/12691b76addf149a224c24ab64b5bdcc) dataset used.
* QRN on 1K dataset with '2rb' (2 layers + reset gate + bidirectional) model and on 10K dataset with '2rvb' (2 layers + reset gate + vector gate + bidirectional) outperforms MemN2N 1K and 10K models respectively.
* Though DMN+ outperforms QRN with a small margin, QRN are simpler and faster to train (the paper made the comment on the speed of training without reporting the training time of the two models).
* With very few layers, the model lacks reasoning ability while with too many layers, the model becomes difficult to train.
* Using vector gates works for large datasets while hurts for small datasets.
* Unidirectional models perform poorly.
* The intermediate query updates can be interpreted in natural language to understand the flow of information in the network.

Large-scale transformers on unsupervised text data have been wildly successful in recent years; arguably, the most successful single idea in the last ~3 years of machine learning. Given that, it's understandable that different domains within ML want to take their shot at seeing whether the same formula will work for them as well. This paper applies the principles of (1) transformers and (2) large-scale unlabeled data to the problem of learning informative embeddings of molecular graphs. 

Labeling is a problem in much of machine learning - it's costly, and narrowly defined in terms of a certain task - but that problem is even more exacerbated when it comes to labeling properties of molecules, since they typically require wetlab chemistry to empirically measure. Given that, and also given the fact that we often want to predict new properties - like effectiveness against a new targetable drug receptor - that we don't yet have data for, finding a way to learn and transfer from unsupervised data has the potential to be quite valuable in the molecular learning sphere. 

There are two main conceptual parts to this paper and its method - named GROVER, in true-to-ML-form tortured acronym style. The first is the actual architecture of their model itself, which combines both a message-passing Graph Neural Network to aggregate local information, and a Transformer to aggregate global information. The paper was a bit vague here, but the way I understand it is: 

https://i.imgur.com/JY4vRdd.png
- There are parallel GNN + Transformer stacks for both edges and nodes, each of which outputs both a node and edge embedding, for four embeddings total. I'll describe the one for nodes, and the parallel for edges operates the same way, except that hidden states live on edges rather than nodes, and attention is conducted over edges rather than nodes
- In the NodeTransformer version, a message passing NN (of I'm not sure how many layers) performs neighborhood aggregation (aggregating the hidden states of neighboring nodes and edges, then weight-transforming them, then aggregating again) until each node has a representation that has "absorbed" in information from a few hops out of its surrounding neighborhood. My understanding is that there is a separate MPNN for queries, keys, and values, and so each nodes end up with three different vectors for these three things.
- Multi-headed attention is then performed over these node representations, in the normal way, where all keys and queries are dot-product-ed together, and put into a softmax to calculate a weighted average over the values
- We now have node-level representations that combine both local and global information. These node representations are then aggregated into both node and edge representations, and each is put into a MLP layer and Layer Norm before finally outputting a node-based node and edge representation. This is then joined by an edge-based node and edge representation from the parallel stack. These are aggregated on a full-graph level to predict graph-level properties

https://i.imgur.com/NNl6v4Y.png

The other component of the GROVER model is the way this architecture is actually trained - without explicit supervised labels. The authors use two tasks - one local, and one global. The local task constructs labels based on local contextual properties of a given atom - for example, the atom here has one double-bonded Nitrogen and one single-bonded Oxygen in its local environment - and tries to predict those labels given the representations of that atom (or node). The global task uses RDKit (an analytically constructed molecular analysis kit) to identify 85 different modifs or functional groups in the molecule, and encodes those into an 85-long one-hot vector that is being predicted on a graph level. 

https://i.imgur.com/jzbYchA.png

With these two components, GROVER is pretrained on 10 million unlabeled molecules, and then evaluated in transfer settings where its representations are fine-tuned on small amounts of labeled data. The results are pretty impressive - it achieves new SOTA performance by relatively large amounts on all tasks, even relative to exist semi-supervised pretraining methods that similarly have access to more data. The authors perform ablations to show that it's important to do the graph-aggregation step before a transformer (the alternative being just doing a transformer on raw node and edge features), and also show that their architecture without pretraining (just used directly in downstream tasks) also performs worse. One thing I wish they'd directly ablated was the value-add of the local (also referred to as "contextual") and global semi-supervised tasks. Naively, I'd guess that most of the performance gain came from the global task, but it's hard to know without them having done the test directly.

This paper introduces Residual Nets (ResNets), which was the
winning submission (152-layer deep) at ILSVRC 2015 and MS-COCO 2015, and achieves
a top-5 error rate of 3.57% (ensemble of two  nets). Main contributions:

- The key idea is that deeper networks face the degradation problem, i.e.
higher training and test error than shallower nets, because they're harder
to optimize for approximating identity mapping by multiple non-linear layers.
    - They mitigate this problem by forcing solvers to learn residual functions
    i.e. $f(x) = H(x) - x$, by adding shortcut connections. If identity mapping is
    the optimal formulation, the learned weights should drive $f(x)$ to 0 (and they
    observe that this is a suitable preconditioning as most residual function responses
    are small).

- Shortcut connections (for identity mapping) don't require additional parameters.
    - Size transformations are done by zero-padding (no parameters) or projections. Projections
    introduce additional parameters and perform slightly better.

- Bottleneck design is used to further reduce computational complexity, i.e. 1x1 convolutional
layers before and after 3x3 convolutions to reduce and increase dimensions.

- For detection and localization tasks, they use ResNets in the Faster-RCNN setting.

## Strengths

- ResNets are significantly deeper and more accurate yet computationally cheaper than VGG.

- A single ResNet outperforms previous state-of-the-art ensembles. Their final winning submission
is an ensemble of two networks.

## Weaknesses / Notes

- The idea of shortcut connections to force blocks to learn residual functions preconditioned
on identity mapping is neat, and more so because it doesn't require additional parameters.

- A lot of results and design decisions merit further investigation and reasoning.
    - Why do shortcuts skip 2 or 3 layers? What happens to performance if we increase the number of layers skipped?
    - How well do shortcut connections work with Inception modules? The statistical principles
    underlying both these architectures seem to be orthogonal, does performance further improve?
    - 152 seems to be an arbitrary number of layers that 'worked'.

- The degradation problem seen when making networks deeper by initializing
layers with identity weight matrices seems to be contradictory to the results
presented in the Net2Net paper.

Serializability is the gold standard of consistency, but databases have always provided weaker consistency modes (e.g. Read Committed, Repeatable Read) that promise improved performance. In this paper, Bailis et al. determine which of these weaker consistency models can be implemented with high availability.

First, why is high availability important?

1. Partitions. Partitions happen, and when they do non-available systems become, well, unavailable.
2. Latency. Partitions may be transient, but latency is forever. Highly available systems can avoid latency by eschewing coordination costs.

Second, are weaker consistency models consistent enough? In short, yeah probably. In a survey of databases, Bailis finds that many do not employ serializability by default and some do not even provide full serializability. Bailis also finds that four of the five transactions in the TPC-C benchmark can be implemented with highly available transactions.

After defining availability, Bailis presents the taxonomy of which consistency can be implemented as HATs, and also argues why some fundamentally cannot. He also performs benchmarks on AWS to show the performance benefits of HAT.

When high-mass (>= 8 Msun) stars end their lives in blinding explosions known as core-collapse supernovae, they can rip through the fabric of space-time and create black holes with similar masses, known as stellar-mass black holes. These vermin black holes dwarf in comparison to their big brothers, supermassive black holes that typically have masses of 10⁶-10⁹ Msun. However, as vermin usually do, they massively outnumber supermassive black holes. It is estimated that 10⁸-10⁹ stellar mass black holes are crawling around our own Milky Way, but we’ve only caught sight of a few dozens of them.

As black holes don’t emit light, we can only infer their presence from their effects on nearby objects. All stellar mass black holes detected so far reside in binary systems, where they actively accrete from their companions. Radiation is emitted as matter from the companion falls onto the accretion disk of the black hole. Isolated black holes don’t have any companions, so they can only accrete from the surrounding diffuse interstellar medium, producing a very weak signal. That is why isolated black holes, which make up the bulk of the black hole population, have long escaped our discovery. Perhaps, until now.

The authors of today’s paper turned the intense gravity of black holes against themselves. While isolated black holes do not produce detectable emission, their gravity can bend and focus light from background objects. This bending and focusing of light through gravity is known as gravitational lensing. Astronomers categorize gravitational lensing based on the source and degree of lensing: strong lensing (lensing by a galaxy or a galaxy cluster producing giant arcs or multiple images), weak lensing (lensing by a galaxy or a galaxy cluster where signals are weaker and detected statistically), and microlensing (lensing by a star or planet). During microlensing, as the lens approaches the star, the star will brighten momentarily as more and more light is being focused, up until maximum magnification at closest approach, after which the star gradually fades as the lens leaves. This effect is known as photometric microlensing (see this astrobite). Check out this microlensing simulation, courtesy of Professor Scott Gaudi at The Ohio State University: the star (orange) is located at the origin, the lens (open red circle) is moving to the right, the gray regions trace out the lensed images (blue) as the lens passes by the star, while the green circle is the Einstein radius. The Einsten radius is the radius of the annular image when the observer, the lens, and the star are perfectly aligned.

Something more subtle can also happen during microlensing, and that is the shifting of the center of light (on a telescope’s detector) relative to the true position of the source — astrometric microlensing. While photometric microlensing has been widely used to search for exoplanets and MACHOs (massive astrophysical compact halo objects), for instance by OGLE (Optical Gravitational Lensing Experiment), astrometric microlensing has not been put to good use as it requires extremely precise measurements. Typical astrometric shifts caused by stellar mass black holes are sub-milliarcsecond (sub-mas), whereas the best astrometric precision we can achieve from the ground is typically ~1 mas or more. Figure 1 shows the signal evolution of photometric and astrometric microlensing and the astrometric shifts caused by different masses.

https://astrobites.org/wp-content/uploads/2016/08/fig1-2.png

Fig. 1 – Left panel shows an example of photometric magnification (dashed line) and astrometric shift (solid line) as function of time since the closest approach between the lens and the star. Note that the peak of the astrometric shift occurs after the peak of the photometric magnification. Right panel shows the astrometric shift as a function of the projected separation between the lens and the star, in units of the Einstein radius, for different lens masses. [Figure 1 in paper]

In this paper, the authors used adaptive optics on the Keck telescope to detect astrometric microlensing signals from stellar mass black holes. Over a period of 1-2 years, they monitored three microlensing events detected by the OGLE survey. As astrometric shift reaches a maximum after the peak of photometric microlensing (see Figure 1), astrometric follow-up was started post-peak for each event. The authors fit an astrometric lensing model to their data, not all of which were taken under good observing conditions. Figure 2 shows the results of the fit: all three targets are consistent with linear motion within the uncertainties of their measurements, i.e. no astrometric microlensing. Nonetheless, as photometric microlensing is still present, the authors used their astrometric model combined with a photometric lensing model to back out various lensing parameters, the most important one being the lens masses. They found one lens to have comparable mass as a stellar-mass black hole, although verification would require future observations.

https://astrobites.org/wp-content/uploads/2016/08/fig2-2.png

Despite not detecting astrometric microlensing signals, the authors demonstrated that they achieved the precision needed in a few epochs; had the weather goddess been on their side during some critical observing periods, some signals could have been seen. This study is also the first to combine both photometric and astrometric measurements to constrain lensing event parameters, ~20 years after this technique was first conceived. For now, we’ll give stellar-mass black holes a break, but it won’t be long until we catch up.

by Suk Sien Tie