* They define four subtasks of image understanding:
    * *Classification*: Assign a single label to a whole image.
    * *Captioning*: Assign a sequence of words (description) to a whole image*
    * *Detection*: Find objects/regions in an image and assign a single label to each one.
    * *Dense Captioning*: Find objects/regions in an image and assign a sequence of words (description) to each one.
  * DenseCap accomplishes the fourth task, i.e. it is a model that finds objects/regions in images and describes them with natural language.

### How
  * Their model consists of four subcomponents, which run for each image in sequence:
    * (1) **Convolutional Network**:
      * Basically just VGG-16.
    * (2) **Localization Layer**:
      * This layer uses a convolutional network that has mostly the same architecture as in the "Faster R-CNN" paper.
      * That ConvNet is applied to a grid of anchor points on the image.
      * For each anchor point, it extracts the features generated by the VGG-Net (model 1) around that point.
      * It then generates the attributes of `k` (default: 12) boxes using a shallow convolutional net. These attributes are (roughly): Height, width, center x, center y, confidence score.
      * It then extracts the features of these boxes from the VGG-Net output (model 1) and uses bilinear sampling to project them onto a fixed size (height, width) for the next model. The result are the final region proposals.
      * By default every image pixel is an anchor point, which results in a large number of regions. Hence, subsampling is used during training and testing.
    * (3) **Recognition Network**:
      * Takes a region (flattened to 1d vector) and projects it onto a vector of length 4096.
      * It uses fully connected layers to do that (ReLU, dropout).
      * Additionally, the network takes the 4096 vector and outputs new values for the region's position and confidence (for late fine tuning).
      * The 4096 vectors of all regions are combined to a matrix that is fed into the next component (RNN).
      * The intended sense of the this component seems to be to convert the "visual" features of each region to a more abstract, high-dimensional representation/description.
    * (4) **RNN Language Model**:
      * The take each 4096 vector and apply a fully connected layer + ReLU to it.
      * Then they feed it into an LSTM, followed by a START token.
      * The LSTM then generates word (as one hot vectors), which are fed back into the model for the next time step.
      * This is continued until the LSTM generates an END token.
  * Their full loss function has five components:
    * Binary logistic loss for the confidence values generated by the localization layer.
    * Binary logistic loss for the confidence values generated by the recognition layer.
    * Smooth L1 loss for the region dimensions generated by the localization layer.
    * Smooth L1 loss for the region dimensiosn generated by the recognition layer.
    * Cross-entropy at every time-step of the language model.
  * The whole model can be trained end-to-end.

* Results
  * They mostly use the Visual Genome dataset.
  * Their model finds lots of good regions in images.
  * Their model generates good captions for each region. (Only short captions with simple language however.)
  * The model seems to love colors. Like 30-50% of all captions contain a color. (Probably caused by the dataset?)
  * They compare to EdgeBoxes (other method to find regions in images). Their model seems to perform better.
  * Their model requires about 240ms per image (test time).
  * The generated regions and captions enable one to search for specific objects in images using text queries.


![Architecture](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/DenseCap__architecture.png?raw=true "Architecture.")

*Architecture of the whole model. It starts with the VGG-Net ("CNN"), followed by the localization layer, which generates region proposals. Then the recognition network converts the regions to abstract high-dimensional representations. Then the language model ("RNN") generates the caption.*


![Elephant image with dense captioning.](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/DenseCap__elephant.png?raw=true "Elephant image with dense captioning.")

![Airplane image with dense captioning.](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/DenseCap__airplane.png?raw=true "Airplane image with dense captioning.")


--------------------

### Rough chapter-wise notes

* (1) Introduction
  * They define four subtasks of visual scene understanding:
    * Classification: Assign a single label to a whole image
    * Captioning: Assign a sequence of words (description) to a whole image
    * Detection: Find objects in an image and assign a single label to each one
    * Dense Captioning: Find objects in an image and assign a sequence of words (description) to each one
  * They developed a model for dense captioning.
  * It has two three important components:
    * A convoltional network for scene understanding
    * A localization layer for region level predictions. It predicts regions of interest and then uses bilinear sampling to extract the activations of these regions.
    * A recurrent network as the language model
  * They evaluate the model on the large-scale Visual Genome dataset (94k images, 4.1M region captions).

* (3) Model
  * Model architecture
    * Convolutional Network
      * They use VGG-16, but remove the last pooling layer.
      * For an image of size W, H the output is 512xW/16xH/16.
      * That output is the input into the localization layer.
    * Fully Convolutional Localization Layer
      * Input to this layer: Activations from the convolutional network.
      * Output of this layer: Regions of interest, as fixed-sized representations.
        * For B Regions:
          * Coordinates of the bounding boxes (matrix of shape Bx4)
          * Confidence scores (vector of length B)
          * Features (matrix of shape BxCxXxY)
      * Method: Faster R-CNN (pooling replaced by bilinear interpolation)
      * This layer is fully differentiable.
      * The localization layer predicts boxes at anchor points.
      * At each anchor point it proposes `k` boxes using a small convolutional network. It assigns a confidence score and coordinates (center x, center y, height, width) to each proposal.
      * For an image with size 720x540 and k=12 the model would have to predict 17,280 boxes, hence subsampling is used.
      * During training they use minibatches with 256/2 positive and 256/2 negative region examples. A box counts as a positive example for a specific image if it has high overlap (intersection) with an annotated box for that image.
      * During test time they use greedy non-maximum suppression (NMS) (?) to subsample the 300 most confident boxes.
      * The region proposals have varying box sizes, but the output of the localization layer (which will be fed into the RNN) is ought to have fixed sizes.
      * So they project each proposed region onto a fixed sized region. They use bilinear sampling for that projection, which is differentiable.
    * Recognition network
      * Each region is flattened to a one-dimensional vector.
      * That vector is fed through 2 fully connected layers (unknown size, ReLU, dropout), ending with a 4096 neuron layer.
      * The confidence score and box coordinates are also adjusted by the network during that process (fine tuning).
    * RNN Language Model
      * Each region is translated to a sentence.
      * The region is fed into an LSTM (after a linear layer + ReLU), followed by a special START token.
      * The LSTM outputs multiple words as one-hot-vectors, where each vector has the length `V+1` (i.e. vocabulary size + END token).
      * Loss function is average crossentropy between output words and target words.
      * During test time, words are sampled until an END tag is generated.
  * Loss function
    * Their full loss function has five components:
      * Binary logistic loss for the confidence values generated by the localization layer.
      * Binary logistic loss for the confidence values generated by the recognition layer.
      * Smooth L1 loss for the region dimensions generated by the localization layer.
      * Smooth L1 loss for the region dimensiosn generated by the recognition layer.
      * Cross-entropy at every time-step of the language model.
    * The language model term has a weight of 1.0, all other components have a weight of 0.1.
  * Training an optimization
    * Initialization: CNN pretrained on ImageNet, all other weights from `N(0, 0.01)`.
    * SGD for the CNN (lr=?, momentum=0.9)
    * Adam everywhere else (lr=1e-6, beta1=0.9, beta2=0.99)
    * CNN is trained after epoch 1. CNN's first four layers are not trained.
    * Batch size is 1.
    * Image size is 720 on the longest side.
    * They use Torch.
    * 3 days of training time.

* (4) Experiments
  * They use the Visual Genome Dataset (94k images, 4.1M regions with captions)
  * Their total vocabulary size is 10,497 words. (Rare words in captions were replaced with `<UNK>`.)
  * They throw away annotations with too many words as well as images with too few/too many regions.
  * They merge heavily overlapping regions to single regions with multiple captions.
  * Dense Captioning
    * Dense captioning task: The model receives one image and produces a set of regions, each having a caption and a confidence score.
    * Evaluation metrics
      * Evaluation of the output is non-trivial.
      * They compare predicted regions with regions from the annotation that have high overlap (above a threshold).
      * They then compare the predicted caption with the captions having similar METEOR score (above a threshold).
      * Instead of setting one threshold for each comparison they use multiple thresholds. Then they calculate the Mean Average Precision using the various pairs of thresholds.
    * Baseline models
      * Sources of region proposals during test time:
        * GT: Ground truth boxes (i.e. found by humans).
        * EB: EdgeBox (completely separate and pretrained system).
        * RPN: Their localization and recognition networks trained separately on VG regions dataset (i.e. trained without the RNN language model).
      * Models:
        * Region RNN model: Apparently the recognition layer and the RNN language model, trained on predefined regions. (Where do these regions come from? VG training dataset?)
        * Full Image RNN model: Apparently the recognition layer and the RNN language model, trained on full images from MSCOCO instead of small regions.
        * FCLN on EB: Apparently the recognition layer and the RNN language model, trained on regions generated by EdgeBox (EB) (on VG dataset?).
        * FCLN: Apparently their full model (trained on VG dataset?).
    * Discrepancy between region and image level statistics
      * When evaluating the models only on METEOR (language "quality"), the *Region RNN model* consistently outperforms the *Full Image RNN model*.
      * That's probably because the *Full Image RNN model* was trained on captions of whole images, while the *Region RNN model* was trained on captions of small regions, which tend to be a bit different from full image captions.
    * RPN outperforms external region proposals
      * Generating region proposals via RPN basically always beats EB.
    * Our model outperforms individual region description
      * Their full jointly trained model (FCLN) achieves the best results.
      * The full jointly trained model performs significantly better than `RPN + Region RNN model` (i.e. separately trained region proposal and region captioning networks).
    * Qualitative results
      * Finds plenty of good regions and generates reasonable captions for them.
      * Sometimes finds the same region twice.
    * Runtime evaluation
      * 240ms on 720x600 image with 300 region proposals.
      * 166ms on 720x600 image with 100 region proposals.
      * Recognition of region proposals takes up most time.
      * Generating region proposals takes up the 2nd most time.
      * Generating captions for regions (RNN) takes almost no time.
  * Image Retrieval using Regions and Captions
    * They try to search for regions based on search queries.
    * They search by letting their FCLN network or EB generate 100 region proposals per network. Then they calculate per region the probability of generating the search query as the caption. They use that probability to rank the results.
    * They pick images from the VG dataset, then pick captions within those images as search query. Then they evaluate the ranking of those images for the respective search query.
    * The results show that the model can learn to rank objects, object parts, people and actions as expected/desired.
    * The method described can also be used to detect an arbitrary number of distinct classes in images (as opposed to the usual 10 to 1000 classes), because the classes are contained in the generated captions.

My objective in reading this paper was to gain another perspective on, and thus a more well-grounded view of, machine learning scoring functions for docking-based prediction of ligand/protein binding affinity. As quick background context, these models are useful because many therapeutic compounds act by binding to a target protein, and it can be valuable to prioritize doing wet lab testing on compounds that are predicted to have a stronger binding affinity. Docking systems work by predicting the pose in which a compound (or ligand) would bind to a protein, and then scoring prospective poses based on how likely such a pose would be to have high binding affinity. It's important to note that there are two predictive components in such a pipeline, and thus two sources of potential error: the searching over possible binding poses, done by physics-based systems, and scoring of the affinity of a given pose, assuming that were actually the correct one. Therefore, in the second kind of modeling, which this paper focuses on, you take in features *of a particular binding pose*, which includes information like which atoms of the compound are nearby to which atoms of the protein. 

The actual neural network structure used here was admittedly a bit underwhelming (though, to be fair, many of the ideas it seems to be gesturing at wouldn't be properly formalized until Graph Convolutional Networks came around). I'll describe the network mechanically first, and then offer some commentary on the design choices. 
https://i.imgur.com/w9wKS10.png

1. For each atom (a) in the compound, a set of neighborhood features are defined. The neighborhood is based on two hyperparameters, one for "how many atoms from the protein should be included," and one for "how many atoms from the compound should be included". In both cases, you start by adding the closest atom from either the compound or protein, and as hyperparameter values of each increase, you add in farther-away atoms. The neighborhood features here are (i) What are the types of the atoms? (ii) What are the partial charges of the atoms? (iii) How far are the atoms from the reference atom? (iiii) What amino acid within the protein do the protein atoms come? 
2. All of these features are turned into embeddings. Yes, all of them, even the ones (distance and charge) that are continuous values. Coming from a machine learning perspective, this is... pretty weird as a design choice. The authors straight-up discretize the distance values, and then use those as discrete values for the purpose of looking up embeddings. (So, you'd have one embedding vector for distance (0.25-0.5, and a different one for 0.0-0.25, say). 
3. The embeddings are concatenated together into a single "atom neighborhood vector" based on a predetermined ordering of the neighbor atoms and their property vectors. We now have one atom neighborhood vector for each atom in the compound. 
4. The authors then do what they call a convolution over the atom neighborhood vectors. But it doesn't act like a normal convolution in the sense of mixing information from nearby regions of atom space. It just is basically a fully connected layer that's applied to atom neighborhood vector separately, but with shared weights, so the same layer is applied to each neighborhood vector. They then do a feature-wise max pool across the layer-transformed version of neighborhood vectors, getting you one vector for the full compound 
5. This single vector is then put into a softmax, which predicts whether this ligand (in in this particular pose) will have strong binding with the protein 

Some thoughts on what's going on here. First, I really don't have a good explanation for why they'd have needed to embed a discretized version of the continuous variables, and since they don't do an ablation test of that design choice, it's hard to know if it mattered. Second, it's interesting to see, in their "convolution" (which I think is more accurately described as a Siamese Network, since it's only convolution-like insofar as there are shared weights), the beginning intuitions of what would become Graph Convolutions. The authors knew that they needed methods to aggregate information from arbitrary numbers of atoms, and also that they need should learn representations that have visibility onto neighborhoods of atoms, rather than single ones, but they do so in an entirely hand-engineered way: manually specifying a fixed neighborhood and pulling in information from all those neighbors equally, in a big concatenated vector. By contrast, when Graph Convolutions come along, they act by defining a "message-passing" function for features to aggregate across graph edges (here: molecular bonds or binaries on being "near enough" to another atom), which similarly allows information to be combined across neighborhoods. And, then, the 'convolution' is basically just a simple aggregation: necessary because there's no canonical ordering of elements within a graph, so you need an order-agnostic aggregation like a sum or max pool. 

The authors find that their method is able to improve on the hand-designed scoring functions within the docking programs. However, they also find (similar to another paper I read recently) that their model is able to do quite well without even considering structural relationships of the binding pose with the protein, which suggests the dataset (DUD - a dataset of 40 proteins with ~4K correctly binding ligands, and ~35K ligands paired with proteins they don't bind to) and problem given to the model is too easy.  It's also hard to tell how I should consider AUCs within this problem - it's one thing to be better than an existing method, but how much value do you get from a given unit of AUC improvement, when it comes to actually meaningfully reducing wetlab time used on testing compounds?  

I don't know that there's much to take away from this paper in terms of useful techniques, but it is interesting to see the evolution of ideas that would later be more cleanly formalized in other works.

As hair bundles move, viscous friction between stereocilia and the surrounding liquid poses a physical challenge to the ear’s high sensitivity and sharp frequency selectivity. This letter proposes that some of that energy is used for frequency-selective sound amplification, through fluid–structure interaction between the liquid within the hair bundle and the stereocilia.

A dynamic model is proposed to simulate hair bundles in a viscous environment, to see what large and small scale insights could be gained. Finite-element analysis, a submodel of hydrodynamic forces, stochastic simulation, and models of interferometric measurement all aimed to simulate both a hair bundle in natural conditions and what might be observed in an experiment involving it.

Forces between stereocilia are estimated, and the results suggest that the closeness of stereocilia reduces drag between them, supporting a sliding but not a squeezing mode. Tip links may couple mechanotransduction to this low-friction sliding mode, with motion between neighboring stereocilia of less than 1nm when the hair bundle moves the larger distance [O(10nm)]needed to stimulate its channels.

  * They analyze the effects of using Batch Normalization (BN) and Weight Normalization (WN) in GANs (classical algorithm, like DCGAN).
  * They introduce a new measure to rate the quality of the generated images over time.

### How
  * They use BN as it is usually defined.
  * They use WN with the following formulas:
    * Strict weight-normalized layer:
      * ![Strict WN layer](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/On_the_Effects_of_BN_and_WN_in_GANs__strict_wn.jpg?raw=true "Strict WN layer")
    * Affine weight-normalized layer:
      * ![Affine WN layer](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/On_the_Effects_of_BN_and_WN_in_GANs__affine_wn.jpg?raw=true "Affine WN layer")
    * As activation units they use Translated ReLUs (aka "threshold functions"):
      * ![TReLU](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/On_the_Effects_of_BN_and_WN_in_GANs__trelu.jpg?raw=true "TReLU")
      * `alpha` is a learned parameter.
      * TReLUs play better with their WN layers than normal ReLUs.
  * Reconstruction measure
    * To evaluate the quality of the generated images during training, they introduce a new measure.
    * The measure is based on a L2-Norm (MSE) between (1) a real image and (2) an image created by the generator that is as similar as possible to the real image.
    * They generate (2) by starting `G(z)` with a noise vector `z` that is filled with zeros. The desired output is the real image. They compute a MSE between the generated and real image and backpropagate the result. Then they use the generated gradient to update `z`, while leaving the parameters of `G` unaltered. They repeat this for a defined number of steps.
    * Note that the above described method is fairly time-consuming, so they don't do it often.
  * Networks
    * Their networks are fairly standard.
    * Generator: Starts at 1024 filters, goes down to 64 (then 3 for the output). Upsampling via fractionally strided convs.
    * Discriminator: Starts at 64 filters, goes to 1024 (then 1 for the output). Downsampling via strided convolutions.
    * They test three variations of these networks:
      * Vanilla: No normalization. PReLUs in both G and D.
      * BN: BN in G and D, but not in the last layers and not in the first layer of D. PReLUs in both G and D.
      * WN: Strict weight-normalized layers in G and D, except for the last layers, which are affine weight-normalized layers. TPReLUs (Translated PReLUs) in both G and D.
  * Other
    * They train with RMSProp and batch size 32.

### Results
  * Their WN formulation trains stable, provided the learning rate is set to 0.0002 or lower.
  * They argue, that their achieved stability is similar to the one in WGAN.
  * BN had significant swings in quality.
  * Vanilla collapsed sooner or later.
  * Both BN and Vanilla reached an optimal point shortly after the start of the training. After that, the quality of the generated images only worsened.
  * Plot of their quality measure:
    * ![Losses over time](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/On_the_Effects_of_BN_and_WN_in_GANs__losses_over_time.jpg?raw=true "Losses over time")
  * Their quality measure is based on reconstruction of input images. The below image shows examples for that reconstruction (each person: original image, vanilla reconstruction, BN rec., WN rec.).
    * ![Reconstructions](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/On_the_Effects_of_BN_and_WN_in_GANs__reconstructions.jpg?raw=true "Reconstructions")
  * Examples generated by their WN network:
    * ![WN Examples](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/On_the_Effects_of_BN_and_WN_in_GANs__wn_examples.jpg?raw=true "WN Examples")

Early study showing effect of an antibiotic on gut microbiome and the majority of community recovered within 4 wks