####  Mask R-CNN framework for instance segmentation
### Goal:
* classify individual objects
*  localize each using a bounding box, 
* semantic segmentation
https://i.imgur.com/XfBRa5O.png
*  classify each pixel into a fixed set of categories without differentiating object instances.
* extends Faster R-CNN  by adding a branch for predicting segmentation masks on each Region of Interest (RoI), in parallel with the existing branch for classification and bounding box regression.
*  FCN applied to each RoI, predicting a segmentation mask in a pixel-to-pixel manner
1. RoIAlign:
 * Used to fix the misalignment that faithfully preserves exact spatial locations
 * improves mask accuracy by relative 10% to 50%, fast speed
2. Decouple mask and class prediction:
 * predict a binary mask for each class independently, without competition among classes

 History:
* RCNN: The Region-based CNN (R-CNN) approach to bounding-box object detection
* Fast RCNN:  Speeding up and Simplifying R-CNN
 *  RoI (Region of Interest) Pooling
 *  jointly train the CNN, classifier, and bounding box regressor in a single model
*  Faster R-CNN - Speeding Up Region Proposal
 * reuse the same CNN results for region proposals instead of running a separate selective search algorithm it can be done by Region Proposal Network
 * only one CNN needs to be trained 

 Related Work
*  Instance Segmentation: “fully convolutional instance segmentation” (FCIS)
* Faster R-CNN: * Region Proposal Network (RPN), proposes candidate object bounding boxes
* Fast R-CNN [12], extracts features using RoIPool from each candidate box and performs classification and bounding-box regression
*  Mask R-CNN: Mask R-CNN adopts the same two-stage of Faster RCNN And has third stage i.e binary mask for each RoI
* Mask Representation: pixel to pixel representation of image done by RoIAlign layer (7X7)
#### Network Architecture
*  convolutional backbone architecture used for feature extraction over an entire image (ResNet-50-C4, FPN)
*  network head for bounding-box recognition (classification and regression) and mask prediction
https://i.imgur.com/pUvKdmx.png
#### Training:
* Images resized:800 pixel                                     
* mini-batch : 2 images per GPU
* N : 64
* train: on 8 GPUs for 160k iterations
* learning : 0.02
*  train images: 80K
* val images: 35K
* minival:5K
https://i.imgur.com/6ZLpewi.png
https://i.imgur.com/5o3um0Y.png

This paper attempts to open up the black box of neural machine translation models and inspect what the representations look like, specifically with respect to morphology. The technique they use is to train word-based and character-based seq2seq-style models on multiple source-target language pairs, of varying morphological complexity, and then ignore the target side to focus on the representations learned about the source language. Once they have an encoder trained to generate these representations, they attempt to use the encoder to create feature representations for external tasks that directly evaluate for morphology and part of speech information. (Contrast this with methods that may, for example, try to inspect activation patterns of individual neurons in a trained model.)

The first experiment shows that representations learned from character-based models are superior for POS tagging in the source language. The gap is bigger for morphologically rich languages like Arabic. The same result holds for morphological tagging. For infrequent words the gap is especially large -- the system can memorize morphological information for frequent words. They also show that the increases in accuracy are due to getting  prevoiusly unseen words correct (both for POS and morph prediction) and that the biggest increase in accuracy is in predicting plural and determined noun categories. Next, they show that in a deeper network, the middle layer (of 3) has the best representations for predicting pos/morph information. The authors suggest the higher layers are more focused on semantics or other higher abstractions.

Overall, this work empirically confirms some conventional wisdom, that character representations are better for unseen words because of their ability to represent morphology.

The authors explore the properties of "Deep Averaging Networks" on text classification problems, specifically sentiment and question answering tasks. DANs extend neural bag of words models, starting with a document representation that is the average of the word embeddings in that document, but extending to multiple feed-forward layers. The authors argue that these models are much simpler and faster to train than syntax and composition-based RNNs, while obtaining similar performance. Since this paper is actually arguing for simpler models, there is little technically here to understand, so the real contribution of the paper are the interesting experiments exploring how the DANs represent various phenomena. They show that differences between graded sentiment words (awesome, cool, ok, underwhelming, the worst) are magnified as layers are added. This shows the benefit of depth relative to a neural bag of words. Then they compare against RNNs with examples containing negation and contrastive conjunctions (e.g., but), which are traditionally modeled syntactically. They show that existing methods that we think can represent syntax/composition in fact are not strong enough. Something like "not bad" fully exposes the DAN -- it doubles the negation. But while the RNN-based models can learn not to simply double the negation, they are not powerful enough to reverse the polarity and get the example correct.

Finally, the authors introduce one novel mechanism for improving training, "word dropout." Similar to standard dropout, they randomly sample a subset of words at the input layer that are not used as part of the document representation. This gives the network multiple looks at each example with part of its feature space removed. Another way to think of this is data augmentation where new training instances are created by sampling feature vectors from existing data points with some features missing.

This paper gets a face image and changes its pose or rotates it (to any desired pose) by passing the target pose as the input to the model. 

https://i.imgur.com/AGNOag5.png

They use a GAN (named DR-GAN) for face rotation. The gan has an encoder and a decoder. The encoder takes the image and gets a high-level feature representation. The decoder gets high-level features, the target pose, and some noise to generate the output image with rotated face. 

The generated image is then passed to a discriminator where it says whether the image is real or fake. The disc also has two other outputs: 1- it estimates the pose of the generated image, 2) it estimated the identity of the person.

no direct loss is applied to the generator, it is trained by the gradient that it gets through discriminator to minimize the three objects: 1- gan loss (to fool disc) 2-pose estimation 3- identity estimation.

They use two tricks to improve the model:
1- using the same parameters for encoder of generator (gen-enc) and the discriminator (they observe this helps better identity recognition)
2- passing two images to gen-enc and interpolating between their high-level features (gen-enc output) and then applying two costs on it: 1) gan loss 2) pose loss. These losses are applied through disc, similar to above.
The first trick improves gen-enc and second trick improves gen-dec, both help on identification.

Their model can also leverage multiple image of the same identity if the dataset provides that to get better latent representation in gen-enc for a given identity.

https://i.imgur.com/23Tckqc.png

These are some samples on face frontalization:
https://i.imgur.com/zmCODXe.png

and these are some samples on interpolating different features in latent space: (sub-fig a) interpolating f(x) between the latent space of two images, (sub-fig b) interpolating pose (c), (sub-fig c) interpolating noise:
https://i.imgur.com/KlkVyp9.png

I find these positive aspects about the paper:
1) face rotation is applied on the images in the wild, 2) It is not required to have paired data. 3) multiple source images of the same identity can be used if provided, 4) identity and pose are used smartly in the discriminator to guide the generator, 5) model can specify the target pose (it is not only face-frontalization).

Negative aspects:
1) face has many artifacts, similar to artifacts of some other gan models. 
2) The identity is not well-preserved and the faces seem sometime distorted compared to the original person.

 They show the models performance on identity recognition and face rotation and demonstrate compelling results.

This paper introduces triangle-GAN ($\triangle$-GAN) that aims at cross-domain joint distribution matching: The model is shown below

https://i.imgur.com/boIDOMu.png

Having two domains of data $x$ and $y$, there are two generators:

1- $G_x(y)$ which takes $y$ and generates $\tilde{x}$

2- $G_y(x)$ which takes $x$ and generates $\tilde{y}$

There are two discriminators in the model:

1- $D_1 (x,y)$ a discriminator that distinguishes between $(x, y)$ and either of $(x, \tilde{y})$ or $(\tilde{x}, y)$.

2- $D_2 (x,y)$ a discriminator that distinguishes between $(x, \tilde{y})$ and $(\tilde{x}, y)$.

The second discriminator is ALI and can be used on un-paired sets of data.
The first discriminator is equivalent to a conditional discriminator where the true paired data $(x, y)$ is compared to either $(x, \tilde{y})$ or $(\tilde{x}, y)$, where one element in the pair is sampled. This discriminator needs paired $(x, y)$ data for training. 

This model can be used for semi-supervised settings, where a small set of paired data is provided. In this paper it is used for:

-  semi-supervised image classification, where a small subset of CIFAR10 is labelled. $x$ and $y$ are images and class labels here.

- image to image translation on edge2shoes dataset, where only a subset of dataset is paired.

- attribute conditional image generation where $x$ and $y$ domains are image and attributes. CelebA and COCO datasets are used here. In one experiment test-set images are projected to attributes and then given those attributes new images are generated:

On celebA:
https://i.imgur.com/EX5tDZ0.png

On COCO:
https://i.imgur.com/GRpvjGx.png

In another experiment some attributes are chosen (as samples shown below in the first row with different noise) and then another feature is added (using the same noise) to generate the samples in the second row:
 https://i.imgur.com/KeHL8Ye.png

The triangle gan demonstrates improved performance compared to triple gan in the experiments shown in the paper. It has been also compared with Disco gan (a model that can be trained on un-paired data) and shows improved performance when some percentage of paired data is provided. In an experiment they pair each MNIST digit with its transposed (as $x$, $y$ pairs). Disco-GAN cannot learn correct mapping between them, while triangle-GAN can learn correct mapping since it leverages paired data.

https://i.imgur.com/Vz9Zfhu.png

In general this model is a useful approach for semi-supervised cross-domain matching and can leverage un-paired data (using ALI) as well as paired data (using conditional discriminator).