Similarity-preserving binary code obtained from quantization can efficiently accelerate the retrieval of large-scale image collections. This paper presents iterative quantization (ITQ) that iteratively minimizes quantization error by rotating the data before quantization. ITQ can be couple with any projection of data onto an orthogonal basis. Experiment results show outstanding performance on the retrieval of large-scale image collections especially when the length of binary code is short.
Technical details and results

Idea of ITQ

Figure 1 illustrate the idea of rotating data before quantization to reduce quantization error.

![](http://2.bp.blogspot.com/-riBFQ71aWKk/VRI4Aprwx5I/AAAAAAAAAxw/ggFrB14UGlw/s1600/toy.png)

Figure 1. Toy illustration of the proposed ITQ method
The data points (blue points) are quantized to the closest vertex of the binary cube. By rotating the data points to that shown in Figure 1 (c), the quantization error is reduced and the partitioning respects the structure of the cluster.

ITQ in unsupervised code learning

Given a set of n data points and let each data point be d dimension, the data matrix is denoted by

$X \in \mathbb{R}^{n \times d}$

The binary code matrix can be computed by 

$B = sgn(XW)$ 

$W \in \mathbb{R}^{d \times c}$

where W denotes the projection matrix computed by PCA in this case.

Let R denote any c x c orthogonal matrix. The quantization error after projection and rotation of the data matrix is denotes by

$Q(B,R) = ||B - VR||^{2}_F$

$V = XW$

The ITQ method minimizes the quantization error by seeking optimal R. Because of the quantization operator, the quantization error is not a smooth function and direct minimization of the quantization error is impractical. This paper propose optimizing R and B alternately like k-mean algorithm.  When R is fixed, B is computed by


When B is fixed, R is computed by

$B = sgn(VR)$

where S and "S hat" are the left-singular vector and right-singular vector of the matrix 
         
$B^TV$

Figure 2 shows the quantization error for learning a 32-bit ITQ code on the CIFAR dataset.

![](http://4.bp.blogspot.com/-F6wZGT5fl3E/VRJW5U2VvFI/AAAAAAAAAzg/aEt4DIDpV6E/s1600/quantization%2Berror.png)

Results

"PCA-ITQ" in the legend of the figures denote the proposed method.


![](http://3.bp.blogspot.com/-WbVup__p9CQ/VRJTBh7NKpI/AAAAAAAAAx8/DCUvfo5Ijuw/s1600/result1.png)

![](http://2.bp.blogspot.com/-dj6F3pd5WiM/VRJTBvF6G5I/AAAAAAAAAyA/WyW7He4TJOw/s1600/result2.png)

This paper is to reduce gender bias in the captioning model. Concretely, traditional captioning models tend to rely on contextual cues, so they usually predict incorrect captions for an image that contains people.

To reduce gender bias, they introduced a new $Equalizer$ model that contains two losses:

(1) Appearance Confusion Loss: When it is hard to tell if there is a man or a woman in the image, the model should provide a fair probability of predicting a man or a woman.

To define that loss, first, they define a confusion function, which indicates how likely a next predicted word belongs to a set of woman words or a set of man words.

https://i.imgur.com/oI6xswy.png
Where, $w~_{t}$ is the next predicted word, $G_{w}$ is the set of woman words, $G_{m}$ is the set of man words.

And the Loss is defined as the normal cross-entropy loss multiplied by the Confusion function.
https://i.imgur.com/kLpROse.png


(2) Confident Loss: When it is easy to recognize a man or a woman in an image, this loss encourages the model to predict gender words correctly.

In this loss, they also defined in-confidence functions, there are two in-confidence functions, the first one is the in-confidence function for man words, and the second one is for woman words. These two functions are the same.

https://i.imgur.com/4stFjac.png

This function tells that if the model is confident when predicting a gender (ex. woman), then the value of the in-confidence function for woman words should be low.

Then, the confidence loss function is as follows:
https://i.imgur.com/1pRgDir.png

They create an LSTM neural language model that 1) has better handling of numerical values, and 2) is conditioned on a knowledge base.

https://i.imgur.com/Rb6V1Hy.png

First the the numerical value each token is given as an additional signal to the network at each time step. While we normally represent token “25” as a normal word embedding, we now also have an extra feature with numerical value float(25). Second, they condition the language model on text in a knowledge base. All the information in the KB is converted to a string, passed through an LSTM and then used to condition the main LM.

They evaluate on a dataset of 16,003 clinical records which come paired with small KB tuples of 20 possible attributes. The numerical grounding helps quite a bit, and the best results are obtained when the KB conditioning is also added.

## General Framework
Only access to a finite set of **ranked demonstrations**. The demonstrations only contains **observations** and **do not need to be optimal** but must be (approximately) ranked from worst to best. 
The **reward learning part is off-line** but not the policy learning part (requires interactions with the environment).

In a nutshell: learns a reward models that looks at observations. The reward model is trained to predict if a demonstration's ranking is greater than another one's. Then, once the reward model is learned, one simply uses RL to learn a policy. This latter outperform the demonstrations' performance.

## Motivations
Current IRL methods cannot outperform the demonstrations because they seek a reward function that makes the demonstrator optimal and thus do not infer the underlying intentions of the demonstrator that may have been poorly executed. 
In practice, high quality demonstrations may be difficult to provide and it is often easier to provide demonstrations with a ranking of their relative performance (desirableness).

## Trajectory-ranked Reward EXtrapolation (T-REX)
![](https://i.imgur.com/cuL8ZFJ.png =400x)

Uses ranked demonstrations to extrapolate a user's underlying intent beyond the best demonstrations by learning a reward that assigns greater return to higher-ranked trajectories. While standard IRL seeks a reward that **justifies** the demonstrations, T-REX tries learns a reward that **explains** the ranking over demonstrations. 

![](https://i.imgur.com/4IQ13TC.png =500x)


Having rankings over demonstrations may remove the reward ambiguity problem (always 0 reward cannot explain the ranking) as well as provide some "data-augmentation" since from a few ranked demonstrations you can define many pair-wise comparisons. Additionally, suboptimal demonstrations may provide more diverse data by exploring a larger area of the state space (but may miss the parts relevant to solving the task...)

## Tricks and Tips
Authors used ground truth reward to rank trajectories, but they also show that approximate ranking does not hurt the performance much. 

To avoid overfitting they used an ensemble of 5 Neural Networks to predict the reward.

For episodic tasks, they compare subtrajectories that correspond to similar timestep (better trajectory is a bit later in the episode than the one it is compared against so that reward increases as the episode progresses). 

At RL training time, the learned reward goes through a sigmoid to avoid large changes in the reward scale across time-steps.

## Results

![](https://i.imgur.com/7ysYZKd.png)

![](https://i.imgur.com/CHO9aVT.png)

![](https://i.imgur.com/OzVD9sf.png =600x)

Results are quite positive and performance can be good even when the learned reward is not really correlated with the ground truth (cf. HalfCheetah). 

They also show that T-REX is robust to different ranking-noises: random-swapping of pair-wise ranking,  ranking by humans that only have access to a description of the task and not the ground truth reward. **They also automatically rank the demonstrations using the number of learning steps of a learning expert: therefore T-REX could be used as an intrinsic reward alongside the ground-truth to accelerate training.** 

![](https://i.imgur.com/IfOeLY6.png =500x)

**Limitations**
They do not show than T-REX can match an optimal expert, maybe ranking demonstrations hurt when all the demos are close to optimality? 

Modification of the 2-level hierarchical softmax for better efficiency. An equation of computational complexity is used to find the optimal number of words in each class. In addition, the most common words are considered on the same level as other classes.

https://i.imgur.com/dbKS3gh.png