This paper is about the analysis of CNNs. It seems to be extremely similar to what Zeiler & Fergus did. I can't see the contribution. Only cited 7 times, although it is from December 2014 -> I suggest to read the Zeiler & Fergus paper instead.

## Related

* 2013, Zeiler & Fergus: [Visualizing and Understanding Convolutional Networks ](http://www.shortscience.org/paper?bibtexKey=journals/corr/ZeilerF13#martinthoma)


## Errors

* " Section ?? provides"

**Introduction**

Object segmentation methods are often produced an imprecise result as objects frequently not always agree with homogeneous regions. Thus this paper provides segmentation of images and videos into homogeneous region in color and texture feature cues called JSEG. Assumptions for the environments used are:
* Image contains homogeneous color and texture regions
* Color is quantized 
* There are distinct colors in neighboring regions

**Related work**
* Present work in image segmentation requires texture model parameter approximation that often needs the homogeneous region to produce good derivation.
* There is an existing technique for segmentation using motion. However, this method is not dependable in noisy data, insufficiency in affine transformation for close-up motion, and errors in the presence of occlusion.

**Approach**
* Method consists of two stages that are color quantization and image segmentation spatially.
* Colors quantized into several appointed classes to distinguish regions by weighting pixels individually using the Lloyd algorithm.
* Result of quantized colors are assigned labels. These labels or class-map also define the composition of textures.

https://i.imgur.com/2AlFD7Z.png

* Class maps are labeled by three symbols that are: *, + and o.
* Symbols indicate positions where line of segmentations need to be drawn, for example a class map with half of the left region that contains + symbol and the other half contains uniform distribution of \* and o can be segmented into two regions: one with + symbol and the other is a collection of * and o symbols. 
* Variance from the class map is computed and the value J is computed using the variance of both the same class and different class.
* Value of J is small when image contains a uniform distribution of color classes and large otherwise.
* The definition of J initiate an assumption of states of the class labels and specify information where line of segmentation could be drawn.
* In the segmented region, the mean of J is calculated and the minimized value of J mean is a criterion to segment image given region numbers.
* In a good segmentation, the value of J means is small as the number of colors that are uniformly distributed is small in the divided region.
* Algorithm of spatial segmentation contains several stages: calculate J values in each region, growing regions by using seeds, and merging regions once scale has exceeded the threshold. Described as follows:

https://i.imgur.com/uZzgrOO.png

* Local J values applied as it has the property to indicate whether an area is in a region or near boundary of a region
* Windows are used to detect region sizes. Large windows classify boundary of texture cues and small windows classify color or intensity edges.
* Multiple sizes of windows are utilized with the circular shape of diameter 9 pixels for the smallest window.
* To grow region using seeds, seeds are set first by finding mean and standard deviation of local J values, setting threshold by adding mean and standard deviation multiplied by preset values, and seed is fixed once it consists of an area larger than predetermined values for each window pixels.
* Seeds is then grown by: removing empty classifications from fixed seeds before, averaging local J values in unsegmented region where if a region is near to only one seed, it is classified as the corresponding seed's region, calculating J values for smaller region, averaging more local J values in the respected remaining unsegmented region and growing region at the smallest scale.
* Similarities in color built the merging of a region. As colors have quantized in histogram bins, distance is calculated between two histograms using Euclidean distance in CIE LUV color space.
* To merge regions, distances are enlisted and pairs with small distance are joined. Next, a new feature vector is computed and process iterates for merging and generating new feature vectors until the maximum threshold is attained.
* JSEG can be implemented in video data by using movements of objects as indirect constraints for tracking and segmentation. The assumption used for implementation is that videos have been parted into shots and shots are continuous scenes.
* Video is decomposed in the spatiotemporal domain and grouped to be segmented for consecutive frames.
* In this paper, 1000 frames are grouped and quantized for its color to generate class-maps
* In frames that have color textures that are close to each other, they are counted as one object.
* After seeds are fixed from the frames, tracking is done by assigning initial seed, overlapped seeds are considered to be one, iteratively checking overlapped seeds, and assign time duration for objects
* To reduce the number of false merges, The value of J track is computed by calculating mean and standard deviance between two frames. When the region is static, the J value will be small and large otherwise
* The running time for JSEG in video segmentation is equal to the application of image segmentation by grouped frames
* Overall, parameters that need to be adjusted in using JSEG are color quantization threshold, number of image scales and object duration for video segmentation
* In video segmentation, in average frames can be grouped as 10 to 15 frames 

**Paper contributions**
* Paper provides a new method called JSEG to segment objects by spatial segmentation and color quantization in images and videos unsupervised.
* Criteria to evaluate a good segmentation in an image is proposed.
* Final segmentation is obtained by dividing region based on seed areas from J-image.

TLDR; The authors apply adversarial training on labeld data and virtual adversarial training on unlabeled data to the embeddings in text classification tasks. Their models, which are straightforward LSTM architectures, either match or surpass the current state of the art on several text classification tasks. The authors also show that the embeddings learned using adversarial training tend to be tuned better to the corresponding classification task.

#### Key Points

- In Image classification we can apply adversarial training directly to the inputs. In Text classification the inputs are discrete and we cannot make small perturbations, but we can instead apply adversarial training to embeddings.
- Trick: To prevent the model from making perturbations irrelevant by learning embeddings with large norms: Use normalized embeddings.
- Adversarial Training (on labeled examples)
  - At each step of training, identify the "worst" (in terms of cost) perturbation `r_adv` to the embeddings within a given constant epsilon, which a hyperparameter. Train on that. In practice `r_adv` is estimated using a linear approximation.
  - Add a `L_adv` adversarial loss term to the cost function.
- Virtual Adversarial Training (on unlabeled examples)
  - Minimize the KL divergence between the outputs of the model given the regular and perturbed example as inputs.
  - Add `L_vad` loss to the cost function.
- Common misconception: Adversarial training is equivalent to training on noisy examples, but it actually is a stronger regularizer because it explicitly increases cost.
- Model Architectures:
  - (1) Unidirectional LSTM with prediction made at the last step
  - (2) Bidirectional LSTM with predictions based on concatenated last outputs
- Experiments/Results
  - Pre-Training: For all experiments a 1-layer LSTM language model is pre-trained on all labeled and unlabeled examples and used to initialize the classification LSTM.
  - Baseline Model: Only embedding dropout and pretraining
  - IMDB: raining curves show that adversarial training acts as a good regularizer and prevents overfitting. VAT matches state of the art using a unidirectional LSTM only.
  - IMDB embeddings: Baseline model places "good" close to "bad" in embedding space. Adv. training ensures that small perturbations in embeddings don't change the sentiment classification result so these two words become properly separated.
  - Amazon Reviews and RCV1: Adv. + Vadv. achieve state of the art.
  - Rotten Tomatoes: Adv. + Vadv. achieve state of the art. Because unlabeled data overwhelms labeled data vadv. training results in decrease of performance.
  - DBPedia: Even the baseline outperforms state of the art (better optimizer?), adversarial training improves on that.

### Thoughts

- I think this is a very well-written paper with impressive results. The only thing that's lacking is a bit of consistency. Sometimes pure virtual adversarial training wins, and sometimes adversarial + virtual adversarial wins. Sometimes bi-LSTMs make things worse, sometimes better. What is the story behind that? Do we really need to try all combinations to figure out what works for a given dataset?
- Not a big deal, but a few bi-LSTM experiments seem to be missing. This just always makes me if they are "missing for a reason" or not ;)
- There are quite a few differences in hyperparameters and batch sizes between datasets. I wonder why. Is this to stay consistent with the models they compare to? Were these parameters optimized on a validation set (the authors say only dropout and epsilon were optimized)?
- If Adversarial Training is a stronger regularizer than random permutations I wonder if we still need dropout in the embeddings. Shouldn't adversarial training take care of that?

multi layer RNN in which first layer is LSTM, following layers $l$ have $t$,$c$ gates that control whether the state of the layer is carried from previous state or transferred previous layer:
$s_l^{[t]} = h_l^{[t]} \cdot t_l^{[t]} + s_{l-1}^{[t]} \cdot c
_l^{[t]}$

__Edit__: The paper has a newer version at https://openreview.net/pdf?id=B1YfAfcgl, where I also copied this review over with some changes.

__Note__: An earlier version of the review (almost identical to the present one) for an earlier version of the paper (available on arXiV) can be found here: http://www.shortscience.org/paper?bibtexKey=journals/corr/1611.01838#csaba
The only change concerns relation to previous work.

__Problem__: The problem considered is to derive an improved version of SGD for training neural networks (or minimize empirical loss) by modifying the loss optimized to that the solution found is more likely to end up in the vicinity of a minimum where the loss changes slowly ("flat minima" as in the paper of Hochreiter and Schmidhuber from 1997). 

__Motivation__: It is hypothetised that flat minima "generalize" better. 

__Algorithmic approach__: Let $f$ be the (empirical) loss to be minimized. Modify this to $$\overline{f}(x) = \rho f(x) - \log \int \exp(-\frac{\gamma}{2}\||z\||^2-f(x+z))\,dz$$ with some $\rho,\gamma>0$ tunable parameters. For $\||z\||^2\gg \frac{1}{\gamma}$, the term $\exp(-\frac{\gamma}{2}\||z\||^2)$ becomes very small, so effectively the second term is close to a constant times the integral of $f$ over a ball centered at $x$ and having a radius of $\propto \gamma^{-1/2}$. This is a smoothened version of $f$, hence one expects that by making this term more important then the first term, a procedure minimizing $\bar f$ will be more likely to end up at a flat minima of $f$. Since the gradient is somewhat complicated, an MCMC algorithm is proposed ("stochastic gradient Langevin dynamics" from Welling and Teh, 2011).

__Results__: There is a theoretical result that quantifies the increased smoothness of $\overline{f}$, which is connected to stability and ultimately to generalization through citing a result of Hardt et al. (2015). Empirical results show better validation error on two datasets: MNIST and CIFAR-10 (the respective networks are LeNet and All-CNN-C). The improvement is in terms of reaching the same validation error as with an "original SGD" but with fewer "epochs".

__Soundness, significance__: The proof of the __theoretical result__ relies on an arbitrary assumption that there exists some $c>0$ such that no eigenvalue of the hessian of $f$ lies in the set $[-2\gamma-c,c]$ (the reason for the assumption is because otherwise a uniform improvement cannot be shown). For $\rho=0$ the improvement of the smoothness (first and second order) is a factor of $1/(1+c/\gamma)$. The proof uses Laplace's method and is more a sketch than a rigorous proof (error terms are dropped; it would be good to make this clear in the statement of the result).

In the experiments the modified procedure did not consistently reach a smaller validation error. The authors did not present running times, hence it is unclear whether the procedure's increased computation cost is offset by the faster convergence. 

__Evaluation__: It is puzzling why a simpler smoothing, e.g., $$\overline{f}(x) = \int f(x+z) g(z) dz$$ (with $g$ being the density of a centered probability distribution) is not considered. The authors note that something "like this" may be infeasible in "deep neural networks" (the note is somewhat vague). However, under mild conditions, $\frac{\partial}{\partial x} \overline{f}(x) = \int \frac{\partial}{\partial x} f(x+z) g(z) dz$, hence, for $Z\sim g(\cdot)$, $\frac{\partial}{\partial x} f(x+Z)$ is an unbiased estimate of $\frac{\partial}{\partial x} \overline{f}(x)$, whose calculation is as cheap as that of vanilla SGD. Also, how much the smoothness of $f$ changes when using this approach is quite well understood.

__Related work__: Hochreiter and Schmidhuber argued a long time ago for the benefits of "flat minima" in an  identically titled Neural computation paper that appeared in 1997. This reviewer may not agree with the arguments in this paper, but the paper is highly relevant and should be cited. It is also strange that the specific modification that appears in this paper was proposed by others (Baldassi et al.), whom this paper also cites, but without giving these authors credit for introducing local entropy as a smoothing technique.