Instance Normalization: The Missing Ingredient for Fast Stylization on ShortScience.org

arxiv.org
arxiv-vanity.com
scholar.google.com

Instance Normalization: The Missing Ingredient for Fast Stylization
Dmitry Ulyanov and Andrea Vedaldi and Victor Lempitsky
arXiv e-Print archive - 2016 via Local arXiv
Keywords: cs.CV
more

Summaries/Notes 2

[link] Summary by Alexander Jung 7 years ago

  * Style transfer between images works - in its original form - by iteratively making changes to a content image, so that its style matches more and more the style of a chosen style image.
  * That iterative process is very slow.
  * Alternatively, one can train a single feed-forward generator network to apply a style in one forward pass. The network is trained on a dataset of input images and their stylized versions (stylized versions can be generated using the iterative approach).
  * So far, these generator networks were much faster than the iterative approach, but their quality was lower.
  * They describe a simple change to these generator networks to increase the image quality (up to the same level as the iterative approach).

### How
  * In the generator networks, they simply replace all batch normalization layers with instance normalization layers.
  * Batch normalization normalizes using the information from the whole batch, while instance normalization normalizes each feature map on its own.
  * Equations
    * Let `H` = Height, `W` = Width, `T` = Batch size
    * Batch Normalization:
      * ![Batch Normalization Equations](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/Instance_Normalization_The_Missing_Ingredient_for_Fast_Stylization__batch_normalization.jpg?raw=true "Batch Normalization Equations")
    * Instance Normalization
      * ![Instance Normalization Equations](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/Instance_Normalization_The_Missing_Ingredient_for_Fast_Stylization__instance_normalization.jpg?raw=true "Instance Normalization Equations")
  * They apply instance normalization at test time too (identically).

### Results
  * Same image quality as iterative approach (at a fraction of the runtime).
  * One content image with two different styles using their approach:
    * ![Example](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/Instance_Normalization_The_Missing_Ingredient_for_Fast_Stylization__example.jpg?raw=true "Example")

Style transfer between images works - in its original form - by iteratively making changes to a content image, so that its style matches more and more the style of a chosen style image.
That iterative process is very slow.
Alternatively, one can train a single feed-forward generator network to apply a style in one forward pass. The network is trained on a dataset of input images and their stylized versions (stylized versions can be generated using the iterative approach).
So far, these generator networks were much faster than the iterative approach, but their quality was lower.
They describe a simple change to these generator networks to increase the image quality (up to the same level as the iterative approach).

How

In the generator networks, they simply replace all batch normalization layers with instance normalization layers.
Batch normalization normalizes using the information from the whole batch, while instance normalization normalizes each feature map on its own.
Equations
- Let H = Height, W = Width, T = Batch size
- Batch Normalization:
- Instance Normalization
They apply instance normalization at test time too (identically).

Results

Same image quality as iterative approach (at a fraction of the runtime).
One content image with two different styles using their approach:

Your comment:

[link] Summary by David Stutz 5 years ago

In the context of stylization, Ulyanov et al. propose to use instance normalization instead of batch normalization. In detail, instance normalization does not compute the mean and standard deviation used for normalization over the current mini-batch in training. Instead, these statistics are computed per instance individually. This also has the benefit of having the same training and test procedure, meaning that normalization is the same in both cases – in contrast to batch normalization.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).

In the context of stylization, Ulyanov et al. propose to use instance normalization instead of batch normalization. In detail, instance normalization does not compute the mean and standard deviation used for normalization over the current mini-batch in training. Instead, these statistics are computed per instance individually. This also has the benefit of having the same training and test procedure, meaning that normalization is the same in both cases – in contrast to batch normalization.

Also find this summary at davidstutz.de.

Your comment:

Write your summary here (You can use $\LaTeX$ and markdown syntax):

Anon Private