Duesterwald et al. study the influence of hyperparameters on adversarial training and its robustness as well as accuracy. As shown in Figure 1, the chosen parameters, the ratio of adversarial examples per batch and the allowed perturbation $\epsilon$, allow to control the trade-off between adversarial robustness and accuracy. Even for larger $\epsilon$, at least on MNIST and SVHN, using only few adversarial examples per batch increases robustness significantly while only incurring a small loss in accuracy.

https://i.imgur.com/nMZNpFB.jpg
Figure 1: Robustness (red) and accuracy (blue) depending on the two hyperparameters $\epsilon$ and ratio of adversarial examples per batch. Robustness is measured in adversarial accuracy.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).

TLDR; The authors propose a new normalization scheme called "Layer Normalization" that works especially well for recurrent networks. Layer Normalization is similar to Batch Normalization, but only depends on a single training case. As such, it's well suited for variable length sequences or small batches. In Layer Normalization each hidden unit shares the same normalization term. The authors show through experiments that Layer Normalization converges faster, and sometimes to better solutions, than batch- or unnormalized RNNs. Batch normalization still performs better for CNNs.

Nayebi and Ganguli propose saturating neural networks as defense against adversarial examples. The main observation driving this paper can be stated as follows: Neural networks are essentially based on linear sums of neurons (e.g. fully connected layers, convolutiona layers) which are then activated; by injecting a small amount of noise per neuron it is possible to shift the final sum by large values, thereby propagating the noisy through the network and fooling the network into misclassifying an example. To prevent the impact of these adversarial examples, the network should be trained in a manner to drive many neurons into a saturated regime – noisy will, so the argument, have less impact then. The authors also give a biological motivation, which I won't go into detail here.

Letting $\psi$ be the used activation function, e.g. sigmoid or ReLU, a regularizer is added to drive neurons into saturation. In particular, a penalty
$\lambda \sum_l \sum_i \psi_c(h_i^l)$
is added to the loss. Here, $l$ indexes the layer and $i$ the unit in the layer; $h_i^l$ then describes the input to the non-linearity computed for unit $i$ in layer $l$. $\psi_c$ is the complementary function defined as
$\psi_c(z) = \inf_{z': \psi'(z') = 0} |z – z'|$
It defines the distance of the point $z$ to the nearest saturated point $z'$ where $\psi'(z') = 0$. For ReLU activations, the complementary function is the ReLU function itself; for sigmoid activations, the complementary function is
$\sigma_c(z) = |\sigma(z)(1 - \sigma(z))|$.
In experiments, Nayebi and Ganguli show that training with the additional penalty yields networks with higher robustness against adversarial examples compared to adversarial training (i.e. training on adversarial examples). They also provide some insight, showing e.g. the activation and weight distribution of layers illustrating that neurons are indeed saturated in large parts. For details, see the paper.

I also want to point to a comment on the paper written by Brendel and Bethge [1] questioning the effectiveness of the proposed defense strategy. They discuss a variant of the fast sign gradient method (FSGM) with stabilized gradients which is able to fool saturated networks.

[1] W. Brendel, M. Behtge. Comment on “Biologically inspired protection of deep networks from adversarial attacks”, https://arxiv.org/abs/1704.01547.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).

Munoz-Gonzalez et al. propose a multi-class data poisening attack against deep neural networks based on back-gradient optimization. They consider the common poisening formulation stated as follows:

$ \max_{D_c} \min_w \mathcal{L}(D_c \cup D_{tr}, w)$

where $D_c$ denotes a set of poisened training samples and $D_{tr}$ the corresponding clea dataset. Here, the loss $\mathcal{L}$ used for training is minimized as the inner optimization problem. As result, as long as learning itself does not have closed-form solutions, e.g., for deep neural networks, the problem is computationally infeasible. To resolve this problem, the authors propose using back-gradient optimization. Then, the gradient with respect to the outer optimization problem can be computed while only computing a limited number of iterations to solve the inner problem, see the paper for detail. In experiments, on spam/malware detection and digit classification, the approach is shown to increase test error of the trained model with only few training examples poisened.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).

## **Keywords**

One pixel attack , adversarial examples , differential evolution , targeted and non-targeted attack

---


## **Summary**

1.  **Introduction **
    1.  **Basics**
        1.  Deep learning methods are better than the traditional image processing techniques in most of the cases in computer vision domain.
        1.  "Adversarial examples" are specifically modified images with imperceptible perturbations  that are classified wrong by the network.
    1.  **Goals of the paper**
        1.  In most of the older techniques excessive modifications are made on the images and it may become perceivable to the human eyes. The authors of the paper suggest a method to create adversarial examples by changing only one , three or five pixels of the image.
        1.  Generating examples under constrained conditions can help in _getting insights about the decision boundaries_ in the higher dimensional space.
    1.  **Previous Work**
        1.  Methods to create adversarial examples :
            1.  Gradient-based algorithms using backpropagation for obtaining gradient information 
            1.  "fast gradient sign" algorithm 
            1.  Greedy perturbation searching method
            1.  Jacobian matrix to build "Adversarial Saliency Map"
        1.  Understanding and visualizing the decision boundaries of the DNN input space.
        1.  Concept of "Universal perturbations" , a perturbation that when added to any natural image can generate adversarial samples with high effectiveness
1.  **Advantages of the new types of attack **
    1.  _Effectiveness_ : One pixel modification  with efficiency ranging from 60% - 75%.
    1.  _Semi-Black-Box attack _: Requires only black-box feedback (probability labels) , no gradient and network architecture required.
    1.  _Flexibility_ : Can generalize between different types of network architectures.
1.  **Methodology**
    1.  Finding the adversarial example as an optimization problem with constraints.** **
    1.  _Differential evolution_
        1.  _"Differential evolution" _, a general kind of  evolutionary algorithms , used to solve multimodal optimization problems.
        1.  Does Not make use of gradient information
        1.  Advantages of DE for generating adversarial images :
            1.  _Higher probability of finding the global optima_
            1.  _Requires less information from the target system_
            1.  _Simplicity_ : Independent of the classifier 
1.  **Results **
    1.  CIFAR-10 dataset was selected with 3 types of networks architectures , all  convolution  network  ,  Network  in  Network  and  VGG16 network . 500 random images were selected to create the perturbations and run both _targeted_ and_ non-targeted attack._
    1.  Adversarial examples were created with only one pixel change in some cases and with 3 and 5 pixel changes in other cases.
    1.  The attack was generalized over different architectures.
    1.  Some specific target-pair classes are more vulnerable to attack compared to the others.
    1.  Some classes are very difficult to perturb to other classes and some cannot be changed at all.
    1.  Robustness of the class against attack can be broken by using higher dimensional perturbations.
    
1.  **Conclusion**
    1.  Few pixels are enough to fool different types of  networks.
    1.  The properties of the targeted perturbation depends on its decision boundary.
    1.  Assumptions made that small changes addictive perturbation on the values of many dimensions will accumulate and cause huge change to the output , might not be necessary for explaining why natural images are sensitive to small perturbation.



---


## **Notes **



*   Location of data points near the decision boundaries might affect the robustness against perturbations.
    *   If the boundary shape is wide enough it is possible to have natural images far away from the boundary such that it is hard to craft adversarial images from it.  
    *   If the boundary shape is mostly long and thin with natural images close to the border, it is easy to craft adversarial images from them but hard to craft adversarial images to them.
*   The data points are moved in small steps and the change in the class probabilities are observed.

## **Open research questions**



1.  Effect of a larger set of initial candidate solutions( Training images) to finding the adversarial image?
1.  Generate better adversarial examples by having more iterations of Differential evolution?
1.  Why imbalances occur when creating perturbations?