* The authors train a variant of AlexNet that has significantly fewer parameters than the original network, while keeping the network's accuracy stable.
  * Advantages of this:
    * More efficient distributed training, because less parameters have to be transferred.
    * More efficient transfer via the internet, because the model's file size is smaller.
    * Possibly less memory demand in production, because fewer parameters have to be kept in memory.

### How
  * They define a Fire Module. A Fire Module contains of:
    * Squeeze Module: A 1x1 convolution that reduces the number of channels (e.g. from 128x32x32 to 64x32x32).
    * Expand Module: A 1x1 convolution and a 3x3 convolution, both applied to the output of the Squeeze Module. Their results are concatenated.
  * Using many 1x1 convolutions is advantageous, because they need less parameters than 3x3s.
  * They use ReLUs, only convolutions (no fully connected layers) and Dropout (50%, before the last convolution).
  * They use late maxpooling. They argue that applying pooling late - rather than early - improves accuracy while not needing more parameters.
  * They try residual connections:
    * One network without any residual connections (performed the worst).
    * One network with residual connections based on identity functions, but only between layers of same dimensionality (performed the best).
    * One network with residual connections based on identity functions and other residual connections with 1x1 convs (where dimensionality changed) (performance between the other two).
  * They use pruning from Deep Compression to reduce the parameters further. Pruning simply collects the 50% of all parameters of a layer that have the lowest values and sets them to zero. That creates a sparse matrix.

### Results
  * 50x parameter reduction of AlexNet (1.2M parameters before pruning, 0.4M after pruning).
  * 510x file size reduction of AlexNet (from 250mb to 0.47mb) when combined with Deep Compression.
  * Top-1 accuracy remained stable.
  * Pruning apparently can be used safely, even after the network parameters have already been reduced significantly.
  * While pruning was generally safe, they found that two of their later layers reacted quite sensitive to it. Adding parameters to these (instead of removing them) actually significantly improved accuracy.
  * Generally they found 1x1 convs to react more sensitive to pruning than 3x3s. Therefore they focused pruning on 3x3 convs.
  * First pruning a network, then re-adding the pruned weights (initialized with 0s) and then retraining for some time significantly improved accuracy.
  * The network was rather resilient to significant channel reduction in the Squeeze Modules. Reducing to 25-50% of the original channels (e.g. 128x32x32 to 64x32x32) seemed to be a good choice.
  * The network was rather resilient to removing 3x3 convs and replacing them with 1x1 convs. A ratio of 2:1 to 1:1 (1x1 to 3x3) seemed to produce good results while mostly keeping the accuracy.
  * Adding some residual connections between the Fire Modules improved the accuracy.
  * Adding residual connections with identity functions *and also* residual connections with 1x1 convs (where dimensionality changed) improved the accuracy, but not as much as using *only* residual connections with identity functions (i.e. it's better to keep some modules without identity functions).


--------------------

### Rough chapter-wise notes

* (1) Introduction and Motivation
  * Advantages from having less parameters:
    * More efficient distributed training, because less data (parameters) have to be transfered.
    * Less data to transfer to clients, e.g. when a model used by some app is updated.
    * FPGAs often have hardly any memory, i.e. a model has to be small to be executed.
  * Target here: Find a CNN architecture with less parameters than an existing one but comparable accuracy.

* (2) Related Work
  * (2.1) Model Compression
    * SVD-method: Just apply SVD to the parameters of an existing model.
    * Network Pruning: Replace parameters below threshold with zeros (-> sparse matrix), then retrain a bit.
    * Add quantization and huffman encoding to network pruning = Deep Compression.
  * (2.2) CNN Microarchitecture
    * The term "CNN Microarchitecture" refers to the "organization and dimensions of the individual modules" (so an Inception module would have a complex CNN microarchitecture).
  * (2.3) CNN Macroarchitecture
    * CNN Macroarchitecture = "big picture" / organization of many modules in a network / general characteristics of the network, like depth
    * Adding connections between modules can help (e.g. residual networks)
  * (2.4) Neural Network Design Space Exploration
    * Approaches for Design Space Exporation (DSE):
      * Bayesian Optimization, Simulated Annealing, Randomized Search, Genetic Algorithms

* (3) SqueezeNet: preserving accuracy with few parameters
  * (3.1) Architectural Design Strategies
    * A conv layer with N filters applied to CxHxW input (e.g. 3x128x128 for a possible first layer) with kernel size kHxkW (e.g. 3x3) has `N*C*kH*kW` parameters.
    * So one way to reduce the parameters is to decrease kH and kW, e.g. from 3x3 to 1x1 (reduces parameters by a factor of 9).
    * A second way is to reduce the number of channels (C), e.g. by using 1x1 convs before the 3x3 ones.
    * They think that accuracy can be improved by performing downsampling later in the network (if parameter count is kept constant).
  * (3.2) The Fire Module
    * The Fire Module has two components:
      * Squeeze Module:
        * One layer of 1x1 convs
      * Expand Module:
        * Concat the results of:
          * One layer of 1x1 convs
          * One layer of 3x3 convs
    * The Squeeze Module decreases the number of input channels significantly.
    * The Expand Module then increases the number of input channels again.
  * (3.3) The SqueezeNet architecture
    * One standalone conv, then several fire modules, then a standalone conv, then global average pooling, then softmax.
    * Three late max pooling laters.
    * Gradual increase of filter numbers.
    * (3.3.1) Other SqueezeNet details
      * ReLU activations
      * Dropout before the last conv layer.
      * No linear layers.

* (4) Evaluation of SqueezeNet
  * Results of competing methods:
    * SVD: 5x compression, 56% top-1 accuracy
    * Pruning: 9x compression, 57.2% top-1 accuracy
    * Deep Compression: 35x compression, ~57% top-1 accuracy
  * SqueezeNet: 50x compression, ~57% top-1 accuracy
  * SqueezeNet combines low parameter counts with Deep Compression.
  * The accuracy does not go down because of that, i.e. apparently Deep Compression can even be applied to small models without giving up on performance.

* (5) CNN Microarchitecture Design Space Exploration
  * (5.1) CNN Microarchitecture metaparameters
    * blabla we test various values for this and that parameter
  * (5.2) Squeeze Ratio
    * In a Fire Module there is first a Squeeze Module and then an Expand Module. The Squeeze Module decreases the number of input channels to which 1x1 and 3x3 both are applied (at the same time).
    * They analyzed how far you can go down with the Sqeeze Module by training multiple networks and calculating the top-5 accuracy for each of them.
    * The accuracy by Squeeze Ratio (percentage of input channels kept in 1x1 squeeze, i.e. 50% = reduced by half, e.g. from 128 to 64):
      * 12%: ~80% top-5 accuracy
      * 25%: ~82% top-5 accuracy
      * 50%: ~85% top-5 accuracy
      * 75%: ~86% top-5 accuracy
      * 100%: ~86% top-5 accuracy
  * (5.3) Trading off 1x1 and 3x3 filters
    * Similar to the Squeeze Ratio, they analyze the optimal ratio of 1x1 filters to 3x3 filters.
    * E.g. 50% would mean that half of all filters in each Fire Module are 1x1 filters.
    * Results:
      * 01%: ~76% top-5 accuracy
      * 12%: ~80% top-5 accuracy
      * 25%: ~82% top-5 accuracy
      * 50%: ~85% top-5 accuracy
      * 75%: ~85% top-5 accuracy
      * 99%: ~85% top-5 accuracy

* (6) CNN Macroarchitecture Design Space Exploration
  * They compare the following networks:
    * (1) Without residual connections
    * (2) With residual connections between modules of same dimensionality
    * (3) With residual connections between all modules (except pooling layers) using 1x1 convs (instead of identity functions) where needed
  * Adding residual connections (2) improved top-1 accuracy from 57.5% to 60.4% without any new parameters.
  * Adding complex residual connections (3) worsed top-1 accuracy again to 58.8%, while adding new parameters.

* (7) Model Compression Design Space Exploration
  * (7.1) Sensitivity Analysis: Where to Prune or Add parameters
    * They went through all layers (including each one in the Fire Modules).
    * In each layer they set the 50% smallest weights to zero (pruning) and measured the effect on the top-5 accuracy.
    * It turns out that doing that has basically no influence on the top-5 accuracy in most layers.
    * Two layers towards the end however had significant influence (accuracy went down by 5-10%).
    * Adding parameters to these layers improved top-1 accuracy from 57.5% to 59.5%.
    * Generally they found 1x1 layers to be more sensitive than 3x3 layers so they pruned them less aggressively.
  * (7.2) Improving Accuracy by Densifying Sparse Models
    * They found that first pruning a model and then retraining it again (initializing the pruned weights to 0) leads to higher accuracy.
    * They could improve top-1 accuracy by 4.3% in this way.

This paper proposes a 3D human pose estimation in video method based on the dilated temporal convolutions applied on 2D keypoints (input to the network). 2D keypoints can be obtained using any person keypoint detector, but Mask R-CNN with ResNet-101 backbone, pre-trained on COCO and fine-tuned on 2D projections from Human3.6M, is used in the paper.
https://i.imgur.com/CdQONiN.png

The poses are presented as 2D keypoint coordinates in contrast to using heatmaps (i.e. Gaussian operation applied at the keypoint 2D location). Thus, 1D convolutions over the time series are applied, instead of 2D convolutions over heatmaps. The model is a fully convolutional architecture with residual connections that takes a sequence of 2D poses ( concatenated $(x,y)$ coordinates of the joints in each frame) as input and transforms them through temporal convolutions.
https://i.imgur.com/tCZvt6M.png
The `Slice` layer in the residual connection performs padding (or slicing) the sequence with replicas of boundary frames (to both left and right) to match the dimensions with the main block as zero-padding is not used in the convolution operations.

3D pose estimation is a difficult task particularly due to the limited data available online. Therefore, the authors propose semi-supervised approach of training the 2D->3D pose estimation by exploiting unlabeled video. Specifically, 2D keypoints are detected in the unlabeled video with any keypoint detector, then 3D keypoints are predicted from them and these 3D points are reprojected back to 2D (camera intrinsic parameters are required). This is idea similar to cycle consistency in the [CycleGAN](https://junyanz.github.io/CycleGAN/), for instance.
https://i.imgur.com/CBHxFOd.png
In the semi-supervised part (bottom part of the image above) training penalizes when the reprojected 2D keypoints are far from the original input. Weighted mean per-joint position error (WMPJPE) loss, weighted by the inverse of the depth to the object (since far objects should contribute less to the training than close ones) is used as the optimization goal.

The two networks (`supervised` above, `semi-supervised` below) have the same architecture but do not share any weights. They are jointly optimized where `semi-supervised` part serves as a regularizer. They communicate through the path aiming to make sure that the mean bone length of the above and below branches match.

The interesting tendency is observed from the MPJPE analysis with different amounts of supervised and unsupervised data available. Basically, the `semi-supervised` approach becomes more effective when less labeled data is available.
https://i.imgur.com/bHpVcSi.png

Additionally, the error is reduced when the ground truth keypoints are used. This means that a robust and accurate 2D keypoint detector is essential for the accurate 3D pose estimation in this setting.
https://i.imgur.com/rhhTDfo.png

The paper proposes a standardized benchmark for a number of safety-related problems, and provides an implementation that can be used by other researchers. The problems fall in two categories: specification and robustness. Specification refers to cases where it is difficult to specify a reward function that encodes our intentions. Robustness means that agent's actions should be robust when facing various complexities of a real-world environment. Here is a list of problems:

1. Specification:
  1. Safe interruptibility: agents should neither seek nor avoid interruption.
  2. Avoiding side effects: agents should minimize effects unrelated to their main objective.
  3. Absent supervisor: agents should not behave differently depending on presence of supervisor.
  4. Reward gaming: agents should not try to exploit errors in reward function.

2. Robustness:
  1. Self-modification: agents should behave well when environment allows self-modification.
  2. Robustness to distributional shift: agents should behave robustly when test differs from train.
  3. Robustness to adversaries: agents should detect and adapt to adversarial intentions in environment.
  4. Safe exploration: agent should behave safely during learning as well.

It is worth noting that problems 1.2, 1.4, 2.2, and 2.4 have been described back in "Concrete Problems in AI Safety".

It is suggested that each of these problems be tackled in a "gridworld" environment — a 2D environment where the agent lives on a grid, and the only actions it has available are up/down/left/right movements. The benchmark consists of 10 environments, each corresponding to one of 8 problems mentioned above. Each of the environments is an extremely simple instance of the problem, but nevertheless they are of interest as current SotA algorithms usually don't solve the posed task.

Specifically, the authors trained A2C and Rainbow with DQN update on each of the environments and showed that both algorithms fail on all of specification problems, except for Rainbow on 1.1. This is expected, as neither of those algorithms are designed for cases where reward function is misspecified. Both algorithms failed on 2.2--2.4, except for A2C on 2.3. On 2.1, the authors swapped A2C for Rainbow with Sarsa update and showed that Rainbow DQN failed while Rainbow Sarsa performed well.

Overall, this is a good groundwork paper with only a few questionable design decisions, such as the design of actual reward in 1.2. It is unlikely to have impact similar to MNIST or ImageNet, but it should stimulate safety-related research.

The general goal of meta-learning systems is to learn useful shared structure across a broad distribution of tasks, in such a way that learning on a new task can be faster. Some of the historical ways this has been done have been through initializations (i.e. initializing the network at a point such that it is easy to further optimize on each individual task, drawn from some distribution of tasks), and recurrent network structures (where you treat the multiple timesteps of a recurrent network as the training iterations on a single task, and train the recurrent weights of the network based on generalization performance on a wide range of tasks). This paper proposes a different approach: a learned proxy loss function. The idea here is that, often, early in the learning process, handcoded rewards aren’t the best or most valuable signal to use to guide a network, both because they may be high variance, and because they might not natively incentivize things like exploration rather than just exploitation. A better situation would be if we had some more far-sighted loss function we could use, that had proved to be a good proxy over a variety of different rewards. 

This is exactly what this method proposes to give us. Training consists of an inner loop, and an outer loop. Each instantiation of the inner loop corresponds to a single RL task, drawn from a distribution over tasks (for example, all tasks involving the robot walking to a position, with a single instantiated task being the task of walking to one specific position). Within the inner loop, we apply a typical policy gradient loop of optimizing the parameters of our policy, except, instead of expected rewards, we optimize our policy parameters according to a loss function we specifically parametrize. Within the outer loop, we take as signal the final reward on the trained policy on this task, and use that to update our parametrized loss. This parametrized loss is itself a neural network, that takes in the agent’s most recent set of states, actions, and rewards at a rolling window of recent timesteps, and performs temporal convolutions on those, to get a final loss value out the other side. In short, this auxiliary network takes in information about the agent’s recent behavior, and outputs an assessment of how well the agent is doing according to this longer-view loss criteria. Because it’s not possible to directly formulate the test performance of a policy in terms of the loss function that was used to train the policy (which would be necessary for backprop), the weights of this loss-calculating network are instead learned via evolutionary strategies. At a zoomed-out level of complexity, this means: making small random perturbations to the current parameters of the network, and moving in the direction of the random change that works the best. 

So, ultimately, you end up with a loss network that takes in recent environmental states and the behavior of the agent, and returns an estimate of the proxy loss value, that has hopefully been trained such that it captures environmental factors that indicate progress on the task, over a wide variety of similar tasks. Then, during testing, the RL agent can use that loss function to adapt its behavior. An interesting note here is that for tasks where the parameters of the task being learned are inferable from the environment - for example, where the goal is “move towards the green dot”, you don’t actually need to give the agent the rewards from a new task; ideally, it will have learned how to infer the task from the environment. One of the examples they use to prove their method has done something useful is train their model entirely on tasks where an ant-agent’s goal is to move towards various different targets on the right, and then shift it to a scenario where its target is towards the left.  In the EPG case, the ant was able to quickly learn to move left, because it’s loss function was able to adapt to the new environment where the target had moved. By contrast, RL^2 (a trained learning algorithm implemented as a recurrent network) kept on moving right as its initial strategy, and seemed unable to learn the specifics of a task outside its original task distribution of “always move right”. 

I think this paper could benefit from being a little bit more concrete about what it’s expected use cases are (like: what kinds of environments lend themselves to having proxy loss functions inferred from environmental data? Which don’t?), but overall, I find the kernel of idea this model introduces interesting, and will be interested to see if other researchers run with it.

De Alfaro proposes a deep radial basis function (RBF) network to obtain robustness against adversarial examples. In contrast to “regular” RBF networks, which usually consist of only one hidden layer containing RBF units, de Alfaro proposes to stack multiple layers with RBF units. Specifically, a Gaussian unit utilizing the $L_\infty$ norm is used:

$\exp\left( - \max_i(u_i(x_i – w_i))^2\right)$

where $u_i$ and $w_i$ are parameters and $x_i$ are the inputs to the unit – so the network inputs or the outputs of the previous hidden layer. This unit can be understood as computing a soft AND operation; therefore, an alternative OR operation

$1 - \exp\left( - \max_i(u_i(x_i – w_i))^2\right)$

is used as well. These two units are used alternatingly in hidden layers in the conducted experiments. Based on these units, de Alfaro argues that the model is less sensitive to adversarial examples, compared to linear operations as commonly used in ReLU networks.

For training a deep RBF-network, pseudo gradients are used for both the maximum operation and the exponential function. This is done for simplifying training; I refer to the paper for details.

In their experiments, on MNIST, a multi-layer perceptron with the proposed RBF units is used. The network consists of 512 AND units, 512 OR units, 512 AND units and finally 10 OR units. Robustness against FGSM and I-FGSM as well as PGD attacks seems to improve. However, the used PGD attack seems to be weaker than usually, it does not manage to reduce adversarial accuracy of a normal networks to near-zero.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).