- Result of thourough research which not only covers major research, but also compares under same criteria/ dataset; This is also a great survey.
- Train on 32-bit FP model, run 8-bit model. No retraining required to convert to 8-bit w/o loss in accuracy.
- Provides comparison concerning computing resource, it's useful to design for typical (ARM) microcontroller systems.
- MobileNet inspired DS-CNN runs small and accurate, achieves the best accuracies of 94.4% ~ 95.4%. Maybe SOTA.
- Apatche licensed code/ pretrained models are available at https://github.com/ARM-software/ML-KWS-for-MCU.

https://i.imgur.com/qahXKBn.png

# Very Short

The authors define a neural network as a nonlinear dynamical system whose fixed points correspond to the minima of some **energy function**.  They then show that if one were to start at a fixed-point and *perturb* the output units in the direction that minimizes a loss, the initial perturbation that would flow back through the network would be proportional to the gradient of the neural activations with respect to this loss.  Thus, the initial propagation of those propagations (i.e. **early inference**) **approximates** the **backpropagated** gradients of the loss.

Aim: generate realistic-looking synthetic data that can be used to train 3D Human Pose Estimation methods. Instead of rendering 3D models, they choose to combine parts of real images.

Input: RGB images with 2D annotations + a query 3D pose.
Output: A synthetic image, stitched from patches of the images, so that it looks like a person in the query 3D pose.

Steps:
- Project 3D pose on random camera to get 2D coords
- For each joint, find an image in the 2D annotated dataset whose annotation is locally similar
- Based on the similarities, decide for each pixel which image is most relevant.
- For each pixel, take the histogram of the chosen images in a neighborhood, and use this as blending factors to generate the result.

They also present a method that they trained on this synthetic dataset.

* Semi-supervised method
* There is a teacher net and a student net, with identical architecture.
* The teacher makes predictions on unlabeled data, which are used as ground-truth for training the student net.
* After each gradient descent update on the student, the teacher's weights are updated so that it becomes an exponential moving average of the weights of the student at previous timesteps. It's called a "mean teacher" because of this moving average.

* It's a semi-supervised method (the goal is to make use of unlabeled data in addition to labeled data).
* They first train a neural net normally, in the supervised way, on a labeled dataset.
* Then **they retrain the net using *its own predictions* on the originally unlabeled data as if it was ground truth** (but only when the net is confident enough about the prediction).
  * More precisely they retrain on the union of the original dataset and the examples labeled by the net itself. (Each minibatch is on average 60% original and 40% self-labeled)
* When making these predictions (that will subsequently used for training), they use **multi-transform inference**.
  * They apply the net to differently transformed versions of the image (mirroring, scaling), transform the outputs back accordingly and combine the results.