This paper describes the Bitcoin peer-to-peer currency system.

The paper first describes a bitcoin as a chain of transactions, the latest of which contains the current owner's public key, and is cryptographically signed by the previous owner.  The integrity of this chain (and therefore the ownership of the coin) is maintained as each transaction contains a hash of the previous transaction.  Therefore, it is impossible to corrupt any one transaction in the chain without affecting all future transactions as well.

https://i.imgur.com/hKbgsdi.png

The key issue with this system is *double counting* in a peer-to-peer environment.  In physical currency, this problem does not exist, as the purchase using a physical token precludes using the same token by the same owner again.  In other digital currencies, the problem of double counting is solved using a central arbiter.

The authors solve this using a timestamp service, which stamps a block of transactions.  The transaction which have the earliest timestamp is used as the true transaction.  The block of transactions is linked to a previous block, thereby forming a *blockchain*.

https://i.imgur.com/BYlVirg.png

To prevent the integrity of the blockchain, each block contains the previous hash, thereby ensuring the integrity of the chain.  There is also a nonce value, which is an input to a computationally difficult problem which must be solved in order to place a new block in the blockchain.  This makes it difficult for an attacker to tack on a new chain.  

The problem that must be solved is to increment the nonce such that the hash of the block contains a specified number of 0 bits.  The difficulty of this problem can be adjusted by increasing the number of 0's, and grows over time to accommodate the growing power of computer hardware.  Note that verifying that this problem is easily solved.

The resulting blockchain therefore constitutes a large investment of computational resources, with the longest blockchain being used as the standard.

Not all participants in bitcoin need to mine, so an incentive is given.  When a new block is mined, the individual who mines it is given some portion of bitcoin either out of thin-air or as a transaction fee.  This also deters would-be corruptors of the blockchain, as the computational resources needed may be better spent adding new blocks on the newest version of the blockchain.

Sources:
- https://arxiv.org/pdf/1512.03385.pdf
- http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf

Summary:
- Took the first place in Imagenet 5 main tracks
- Revolution of depth: GoogLeNet was 22 layers with 6.7 top-5 error, 
Resnet is 152 layers with 3.57 top-5 error
- Light on complexity: the 34 layer baseline is 18% of the FLOPs(multiply-adds) of VGG.
    - Resnet 152 has lower time complexity than VGG-16/19
- Extends well to detection and segmentation tasks
- Just stacking more layers gives worse performance. Why? In theory:
    > A deeper model should not have
    higher training error
    • A solution by construction:
    • original layers: copied from a
    learned shallower model
    • extra layers: set as identity
    • at least the same training error
    • Optimization difficulties: solvers
    cannot find the solution when going
    deeper…
- Why do the residual connections help? it's easier to learn a residual mapping w.r.t. identity. 
    - If identity were optimal, easy to set weights as 0
    - >If the optimal function is closer to an identity
mapping than to a zero mapping, it should be easier for the
solver to find the perturbations with reference to an identity
mapping, than to learn the function as a new one. We show
by experiments (Fig. 7) that the learned residual functions in
general have small responses, suggesting that identity mappings
provide reasonable preconditioning.
- Basic design (VGG-style)
    - all 3x3 conv (almost)
    - spatial size /2 => # filters x2
    - Simple design; just deep!
    - Other remarks:
        - no max pooling (almost)
        - no hidden fc
        - no dropout
- Training
    - All plain/residual nets are trained from scratch
    - All plain/residual nets use Batch Normalization
    - Standard hyper-parameters & augmentation
- The learned features are well transferable to other tasks
    - Works well with Faster RCNN
    - Works well with semantic instance segmentation
 
Also skimmed:
- Deep Residual Networks with 1K Layers
https://github.com/KaimingHe/resnet-1k-layers 
https://arxiv.org/pdf/1603.05027.pdf

This paper presents a multi-task Bayesian optimization approach to hyper-parameter setting in machine learning models. In particular, it leverages previous work on multi-task GP learning with decomposable covariance functions and Bayesian optimization of expensive cost functions. Previous work has shown that decomposable covariance functions can be useful in multi-task regression problems (e.g. \cite{conf/nips/BonillaCW07}) and that Bayesian optimization based on response-surfaces can also be useful for hyper-parameter tuning of machine learning algorithms \cite{conf/nips/SnoekLA12} \cite{conf/icml/BergstraYC13}. 

The paper combines the decomposable covariance assumption \cite{conf/nips/BonillaCW07} and Bayesian optimization based on expected improvement \cite{journals/jgo/Jones01} and entropy search \cite{conf/icml/BergstraYC13} to show empirically that it is possible to : 
1. Transfer optimization knowledge across related problems, addressing e.g. the cold-start problem 
2. Optimize an aggregate of different objective functions with applications to speeding-up cross validation 
3. Use information from a smaller problem to help optimize a bigger problem faster 

Positive experimental results are shown on synthetic data (Branin-Hoo function), optimizing logistic regression hyper-parameters and optimizing hyper-parameters of online LDA on real data. 

The paper suggests using a differentiable function which can smoothly interpolate between multiplicative and additive gates in neural networks.  It is an intriguing idea and the paper is well written.  The mathematical ideas introduced are perhaps not novel (a cursory search seems to indicate that Abel's functional equation with f=exp is called the tetration equation and its solution called the iterated logarithm), but their use in machine learning seem to be.

Gehr et al. propose a method based on abstract interpretations in order to verify robustness guarantees of neural networks. First of all, I want to note that (in contrast to most work in adversarial robustness) the proposed method is not intended to improve robustness, but to get robustness certificates. Without going into details, abstract interpretations allow to verify conditions (e.g., robustness) of a function (e.g., a neural network) based on abstractions of the input. In particular, by abstracting a norm-ball around a test sample (as is typically considered in adversarial robustness) using box constraints or polyhedra, leading to an over-approximation of the norm-ball, and transforming these abstractions according to the layers of a network, the network’s output can be checked against robustness conditions without running the network on all individual points in the norm-ball. As a result, if the proposed method certifies robustness for a given input sample and an area around it, the network s indeed robust in this area (soundness). If not, the network might indeed not be robust, or robustness could not be certified due to the method’s over-approximation. For details, I refer to the paper, as well as follow-up work [1] and [2].

[1] Matthew Mirman, Timon Gehr, Martin T. Vechev: Differentiable Abstract Interpretation for Provably Robust Neural Networks. ICML 2018: 3575-3583

[2] Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, Martin T. Vechev: Fast and Effective Robustness Certification. NeurIPS 2018: 10825-10836

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).