While preserving accuracy,
- Network architecture improvement decreases parameters 51X (240MB to 4.8MB).
- By using Deep Compression, parameters shrinks more 10X more  (4.8MB to 0.47MB).

Even improves more accuracy for about 2% by using Simple Bypass (shortcut connection).

They show insightful architectural design strategies;
1. Less 3x3 filters to decrease size,
2. Decrease input channels also to decrease size,
3. Downsample late to have larger activation maps to lead to higher accuracy.

And great insights about CNN design space exploration by parametrize microarchitecture,
- Squeeze Ratio to find good balance between weight size and accuracy.
- 3x3 filter percentage to find enough number of it.

Single cells behave in complex and sometimes seemingly random ways. We have applied Generative Adversarial Networks (GANs), a form of artificial intelligence, to make sense of the way genes are controlled in skin cells. A GAN involves two separate neural networks, a ‘generator’ and a ‘discriminator’. The generator simulates cells and the discriminator tries to tell the difference between the fake data created by the simulator, and data from real cells. As they compete against each other they improve at their tasks and provide new insights into the way cells behave.   

Our neural network approach allows us to understand the relationship between different genes and how this contributes to cell behaviour. One of the networks, the generator, is responsible for simulating cells and we use this to predict how genes are controlled under different conditions, effectively simulating what would previously have been laborious and painstaking experiments. GANs also make it possible to compare data from multiple labs produced in different conditions, opening up the opportunity to answer new questions.

https://i.imgur.com/mYFkCxk.png
Main contributions:
The paper proposed a new method to provide large scale supervised reading comprehension and also developing attention based deep neural networks that can answer complex questions from real documents.

Importance:
Obtaining supervised natural language comprehensive data in large scale is difficult. On the other hand, reading comprehension methods constructed based on synthetic data failed in real environment when facing real data. This work addresses lack of real supervised reading comprehension data. In addition, they build novel deep learning models for reading comprehension by incorporating attention mechanism into recurrent neural networks. Attention mechanism allows a model to focus on the parts of a document that it believes will help it answer a question.

Method:
First part, two machine reading corpora is created by exploiting CNN and Daily Mail articles along with their corresponding summaries in form of the bullet points. These bullet points are abstractive and they are paraphrasing important parts of the article rather than copying sentences from the text. The bullet points turn into Cloze type questions by replacing one entity at a time with an entity marker, for example, “producer X will not press charges against ent212 ,his lawyer says.”. All the entities are replaced by entity markers and using a coreference and also entity markers are permuted for each data points to avoid world knowledge and co-occurrence effects in the reading comprehension.

Second part, For the reading comprehension task, they used 2 simple base line models(A), 2 symbolic matching models(B), and 4 recurrent neural networks models(C):
A1) Majority Baseline: It picks the most frequently observed entity in the context document.

A2) Exclusive Majority: It picks the most frequently observed entity in the context document which is not observed in the query.

B1) Frame-Semantic Parsing: This method parses the sentence to find "who did what to whom" using state-of-the-art frame semantic parser on the anonymized data points.

B2) Word Distance Benchmark: It aligns placeholder of Cloze form questions with each possible entity in the context document and calculates the distance between the question and the context around the aligned entity. Then sum of the distance of every word in a query to their nearest aligned word in the document is calculated.

C1) Deep LSTM Reader (2-layer LSTM)
This model feeds the [document | query] pair separated by a delimiter as a single large document, one word at a time. LSTM cells have skip connections from input to hidden layers and hidden layer to output.

C2) Attentive Reader (bi-directional LSTM with attention)
This model employs attention mechanism to overcome the bottleneck of fixed width hidden vector. First, it encodes the document and the query using separate bi-directional single layer LSTM. Then, query encoding is obtained by concatenating the final forward and backwards outputs. Document encoding is obtained by a weighted sum of output vectors (obtained by concatenating the forward and backwards outputs). The weights can be interpreted as the degree to which the network attends to a particular token in the document. Finally, the model is completed by defining a non-linear combination of document and query embedding.

C3) Uniform Reader (bi-directional LSTM)
It is Attentive Reader without attention mechanism, which is used here to see the effect of attention mechanism on the results.

C4) Impatient Reader (bi-directional LSTM with attention per each query token)
This one is similar to Attentive Reader except that the attention weights are computed per each query token. The intuition is that for each token the model finds which part of the context document is more relevant. The model accumulates the information from the document as each query token is seen and finally outputs a joint document query representation using a non-linear combination of document embedding and query embedding.

Results:
As expected, Attentive and Impatient Readers outperform all other models which show the benefits of attention model. Also Uniform Reader supports this hypothesis. The accuracies on two datasets (CNN, Daily Mail) are  Maximum Frequency: 33.2 / 25.5, Exclusive Frequency: 39.3 / 32.8, Frame-semantic model: 40.2 / 35.5, Word distance model: 50.9 / 55.5, Deep LSTM Reader: 57.0 / 62.2, Uniform Reader: 39.4 / 34.4, Attentive Reader: 63.0 / 69.0, Impatient Reader: 63.8 / 68.0.

This paper proposes a simple method for sequentially training new tasks and avoid catastrophic forgetting. The paper starts with the Bayesian formulation of learning a model that is
$$
\log P(\theta | D) = \log P(D | \theta) + \log P(\theta) - \log P(D)
$$
By switching the prior into the posterior of previous task(s), we have
$$
\log P(\theta | D) = \log P(D | \theta) + \log P(\theta | D_{prev}) - \log P(D)
$$
The paper use the following form for posterior
$$
P(\theta | D_{prev}) = N(\theta_{prev}, diag(F))
$$
where $F$ is the Fisher Information matrix $E_x[ \nabla_\theta \log P(x|\theta) (\nabla_\theta \log P(x|\theta))^T]$. Then the resulting objective function is
$$
L(\theta) = L_{new}(\theta) + \frac{\lambda}{2}\sum F_{ii} (\theta_i - \theta^{prev*}_i)^2
$$
where $L_{new}$ is the loss on new task, and $\theta^{prev*}$ is previous best parameter. It can be viewed as a distance which uses Fisher Informatrix to properly scale each dimension, and it further proves that the Fisher Information matrix is important in the experienment by comparing with simple $L_2$ distance.

The overarching goal of precision oncology is to use personal genomic information, like gene expression data from a tumor biopsy, to devise individualized treatment plans. Network-based analytics support this goal in three ways. First, by helping us to identify malfunctioning gene pathways by observing patient-specific gene expression levels in relation to one another. Second, by enabling us to predict the phenotype, or observable characteristics, of a patient's cancer. Third, by recommending drugs to re-purpose as chemotherapies that targeting a specific gene or protein that has been dysregulated in a pathway. 

## Identifying Pathological Gene Pathways
Cancer is best characterized as a disease caused by frequently mutated genes that cause dysregulated biological pathways. There are many kinds of biological pathways, such as metabolic pathways that may synthesize useful molecules or break down complex molecules, or a gene-regulatory pathway wherein molecules like proteins, mRNA, or DNA regulate the production, expression, and activity of other molecules thereby affecting the behaviour of the cell.

Systems biology makes use of many types of pathways, which may be viewed as molecular networks like the gene-regulatory network, the protein-protein interaction network, and the cellular signal transduction network. One goal of network-based analysis of personal genomic profiles is to identify network modules that are causative of cancer and informative of cancer phenotype. This paper outlines three ways of integrating molecular networks to make these predictions: model-based integration, pre-processing integration, and post-analysis integration.

The authors use the phrase **pre-processing integration** to indicate a two-step process. The first step detects molecular sub-networks by using highly discriminative genes as seed genes in a greedy search to find discriminative sub-networks. The gene expression in each sub-network is then normalized as one feature value for classification with logistic regression or another method.  This method allows you to customize your sub-network features, but generally does not produce optimal predictions.

An example application of this type of method comes from a paper called [Network-based classification of breast cancer metastasis](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2063581/). Their goal is to determine which gene pathways give rise to metastasis. They note that previous methods had identified certain genes as biomarkers correlated with metastasis. Their approach was to use these genes as seeds for an algorithm to identify discriminative sub-networks in a protein-protein interaction network.
https://i.imgur.com/WCf18fZ.png

To conduct the **post-analysis integration** of oncogenic alterations in networks, first analyze the patient genomic profiles to generate a list of oncogenic alterations. Second, map those alterations to the network as seed genes for the module analysis. This method is highly informative of cancer mechanisms in the network, but relies on accurate identification of oncogenic alterations in the patient data.
https://i.imgur.com/5CbqQTg.png

When using a **model-based integration** approach, you incorporate the molecular network into the machine learning model as a regularizing graph Laplacian. The coefficients that the model learns form dense sub-networks that are used to enable the model to predict patient survival or cancer phenotype. This form of framework has a global optimization strategy and generally produces the best outcome predictions.
https://i.imgur.com/vRH1K8R.png

## Methods for Drug Re-purposing
If you want to re-purpose a drug for use against a cancer target, there are three networks that are useful to inform your work: drug-drug similarity, drug-target relations, and gene-gene relations. Additionally, there are three types of methods that you can use with this data to create a predictive model: graph connectivity measures, link prediction models, and network based classification methods.

## Graph Connectivity Methods
Several studies have shown that drugs sharing similar chemical structures, side-effects, or similarities in gene-expression profiles following drug treatment, can be good candidates for re-purposing. It follows that we may combine a drug-drug interaction network, a drug-target interaction network, and a target-target interaction network to estimate the likelihood that a specific drug may be re-purposed for a specific target. The proxy that is used for similarity is simply length of the path from the query drug in the drug-drug interaction network to the target in the target-target interaction network.
https://i.imgur.com/abmHkKl.png

## Link Prediction Models
The link prediction model is close to the Graph Connectivity method, but relies on more advanced similarity measures and the global graph structure to determine relations between drugs and targets. For example, one paper using a link prediction model for drug repurposing uses drug-drug two dimensional structural similarity, target-target genomic sequence similarity, and a dataset of known drug-target interactions. Once these measures have been calculated two established methods are used to predict new drug-target relations: matrix completion and random walks.
https://i.imgur.com/kElewfE.png

## Network-based Classification Methods
Here we view drug repositioning as a classification problem and apply standard classification methods such as SVM. The inputs to the classifier are network topological features (local neighborhood) and the labels are from a dataset of drug-target relations describing whether a specific drug is effective against a specific target. You use the classifier to make predictions on the gene neighborhoods present in your test set.
https://i.imgur.com/cwfJr8t.png

## Network-Based Analysis of TCGA Mutation Data 
The paper also conducts a network-based analysis mutated genes in 31 cancer genome projects in TCGA using enriched KEGG pathways. The authors find that they are able to re-identify an important signalling pathway that requires the gene AMPK and is known to be affected in breast cancer (BRCA) and uterine corpus endometrial cancer (UCEC).

## Conclusion
This article surveys precision oncology, primarily discussing drug re-positioning and alorithms for integrating molecular networks with patient data. It also contains a case-study on ovarian cancer, and a network-based analysis of TCGA data using gene mutations.

## Future Work
The authors outline several directions for future work. One direction is to explore higher-resolution graphs that distinguish between isoforms of genes and capture more complex relationships between molecules. Another is to research heterogeneous cell populations in tumors using single-cell RNA sequencing to differentiate cells with different drug targets. Finally, it is essential to develop a standardized software platform that integrates biomedical, biological network data, and analytic software to support comprehensive network-based analysis of patient genomic data and drug re-positioning.