This talk will cover various medical applications of deep learning including tumor segmentation in histology slides, MRI, CT, and X-Ray data. Also, more complicated tasks such as cell counting where the challenge is to count how many objects are in an image. It will also cover generative adversarial networks and how they can be used for medical applications. This presentation is accessible to non-doctors and non-computer scientists.

1. Deep learning approaches to medical applications
1. What is Deep Learning?
2. Image and Volume
a. Segmentation
b. Counting
c. Interpreting
3. Genomics
a. Representation
Joseph Paul Cohen
Postdoctoral Fellow at Montreal Institute for Learning Algorithms
Friend of the Farlow Fellow at Harvard University
U.S. National Science Foundation Graduate Fellow

2. What is Deep Learning?
Logistic Regression
[200, 45, -4, 80] [1, 1, 0, 1]

3. What is Deep Learning?
Logistic Regression

4. What is Deep Learning?
Image credit Pascal Vincent
Logistic Regressions

5. What is Deep Learning?
Image credit Pascal Vincent

6. What is Deep Learning?
Logistic Regressions

8. Automated pathology

9. Tumor Segmentation
Automated detection of metatases in hematoxylin
and eosin (H&E) stained whole-slide images of
lymph node sections.
Reduce the workload of the pathologists and the
subjectivity in diagnosis!
https://camelyon16.grand-challenge.org/
Determining the
probability that an
image patch is a
tumour can be solved
using deep learning
models.
Image credit: Harvard Medical School, MIT, and EXB Research

10. https://camelyon16.grand-challenge.org/
Image credit: Harvard Medical School, MIT, and EXB Research

11. https://camelyon16.grand-challenge.org/
Image credit: Harvard Medical School, MIT, and EXB Research

12. https://camelyon16.grand-challenge.org/
Image credit: Harvard Medical School, MIT, and EXB Research
High accuracy screening and very reproducible!

13. How does it work?

14. DownsampleUpsample
Segmentation with Deep Learning
Image by Qikui Zhu
[Ronneberger et al., U-Net: Convolutional Networks for
Biomedical Image Segmentation 2015]
[Honari et al., Recombinator Networks: Learning
Coarse-to-Fine Feature Aggregation 2015]
Images by Vincent Dumoulin (UdeM)

15. Image credit: Lewis Fishgold and Rob Emanuele
Segmentation training in progress

16. How can the network learn?

17. Example
Given a patch predict the class of the center pixel
Image credits: http://cs231n.github.io/linear-classify/
Tumor
No Tumor
Necrosis
Image Patch

18. Softmax
To predict multiple classes we project to a probability distribution
Tumor
No Tumor
Necrosis
3.2
5.1
-1.7
24.5
164.0
0.18
exp normalize
0.13
0.87
0.0

19. Simplex
Because it is on a simplex; the correction of one term impacts all
Tumor
Necrosis
TumorNo
Tumor
Necrosis
Image credits: http://gureckislab.org/
No
Tumor
Tumor
Tumor
No Tumor
Necrosis
0.13
0.87
0.0
x

20. Infinite ways to generate the same
output.
A correction of one sends gradients
to others
We can learn unseen classes
through a process of elimination.
https://github.com/ieee8023/NeuralNetwork-Examples/blob/master/general/simplex-softmax.ipynb
No Tumor
TumorNecrosis
Necrosis
Tumor
No Tumor

21. Softmax and Cross-entropy loss
To predict multiple class we can project the output onto a simplex and
compute the loss there.
Tumor
No Tumor
Necrosis
3.2
5.1
-1.7
24.5
164.0
0.18
exp normalize
0.13
0.87
0.0
loss
0.0
0.13
0.0

23. More segmentation tasks!
CT Pancreas Segmentation MRI Brain Tumor Segmentation

24. More segmentation tasks!
CT Pancreas Segmentation MRI Brain Tumor Segmentation

25. Sub-acute ischemic stroke
lesion segmentation
Stroke Perfusion
Estimation
Brain tumor
segmentation
More segmentation tasks!
Image Credit: Mohammad Havaei

26. Multi-modal MRI acquisition
T1 T2
T1C Flair
Sub-acute ischemic stroke
lesion segmentation
Stroke Perfusion
Estimation
Brain tumor
segmentation
Image Credit: Mohammad Havaei

27. Multi-modal MRI acquisition
T1 T2
T1C Flair
Sub-acute ischemic stroke
lesion segmentation
Stroke Perfusion
Estimation
Brain tumor
segmentation
MissingMissing
Missing
Missing
Image Credit: Mohammad Havaei

28. HeMIS: Hetero-Modal Image Segmentation

29. HeMIS: Hetero-Modal Image Segmentation
Missing

30. Results (BRATS)

31. Results (BRATS)

32. Results (BRATS)

33. Results (BRATS)

34. Brain tumor segmentation (BRATS2013 dataset)
T1 T2
T1C Flair
GT
Edema
Necrosis
Non-enhanced
Enhanced
Training data:
220 subjects with high grade and
54 subjects with low grade tumors
Dice Similarity
[Havaei et al. HeMIS: Hetero-Modal Image Segmentation, MICCAI 2016]

36. Interpreting Chest X-Rays
Wang, ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common
Thorax Diseases, 2017

37. Convolutional Neural Net Predictions
32,717 patients
108,948 X-Rays
Text extracted using NLP from
doctors notes
AUCs for each class of a multi-target model

41. [Cohen et al. 2017]
Complicated cell structure
Cell counting from images

42. Cell counting from images
Cohen et al., "Count-ception: Counting by Fully Convolutional Redundant Counting," 2017

43. Cell counting from images
Cohen et al., "Count-ception: Counting by Fully Convolutional Redundant Counting," 2017

44. [Cohen et al. 2017]

45. [Cohen et al. 2017]

46. Kaggle sea lion challenge (37th/385 place)
Implemented by Robin Dinse (Universität Koblenz-Landau)

48. Semi-supervised Segmentation with GANs
Images with segmentation labels Images without segmentation labels

49. Semi-supervised Segmentation with GANs
Zhang et al., "Deep Adversarial Networks for Biomedical Image Segmentation Utilizing Unannotated Images," 2017

52. Genomics - Gene Understanding
Costanzo, Michael, et al. "The genetic
landscape of a cell." Science (2010)

53. Gene understanding
How can we represent different tissue types?
[Trofimov et al. ICML WCB 2017]

54. Factorized Embeddings
Gene expression is predicted given
(Tissue, Gene) pair
Genes condition Tissue Prediction
Tissues condition Gene Prediction
Embedding locations are updated
based on function
[Trofimov et al. ICML WCB 2017]

56. MYH6
"Muscle contraction"
[uniprot.org]
We can visualize an
expression level for every
tissue embedding

57. MYH6
"Muscle contraction"
[uniprot.org]
What is this gene
involved in?

58. MYH6
"Muscle contraction"
[uniprot.org]

59. KRT (Keratin)
Keratin is the protein
that protects epithelial
cells from damage or
stress
What is this gene
involved in?

60. KRT (Keratin)
Keratin is the protein
that protects epithelial
cells from damage or
stress

62. Thanks!
ShortScience.org
ShortScience.orgConferences