Detecting Nuclei from Microscopy Images with Deep Learning
•
2 gefällt mir•661 views
Lecture about identifying cell nuclei from standard microscopy images with deep learning models.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Detecting Nuclei from Microscopy Images with Deep Learning
Ähnlich wie Detecting Nuclei from Microscopy Images with Deep Learning (20)
Mehr von Dmytro Fishman
Mehr von Dmytro Fishman (14)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Detecting Nuclei from Microscopy Images with Deep Learning
- 1. BIIT Detecting Nuclei from Microscopy Images with Deep Learning (and not only) Nuclei:99% Nuclei:79% Nuclei:99% Nuclei:99% Nuclei:99% Nuclei:99% Nuclei:99% Dmytro Fishman dmytro@ut.ee
- 3. Cells are building blocks of all living organisms They come in different shapes, carry different functions and have various properties Biology 101
- 4. Human cells
- 5. Human cells
- 8. Neuron Red blood cell Human cells
- 9. Neuron Red blood cell Human cells
- 10. Neuron White blood cell Red blood cell Human cells
- 11. Neuron White blood cell Red blood cell Human cells
- 12. Neuron White blood cell Red blood cell Bone cell Human cells
- 13. Neuron White blood cell Red blood cell Bone cell Human cells
- 14. Neuron White blood cell Red blood cell Bone cell Egg cell Human cells
- 15. Neuron White blood cell Red blood cell Bone cell Egg cell Human cells
- 16. Neuron White blood cell Red blood cell Bone cell Egg cell Cancer cell Human cells
- 17. They all contain one DNA, how come they are so different? Neuron White blood cell Red blood cell Bone cell Egg cell Cancer cell Human cells
- 18. …AACCTGTTACAAACCG… DNA in a specific region
- 22. …AACCTGTTACAAACCG… Gene 1 Gene 2 White blood cell
- 24. …AACCTGTTACAAACCG… White blood cell Cancer cell …AACCTGTTACAAACCG…
- 25. …AACCTGTTACAAACCG… White blood cell Cancer cell …AACCTGTTACAAACCG… In different cell types different genes are expressed
- 26. But we don’t have to measure gene expression in order to say that these cells are different… Neuron White blood cell Red blood cell Bone cell Egg cell Cancer cell
- 27. We can look at them
- 29. Extraction Seeding Producing Microscopy imaging
- 30. Extraction Seeding Treating Producing Microscopy imaging
- 31. Extraction Seeding Treating Staining Producing Microscopy imaging
- 32. Extraction Seeding Treating StainingImaging Producing Microscopy imaging
- 33. Extraction Seeding Treating StainingImaging Fluorescent Producing Microscopy imaging
- 34. Extraction Seeding Treating StainingImaging Fluorescent Histology Producing Microscopy imaging
- 35. Extraction Seeding Treating StainingImaging Fluorescent Histology Producing Microscopy imaging
- 36. Extraction Seeding Treating StainingImaging Fluorescent Histology Producing Microscopy imaging
- 37. Extraction Seeding Treating StainingImaging Fluorescent Histology Brightfield Producing Microscopy imaging
- 38. Extraction Seeding Treating StainingImaging Fluorescent Histology Brightfield Producing Microscopy imaging
- 39. Extraction Seeding Treating Fluorescent Histology Brightfield Opera Phenix High-Content Screening System Producing Microscopy imaging
- 40. Fluorescent Histology Opera Phenix High-Content Screening System Extraction Seeding Treating Brightfield Producing Microscopy imaging
- 43. Let’s take a look at the data 1_Load_data.ipynb
- 44. We need to be able to detect nuclei for each of this images!
- 45. Ok, that is great, but…
- 46. Ok, that is great, but…
- 47. Mitotic Cells Breast cancer diagnostics Mitotic cell count is one of the key diagnostic markers of the diseaseHistology images Breast cancer is the second most common cancer in the world with an estimated 1.67 million new cancer cases annually
- 48. Fluorescent images Treatment Ebola Ebola virus vaccine
- 49. Live cell imaging t = 0
- 50. Live cell imaging t = 0 t = 1
- 51. Brightfield images Live cell imaging t = 0 t = 1 t = 2
- 53. Eroom’s law: #drugs discovered per $1billion
- 54. Current Best Method for Microscopy Image Analysis?
- 55. Thousand man-hours are spent manually looking at images, counting and classifying cells Current Best Method for Microscopy Image Analysis
- 56. Automated Microscopy Image Analysis Pipeline
- 58. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Histogram of pixel brightness
- 59. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Histogram of pixel brightness Background Nuclei
- 60. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Background Nuclei Magical Threshold
- 61. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Magical Threshold Background Nuclei
- 62. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Background Nuclei Magical Threshold
- 63. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Background Nuclei Magical Threshold
- 64. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Background Nuclei Magical Threshold
- 67. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features Thresholding
- 68. nuclei #1: blue, size 29px; Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features Thresholding
- 69. nuclei #2: red, size 25px; nuclei #1: blue, size 29px; Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features Thresholding
- 70. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Thresholding
- 71. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Thresholding
- 72. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Thresholding
- 73. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Thresholding Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Classical Microscopy Image Analysis Pipeline
- 74. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Thresholding Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification We are not going to cover this today
- 75. Classical Microscopy Image Analysis Pipeline 2_Tresholding.ipynb
- 76. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Can you see a problem? Thresholding
- 77. Histology Cell types Image modalities Brightfield Fluorescence Lightning conditionsDifferent magnifications
- 78. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Thresholding Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification
- 79. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Thresholding
- 80. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Can Deep Learning step in? Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification
- 81. Approach I
- 82. Original Image Ground Truth
- 83. Original Image Ground Truth
- 85. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches
- 86. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches Nuclei
- 87. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches Nuclei Empty
- 88. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches Nuclei Empty Empty
- 89. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches Nuclei Empty Empty Empty
- 90. Ground Truth Then we randomly assign extracted patches into training and validation sets (70/30) Patches Nuclei Empty Empty Empty
- 91. Ground TruthPatches Nuclei Empty Empty Empty Then we randomly assign extracted patches into training and validation sets (70/30)
- 92. Training set Empty Empty NucleiN E Compare predictions with true labels Validation set Train the network using extracted patches from training set. Predictions generated by the network should be consistent with true labels
- 93. Training set Empty Empty Empty NucleiN E Compare predictions with true labels Validation set Train the network using extracted patches from training set. Predictions generated by the network should be consistent with true labels After network has been trained, evaluate its performance on unseen validation set patches Validation set
- 94. New Image N E Trained Neural Network Segmentation
- 95. New Image N E Trained Neural Network Segmentation
- 96. New Image N E Trained Neural Network Segmentation
- 97. New Image N E Trained Neural Network Segmentation
- 98. New Image N E Trained Neural Network Segmentation
- 99. New Image N E Trained Neural Network Segmentation This is a very slow procedure Dmytro Fishman, Ardi Tampuu
- 100. Approach II
- 101. Convolutional Neural Network Let’s consider the following image
- 102. Convolutional Neural Network Let’s consider the following image
- 103. Convolutional Neural Network Convolutional layer works as a filter applied to the original image
- 104. Convolutional Neural Network Convolutional layer works as a filter applied to the original image There are many filters in the convolutional layer, they detect different patterns 4 filters
- 105. Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
- 106. Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
- 107. Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
- 108. Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
- 109. Convolutional Neural Network Each filter applied to all possible 2x2 patches of the original image produces one output value Repeat this process for all filters in this layer
- 110. Convolutional Neural Network Each filter applied to all possible 2x2 patches of the original image produces one output value Repeat this process for all filters in this layer and the next
- 111. Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network
- 112. Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network 0 1 2 7 8 9 This vector is fed into fully connected layer with as many neutrons as possible classes
- 113. Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network 0 1 2 8 9 This vector is fed into fully connected layer with as many neutrons as possible classes Each neuron outputs probabilities 7
- 114. Training Neural Networks (part III) http://scs.ryerson.ca/~aharley/vis/conv/
- 120. Training U-net model to detect nuclei 3_Deep_Learning.ipynb
- 121. Splash of colour ResultPredicted maskOriginal image
- 123. Mask R-CNN He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask r cnn. arXiv preprint arXiv:1703.06870.
- 124. Mask R-CNN
- 125. Mask R-CNN 1. Proposes bounding boxes for objects (RoI)
- 126. Mask R-CNN 1. Proposes bounding boxes for objects (RoI) 2. Filters out bad RoIs
- 127. Mask R-CNN 1. Proposes bounding boxes for objects (RoI) 2. Filters out bad RoIs 3. For each RoI builds a mask
- 128. Mask R-CNN 1. Proposes bounding boxes for objects (RoI) 2. Filters out bad RoIs 3. For each RoI builds a mask
- 129. Mask R-CNN By Daniel Majoral 1. Proposes bounding boxes for objects (RoI) 2. Filters out bad RoIs 3. For each RoI builds a mask
- 130. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Can Deep Learning step in? Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification
- 131. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Can Deep Learning step in? Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Yes!
- 132. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III filters contrast denoising
- 133. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III E N filters contrast denoising Pixel wise segmentation
- 134. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III Autoencoders E N filters contrast denoising Pixel wise segmentation
- 135. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III Autoencoders U-net E N filters contrast denoising Pixel wise segmentation
- 136. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III Autoencoders U-net Mask R-CNN E N filters contrast denoising Pixel wise segmentation
- 138. E N VS Almighty Deep Learning Dude in the lab
- 139. E N Almighty Deep Learning Dude in the lab VS
- 140. Histology Cell types Image modalities Brightfield Fluorescence Lightning conditionsDifferent magnifications
- 144. Can we detect nuclei without fluorescent dye?
- 145. Sten-Oliver Salumaa
- 146. Nuclei:99% Nuclei:79% Nuclei:99% Nuclei:99% Nuclei:99% Nuclei:99% Nuclei:99% Team: Sten-Oliver Salumaa, Daniel Majoral, Dmytro Fishman, Mikhail Papkov, Ardi Tampuu, William Jones, Elizabeth Bell, Ilya Kuzovkin, Tanel Pärnamaa, Leopold Parts, Jaak Vilo, Raul Vicente, Kaupo Palo and Martin Daffertshofer
- 147. Something important that we learnt today
- 148. Something important that we learnt today Difference in gene expression profile causes cells to be different
- 149. Something important that we learnt today Difference in gene expression profile causes cells to be different t = 0 t = 1 t = 2 Effective cell detection can help develop better and cheaper drugs
- 150. Something important that we learnt today t = 0 t = 1 t = 2 Thresholding Difference in gene expression profile causes cells to be different Classical image analysis pipeline includes manual parameter tuning Effective cell detection can help develop better and cheaper drugs
- 151. Something important that we learnt today t = 0 t = 1 t = 2 Thresholding There are ways in which Deep Learning can make a difference Difference in gene expression profile causes cells to be different Classical image analysis pipeline includes manual parameter tuning Effective cell detection can help develop better and cheaper drugs
- 152. Something important that we learnt today t = 0 t = 1 t = 2 Thresholding Classical image analysis pipeline includes manual parameter tuning There are ways in which Deep Learning can make a difference Dude is hard to beat (human is still superior when it comes to cell detection) Difference in gene expression profile causes cells to be different Effective cell detection can help develop better and cheaper drugs
- 153. Something important that we learnt today t = 0 t = 1 t = 2 Thresholding There are ways in which Deep Learning can make a difference Dude is hard to beat (human is still superior when it comes to cell detection) and last but not least… Difference in gene expression profile causes cells to be different Classical image analysis pipeline includes manual parameter tuning Effective cell detection can help develop better and cheaper drugs