Scientists meet Entrepreneurs - AI & Machine Learning, Dima Fishman, University of Tartu

1. Deep Learning in Medicine and Computational Biology Dmytro Fishman (dmytro@ut.ee)

2. Cat

3. Cat

4. Cat Representation

5. Cat Representation

6. Cat Representation Cat

7. Cat Representation Cat Not a cat Machine Learning

8. http://www.asimovinstitute.org/wp-content/uploads/2016/09/neuralnetworks.png

9. 1.2 million images 1000 categories

10. Errors 2010 2011 28% 26% http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/

11. Errors 2010 2011 2012 28% 26% 16% http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ AlexNet (A. Krizhevsky et al. 2012)

12. Errors 2010 2011 2012 2013 2014 2015 2016 28% 26% 16% 12% 7% 3% <3% AlexNet (A. Krizhevsky et al. 2012) http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/

13. Errors 2010 2011 2012 2013 2014 2015 2016 28% 26% 16% 12% 7% 3% <3% Hypothetical super- dedicated ﬁne- grained expert ensemble of human labelers AlexNet (A. Krizhevsky et al. 2012) http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/

14. Errors 2010 2011 2012 2013 2014 2015 2016 28% 26% 16% 12% 7% 3% <3% Hypothetical super- dedicated ﬁne- grained expert ensemble of human labelers AlexNet (A. Krizhevsky et al. 2012) https://www.semanticscholar.org/paper/Fine-grained-Categorization-Short-Summary-of-our-E-G%C3%B6ring-Freytag/0f3b7d252c236d47cf4185fd81bbb40767baf3d8 Different breeds The same breed

15. http://thelibertarianrepublic.com/wp-content/uploads/2016/01/self-driving-car1.jpg

16. http://static.dnaindia.com/sites/default/ﬁles/styles/half/public/2016/03/09/435253-skype-translator.jpg?itok=1kAl-u1D

19. https://github.com/deepmind/pysc2 Atari Starcraft Go Dota 2

20. How about medical ﬁeld?

21. Medicine is complex

22. Medicine is really complex

23. ReconMap 2.0

24. ReconMap 2.0

25. Medicine is massive

26. Deep Learning Super rapid growth

27. Deep Learning Fever Super rapid growth

28. Medicine is weird

29. Bloodletting

30. Bloodletting Soothing

31. Bloodletting Lobotomy Soothing

32. Nevertheless…

33. https://pancreaticcanceraction.org/wp-content/uploads/2012/10/Trends-in-cancer-survival-by-tumour-site-1971_2011.png Year1970 2010 Trends in cancer survival Skin Breast Prostate Leukaemia Bowel Kidney Myeloma Ovarian Pancreatic

34. Merck Molecular Activity Challenge 2012

35. Diabetic Retinopathy Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs Dermatologist-level classiﬁcation of skin cancer with deep neural networks Skin Cancer

36. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs Dermatologist-level classiﬁcation of skin cancer with deep neural networks Skin Cancer Diabetic Retinopathy

37. Diabetic Retinopathy

41. Diagnostics done manually 9 - 12 minutes per patient

42. 128 175 images for training

43. 128 175 images for training 54 US licensed ophthalmologists

44. 128 175 images for training 54 US licensed ophthalmologists Classify into healthy, mild and severe

45. Sample annotation 1st doctor 2nd doctor 3rd doctor 4th doctor Healthy Disease

46. 0 0 0 0 1st doctor 2nd doctor 3rd doctor 4th doctor Healthy Disease Sample annotation

47. Healthy Disease 0 1st doctor 2nd doctor 3rd doctor 4th doctor Sample annotation

48. Healthy Disease 0 1 4 3 4 1st doctor 2nd doctor 3rd doctor 4th doctor Sample annotation

49. Healthy Disease 0 Major vote is used 1 4 3 4 1st doctor 2nd doctor 3rd doctor 4th doctor Sample annotation

50. Healthy Disease 0 4 Algorithm might only marginally outperform doctors 1st doctor 2nd doctor 3rd doctor 4th doctor Sample annotation

51. Algorithm vs Ophthalmologists

52. Algorithm vs Ophthalmologists Sensitivity,% 100 - Speciﬁcity, % 0100 0 100 AUC of 97.4%

53. Algorithm vs Ophthalmologists Sensitivity,% 100 - Speciﬁcity, % 0100 0 100 AUC of 97.4% The black curve is ROC for the Deep Learning algorithm

54. Algorithm vs Ophthalmologists Sensitivity,% 100 - Speciﬁcity, % 0100 0 100 AUC of 97.4% Points on ROC are performances of individual ophthalmologists The black curve is ROC for the Deep Learning algorithm

55. Performances are very similar Algorithm vs Ophthalmologists Points on ROC are performances of individual ophthalmologists The black curve is ROC for the Deep Learning algorithm Sensitivity,% 100 - Speciﬁcity, % 0100 0 100 AUC of 97.4%

56. Deep Learning algorithm can operate in any point on the curve Algorithm vs Ophthalmologists Sensitivity,% 100 - Speciﬁcity, % 0100 0 100 AUC of 97.4%

57. Deep Learning algorithm can operate in any point on the curve Sensitivity,% 100 - Speciﬁcity, % 0100 0 100 AUC of 97.4% Algorithm vs Ophthalmologists High speciﬁcity mode (diagnosis)

58. Deep Learning algorithm can operate in any point on the curve Sensitivity,% 100 - Speciﬁcity, % 0100 0 100 AUC of 97.4% Algorithm vs Ophthalmologists High speciﬁcity mode (diagnosis) High sensitivity mode (screening)

59. Deep Learning algorithm can operate in any point on the curve Sensitivity,% 100 - Specificity, % 0100 0 100 AUC of 97.4% Algorithm vs Ophthalmologists High specificity mode (diagnosis) High sensitivity mode (screening) While ophthalmologists’s mode is fixed by experience

60. Diabetic Retinopathy Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs Dermatologist-level classiﬁcation of skin cancer with deep neural networks Skin Cancer

61. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs Dermatologist-level classiﬁcation of skin cancer with deep neural networks Diabetic Retinopathy Skin Cancer

62. ?!

63. 0 1

64. 0 1

65. 0 1 90% VS 14% Difference in survival rates is drastic

66. 0 1 90% VS 14%

67. 129 450 images for training set

68. 129 450 images for training set 21 board- certiﬁed dermatologists

69. Skin disease

70. Non-neoplastic Benign Malignant

71. DermalEpidermal Melanocytic GenodermatosisInﬂammatory Epidermal Lymphoma Melanoma Dermal

72. DermalEpidermal Melanocytic Inﬂammatory Epidermal Lymphoma Melanoma Dermal Genodermatosis

73. Google Inception v3

74. Google Inception v3

75. 9% 1% 5% 4.5% 8.5% 2% 2% 4% 1.5% 0.5% 1.5% 4.5% 1% 3% 7% 15% 10% 1%3.3% 6.3% 0.4% 4% 5%

76. 9% 1% 5% 4.5% 8.5% 2% 2% 4% 1.5% 0.5% 1.5% 4.5% 1% 3% 7% 15% 10% 1%3.3% 6.3% 0.4% 4% 5% 15% Sum leaf nodes

77. 9% 1% 5% 4.5% 8.5% 2% 2% 4% 1.5% 0.5% 1.5% 4.5% 1% 3% 7% 15% 10% 1%3.3% 6.3% 0.4% 4% 5% 15% 13% 8% 10% 5.5% 2% 20% 25% 15%

78. 9% 1% 5% 4.5% 8.5% 2% 2% 4% 0.5% 1.5% 4.5% 1% 3% 7% 15% 10% 1%3.3% 6.3% 0.4% 4% 5% 36% 45% 19%

79. 9% 1% 5% 4.5% 8.5% 2% 2% 4% 0.5% 1.5% 4.5% 1% 3% 7% 15% 10% 1%3.3% 6.3% 0.4% 4% 5% 36% 45% 19%

80. Algorithm vs Dermatologists

81. Algorithm vs Dermatologists Speciﬁcity,% Sensitivity, % AUC of 96% Carcinoma: 135 images Dermatologists (25)

82. Each of the cases was veriﬁed by biopsy Speciﬁcity,% Sensitivity, % AUC of 96% Carcinoma: 135 images Dermatologists (25) Algorithm vs Dermatologists

83. Speciﬁcity,% Sensitivity, % AUC of 96% Carcinoma: 135 images Dermatologists (25) Algorithm vs Dermatologists

84. Speciﬁcity,% Sensitivity, % AUC of 96% Carcinoma: 135 images Dermatologists (25) Algorithm vs Dermatologists Performance of the algorithm was compared to dermatologists

85. Performance of the algorithm was compared to dermatologists Average dermatologist’s performance was marked as Speciﬁcity,% Sensitivity, % AUC of 96% Carcinoma: 135 images Dermatologists (25) Algorithm vs Dermatologists

86. Performance of the algorithm was compared to dermatologists Average dermatologist’s performance was marked as Speciﬁcity,% Sensitivity, % AUC of 96% Carcinoma: 135 images Dermatologists (25) Algorithm vs Dermatologists

87. Specificity,% Sensitivity, % AUC of 96% Specificity,% Sensitivity, % AUC of 94% Specificity,% Sensitivity, % AUC of 91% Carcinoma: 135 images Melanoma: 130 images Melanoma: 111 images Dermatologists (25) Dermatologists (22) Dermatologists (21) Algorithm vs Dermatologists

88. Specificity,% Sensitivity, % AUC of 96% Specificity,% Sensitivity, % AUC of 94% Specificity,% Sensitivity, % AUC of 91% Carcinoma: 135 images Melanoma: 130 images Melanoma: 111 images Dermatologists (25) Dermatologists (22) Dermatologists (21) Algorithm vs Dermatologists Across all biopsy verified datasets Deep Neural Network was superior

89. Diabetic Retinopathy Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs Dermatologist-level classiﬁcation of skin cancer with deep neural networks Skin Cancer https://jamanetwork.com/journals/jama/fullarticle/2588763 https://www.nature.com/nature/journal/v542/n7639/full/ nature21056.html

90. Few more interesting applications

91. Diagnosing Parkinson from voice (Al-Fatlawi et al., 2016) Detection of hypoglycemic episodes in children (San et al., 2016) HemoglobinA1c 03.2010 Timeline 07.2010 12.2010 02.2011 04.2011 ? Painintensity Frames Pain estimation from video (Zhou et al., 2016) Predicting subsequent hospitalisation (Choi et al., 2016)

92. Sleep

93. Sleep Hunger

94. Sleep Pain Hunger

95. Seems like revolution did not happened

96. Why Deep Learning has not revolutionised medicine yet?

97. Chart of possible reasons why deep learning may fail to revolutionise medicineLikelihood Effect UnlikelyHighlylikely Not nice, but ok Terrible consequences

98. We may fail to compose large enough datasets

99. Collecting data in medicine is very expensive We may fail to compose large enough datasets

100. Collecting data in medicine is very expensive Medical data is often protected (for a good reason) We may fail to compose large enough datasets

104. We can build a model that can distinguish them from other objects We may fail to compose large enough datasets

105. We can build a model that can distinguish them from other objects We may fail to compose large enough datasets

106. We can build a model that can distinguish them from other objects We cannot build a robust representation for all of them We may fail to compose large enough datasets

107. We can build a model that can distinguish them from other objects We would need a separate ImageNet for each type We may fail to compose large enough datasets We cannot build a robust representation for all of them

108. http://langlotzlab.stanford.edu/projects/medical-image-net/ http://langlotzlab.stanford.edu/projects/medical-image Possible solution

109. http://langlotzlab.stanford.edu/projects/medical-image-net/

110. Chart of possible reasons why deep learning may fail to revolutionise medicineLikelihood Effect UnlikelyHighlylikely Not nice, but ok Terrible consequences Data

111. How doctors diagnose melanomas?

112. There is a ABCD rule they learned in college How doctors diagnose melanomas?

113. There is a ABCD rule they learned in college Melanomas are Asymmetrical How doctors diagnose melanomas?

114. There is a ABCD rule they learned in college Melanomas are Asymmetrical How doctors diagnose melanomas?

115. There is a ABCD rule they learned in college Melanomas are Asymmetrical How doctors diagnose melanomas? Their Borders are uneven

116. There is a ABCD rule they learned in college Melanomas are Asymmetrical Their Borders are uneven How doctors diagnose melanomas?

117. There is a ABCD rule they learned in college Melanomas are Asymmetrical Colour can be patchy and variegated How doctors diagnose melanomas? Their Borders are uneven

118. There is a ABCD rule they learned in college Melanomas are Asymmetrical Colour can be patchy and variegated Their Borders are uneven How doctors diagnose melanomas?

119. There is a ABCD rule they learned in college Melanomas are Asymmetrical Colour can be patchy and variegated How doctors diagnose melanomas? Their Borders are uneven

120. There is a ABCD rule they learned in college Colour can be patchy and variegated their Diameter is usually > 6 millimetres How doctors diagnose melanomas? Melanomas are Asymmetrical Their Borders are uneven

121. There is a ABCD rule they learned in college Melanomas are Asymmetrical Their Borders are uneven Colour can be patchy and variegated their Diameter is usually > 6 millimetres How doctors diagnose melanomas?

122. How computers diagnose melanomas?

123. How computers diagnose melanomas?

124. How computers diagnose melanomas? Melanoma

125. How computers diagnose melanomas? Melanoma

126. https://arxiv.org/pdf/1708.08296v1.pdf

127. Chart of possible reasons why deep learning may fail to revolutionise medicineLikelihood Effect UnlikelyHighlylikely Not nice, but ok Terrible consequences DataInterpretability

128. Your computer

129. Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

130. Gene Technology Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more… Genome

131. Genome Schizofrenia - 0.15% Diabetes - 0.05% Cancer - 0.01% Gene Technology Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

132. Risks Genome Schizofrenia - 0.15% Diabetes - 0.05% Cancer - 0.01% Gene Technology Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

133. Risks Genome Schizofrenia - 0.15% Diabetes - 0.05% Cancer - 0.01% Gene Technology Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

134. Application Cool company Risks Genome Schizofrenia - 0.15% Diabetes - 0.05% Cancer - 0.01% Gene Technology Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

135. Application Cool company Risks Genome Schizofrenia - 0.15% Diabetes - 0.05% Cancer - 0.01% Gene Technology Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

136. Risks Application Cool company Risks Genome Schizofrenia - 0.15% Diabetes - 0.05% Cancer - 0.01% Gene Technology Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

137. Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

138. Encrypting your genome Your computer ACCCTTAAGGAGATCCTT TAACCGAACCTCACCCTT AAGGAGATCCTTTAACCG CCCTTTTATTCCTATTACGT Read 3.5 B more…

139. !#$#*(#$@&#$@^@ %#@%@^#@#$*(@#) $#@&#$@^#@$)@# $#@*#@$#@####@@!! Read 3.5 B more… Encrypting your genome Your computer

140. Encrypted genome Gene Technology !#$#*(#$@&#$@^@ %#@%@^#@#$*(@#) $#@&#$@^#@$)@# $#@*#@$#@####@@!! Read 3.5 B more… Your computer

141. #@#$!!@@# - #?@% #@$(%&&& - ?@#% $&@))#^%? - ??@% Encrypted genome Gene Technology !#$#*(#$@&#$@^@ %#@%@^#@#$*(@#) $#@&#$@^#@$)@# $#@*#@$#@####@@!! Read 3.5 B more… Your computer

142. Encrypted risks #@#$!!@@# - #?@% #@$(%&&& - ?@#% $&@))#^%? - ??@% Encrypted genome Gene Technology !#$#*(#$@&#$@^@ %#@%@^#@#$*(@#) $#@&#$@^#@$)@# $#@*#@$#@####@@!! Read 3.5 B more… Your computer

143. Encrypted risks #@#$!!@@# - #?@% #@$(%&&& - ?@#% $&@))#^%? - ??@% Encrypted genome Gene Technology !#$#*(#$@&#$@^@ %#@%@^#@#$*(@#) $#@&#$@^#@$)@# $#@*#@$#@####@@!! Read 3.5 B more… Your computer Decrypt your risks with your private key

144. Encrypted risks #@#$!!@@# - #?@% #@$(%&&& - ?@#% $&@))#^%? - ??@% Encrypted genome Gene Technology Your computer Decrypt your risks with your private key Schizofrenia - 0.15% Diabetes - 0.05% Cancer - 0.01%

145. and start panicking changing your lifestyle Encrypted risks #@#$!!@@# - #?@% #@$(%&&& - ?@#% $&@))#^%? - ??@% Encrypted genome Gene Technology Your computer Decrypt your risks with your private key Schizofrenia - 0.15% Diabetes - 0.05% Cancer - 0.01%

146. Chart of possible reasons why deep learning may fail to revolutionise medicineLikelihood Effect UnlikelyHighlylikely Not nice, but ok Terrible consequences DataInterpretability ! # $ # * ( # $ @ & # $ @ ^ @ % # @ % @ ^ # @ # $ * ( @ # ) $ # @ & # $ @ ^ # @ $ ) @ # $ # @ * # @ $#@####@@!! Read 3.5 B more… Privacy

147. Original image

148. Original image Correct segmentation Segmentation Road Cars Trees

149. Original image Correct segmentation Segmentation Road Cars Trees Original image Adversarial example

150. Original image Correct segmentation Segmentation Road Cars Trees Original image Adversarial example Altered image

151. Original image Correct segmentation Segmentation Road Cars Trees Original image Adversarial example Altered image New funny segmentation

152. Chart of possible reasons why deep learning may fail to revolutionise medicineLikelihood Effect UnlikelyHighlylikely Not nice, but ok Terrible consequences DataInterpretability ! # $ # * ( # $ @ & # $ @ ^ @ % # @ % @ ^ # @ # $ * ( @ # ) $ # @ & # $ @ ^ # @ $ ) @ # $ # @ * # @ $#@####@@!! Read 3.5 B more… Privacy Adversarial attacks

153. This is all great stuff, what is next?

154. kaggle.com

155. http://www.datasciencebowl.com/competitions/turning-machine-intelligence-against-lung-cance Team: Lauri Listak Supervisor: Dmytro Fishman Turning Machine Intelligence Against Lung Cancer

156. Turning Machine Intelligence Against Lung Cancer http://www.datasciencebowl.com/competitions/turning-machine-intelligence-against-lung-cance Team: Lauri Listak Supervisor: Dmytro Fishman

157. Turning Machine Intelligence Against Lung Cancer http://www.datasciencebowl.com/competitions/turning-machine-intelligence-against-lung-cance Team: Lauri Listak Supervisor: Dmytro Fishman 20% of lung cancer deaths can be reduced with early detection

158. High False Positives rates lead to interventional treatments, additional costs and patient anxiety 20% of lung cancer deaths can be reduced with early detection Turning Machine Intelligence Against Lung Cancer http://www.datasciencebowl.com/competitions/turning-machine-intelligence-against-lung-cance Team: Lauri Listak Supervisor: Dmytro Fishman

159. 2023

160. github.com/concept-to-clinic/concept-to- clinic

161. http://dreamchallenges.org/

162. References • Series of blog posts “Do machines actually beat doctors?” by Luke Oakden-Rayner (https://lukeoakdenrayner.wordpress.com/ 2016/11/27/do-computers-already-outperform-doctors/) • Opportunities and obstacles for deep learning in biology and medicine by Ching et al. (http://www.biorxiv.org/content/biorxiv/ early/2017/05/28/142760.full.pdf) • Computational biology - deep learning by William Jones, Kaur Alasoo, Dmytro Fishman et al. (accepted)

163. Contact: dmytro@ut.ee

Scientists meet Entrepreneurs - AI & Machine Learning, Dima Fishman, University of Tartu

Du liest eine Vorschau.

Hier ansehen

Scientists meet Entrepreneurs - AI & Machine Learning, Dima Fishman, University of Tartu

Weitere Verwandte Inhalte

Ähnlich wie Scientists meet Entrepreneurs - AI & Machine Learning, Dima Fishman, University of Tartu (20)

Weitere von MobileMonday Estonia (20)

Aktuellste (20)