Presenter: Konstantin Pogorelov, Simula Research Laboratory, University of Oslo, Norway Paper: http://ceur-ws.org/Vol-1984/Mediaeval_2017_paper_3.pdf Video: https://youtu.be/V2vFNXKSFrM Authors: Michael Riegler, Konstantin Pogorelov, Pål Halvorsen, Carsten Griwodz, Thomas de Lange, Kristin Ranheim Randel, Sigrun Losada Eskeland, Duc-Tien Dang-Nguyen, Mathias Lux, Concetto Spampinato Abstract: The Multimedia for Medicine Medico Task, running for the first time as part of MediaEval 2017, focuses on detecting abnormalities, diseases and anatomical landmarks in images captured by medical devices in the gastrointestinal tract. The task characteristics are described, including the use case and its challenges, the dataset with ground truth, the required participant runs and the evaluation metrics.

1. Multimedia for Medicine:
The Medico Task
Michael Riegler, Konstantin Pogorelov, Pål Halvorsen, Carsten Griwodz,
Thomas de Lange, Kristin Ranheim Randel, Sigrun Losada Eskeland,
Duc-Tien Dang-Nguyen, Mathias Lux, Concetto Spampinato
Email: michael@simula.no, konstantin@simula.no

2. Medico
 Bringing together IT and medicine, focusing on detecting
abnormalities, diseases and anatomical landmarks in images captured
by medical devices in the gastrointestinal tract
 Goals of the task: Help to improve the health care system using
multimedia methods to reach the next level of multimedia-assisted
computer diagnosis, detection and interpretation of abnormalities
 Attract more researchers to medical use cases

3. GI Tract Challenges
 Many types of diseases can affect the human digestive system
 Screening of the gastrointestinal (GI) tract using different
types of traditional endoscopy
– is costly (colonoscopy: US - $1100/patient, $10 billion dollars)
– consumes valuable medical personnel time (1-2 hours)
– does not scale to large populations
– is intrusive to the patient
 Current developments in technology may potentially enable
automatic algorithmic screening and assisted examinations
 a true interdisciplinary activity with high chances of societal impact

4. Colorectal Cancer Mortality 2012
 Most common cancer for men in Norway
 Second most common cancer for women in Norway

5. Live Automatic Detection
 System to assist doctors during live
traditional endoscopy procedures
 Second pair of eyes
 Support for inexperienced doctors
 Automatic tagging of lesions
 Automated reports generation
 Better procedure documentation
[1] van Rijn, J. C., Reitsma, J. B., Stoker, J., Bossuyt, P. M., van Deventer, S. J., and Dekker, E.
Polyp miss rate determined by tandem colonoscopy: a systematic review.
The American journal of gastroenterology 101, 2 (2006)

6. Wireless Video Capsule (Capsular Endoscopy)
 better scale
 less intrusive
 possible to combine
examinations!?
 less expensive?
(detection might lead to an endoscopy)
 expensive
 does not scale
 intrusive

7. Our Goals
 A complete system for:
– Live Traditional Endoscopy
– Capsular Endoscopy
 Automatic detection of different abnormalities in the
digestive system
– HD sources
– Real-time and faster
– High recall and precision
– Automated reports generation

8. EIR Overview

9. Common Challenges
 Blurry images due to camera motion
 Objects too close to camera
 Under or over scene lighting
 Flares
 Artificial objects and natural "contaminants“
 Low resolution of Capsular Endoscopes
 No proper support for medical reports generation

10. Common Challenges

11. Subtasks
 Detection
 Detection of different diseases and landmarks
 Use as few images in the training dataset as possible
 Efficient detection
 Solve the classification problem as fast as possible
 Efficient detection (fare evaluation, the same hardware)
 Evaluate detection algorithms on the same hardware
 Report generation (Experimental)
 Automatically create a text-report for a medical doctor for three
video cases

12. Dataset
 Hard to find annotated data allowed to use (even) for research
 Two data sets have been published at the MMSys 2017 open
dataset track: Kvasir and Nerthus (bowel preparation quality)
 Kvasir: the open source dataset consist of 8,000 annotated GI tract
images in 8 different classes: 500 in dev. + 500 in test set per class

13. Participants and submissions
 12 teams registered and requested the data
 5 teams successfully submitted the results
 Austria, China, India, Italy, Norway, Pakistan, US…
 46 submissions in total
 Detection – Successful
 Efficient detection – Partially Successful
 Efficient detection – No-Show
 Report generation – No-Go

14. The Approaches
 Machine Learning
– Manifold learning
– Support vector machine
– Random forest and random tree
– Logistic model tree
– Regression-based
– Unsupervised clustering
 Convolutional Neural Networks
– Direct classification
– Transfer learning
 Features Extraction
– Deep features
– Global and Local features

15. Metrics
 Multi-class generalization of Matthews correlation coefficient (R_K
statistic)
– Correlation coefficient between the observed and predicted
classifications
– Perfect for unbalanced multi-class case
– Maximum value +1 for perfect prediction
– Minimum value between -1 and 0 depending on the true
distribution
• Frames per second (FPS)

16. The Results – Best Performing
Team Training
set size
Multi-class
MCC
F1 FPS
SCL-UMD 3200 0.827 0.848 1.3
FAST-NU-DS 4000 0.736 0.767 2.3
ITEC-AAU 3600 0.724 0.755 1.4
HKBU 1000 0.663 0.703 2.2
SIMULA 4000 0.802 0.826 46.0
Random - -0.001 0.124 -
ZeroR - 0 0.125 -

17. The Results – Best Submission Confusion Matrix
Predicted class
Actual class
polyps
normal-
cecum
normal-z-
line
normal-
pylorus
esophagitis
dyed-
resection-
margins
dyed-
lifted-
polyps
ulcerative
-colitis
polyps 448 13 1 0 0 0 23 33
normal-cecum 22 478 0 0 0 0 0 27
normal-z-line 0 0 427 8 202 0 0 0
normal-pylorus 4 0 5 480 5 0 0 0
esophagitis 0 0 67 10 293 0 0 2
dyed-resection-
margins 0 0 0 0 0 406 55 1
dyed-lifted-
polyps 2 0 0 0 0 94 421 0
ulcerative-colitis 24 9 0 2 0 0 1 437

18. The Results – Smallest Training Set
Team Training
set size
Multi-class
MCC
F1
ITEC-AAU 400 0.607 0.649
FAST-NU-DS 732 0.649 0.689
HKBU 800 0.648 0.692
SCL-UMD 3200 0.827 0.848
SIMULA 4000 0.802 0.826
Random - -0.001 0.124
ZeroR - 0 0.125

19. Conclusions
1. Good classification performance achieved
2. Small amount of training data would not stop us!
3. Real-time is still challenging
4. Combined approaches are required for medical image
analysis
5. Medical task is interesting for the community

20. Future of the task
1. Localization/segmentation of findings and lesions
2. Exploiting domain expert knowledge – more data!
3. Integration of various data, multi-modality – new sensors,
doctors’ records, audio recordings, patient context
information
4. Automated reporting

21. Thank You! Questions?
All the data is released publicly and available at:
http://datasets.simula.no/kvasir/