Deep Learning is proving to be a powerful tool that can improve healthcare for both patients and care-providers. In this talk I’ll cover an intro to some of the medical problems currently being solved by deep learning, market adoption, healthcare challenges (e.g regulation, data quality, data acquisition), deep learning challenges (e.g. model stability, training/convergence time, scalable training environment), and tips learned by tackling these problems head-on. This talk was presented Oct 15, 2017 at http://ai.withthebest.com/.

1. Medical Deep Learning:
Clinical, Technical, & Regulatory Challenges
and How to Overcome Them
Devon Bernard
VP of Engineering @ Enlitic

2. Radiology Pathology Genomics
Pharma Insurance Consultation
AI + Healthcare Landscape

3. Enlitic uses deep learning to help doctors
provide faster, earlier, and more accurate diagnostics.
39TH
39
SMARTEST
COMPANY
MIT Technology Review
2015
39TH
14
SMARTEST
COMPANY
MIT Technology Review
2016
WORLD
TECHNOLOGY
AWARD
Health and Medicine
2016
MOST
INNOVATIVE
COMPANIES
FastCompany
2016
€1M
GRAND
PRIZE
1st CUBE Global Fair
2017

4. In an internal study, Enlitic found that human radiologists take an average of 2.5 minutes (150 seconds) to interpret a chest x-ray. Enlitic’s models were found to perform the same task in 15ms. Human radiologists were measured as having an AUC of
0.72 at the task of determining whether or not a patient needed follow-up whereas Enlitic’s models had an AUC of 0.84. Performance was measured against a set of 2000 chest x-rays each read by 8 radiologists with consensus used as ground truth.
Generates a differential diagnosis from a chest x-ray.
Used as a triage tool separating normal from abnormal, or for QA.
Interprets chest x-rays
10,000 times faster
than human radiologists.
16% more accurate than
radiologists and trained to look
for rare/orphan diseases.
Chest X-ray Interpretation and Triage

5. Assists radiologists in detecting, further characterizing, and
diagnosing suspicious pulmonary lung nodules in Chest CTs
50% more accurate
than a panel of
expert radiologists.
Capable of finding cancer
2 years earlier
than radiologists.
Currently in clinical study
Chest CT Lung Cancer Screening
In an internal study, Enlitic achieved results that surpassed by 50% a panel of 4 radiologists’ performance in malignancy classification. The study showed that panel of 4 human radiologists had the precision of 33.7% and recall of 93.0%, while Enlitic's DL
models exhibited the precision of 56.3% and recall of 100%. This study was performed using the Lung Image Database Consortium (“LIDC”) dataset with about 1,000 studies.

6. 99%+ accuracy in reading
radiology reports in focus areas
(chest XR and CT).
Detects diagnostic / procedural
info and tags as SNOMED,
ICD9/10, and CPT codes.
Can be used alongside imaging
models to detect overbilled
procedures.
Uses Natural Language Processing to assist medical coders in
assigning insurance and billing codes from medical reports.
Medical Billing and Coding

7. Clinical Technical Regulatory
Challenges

8. Clinical Challenges

9. Acquiring Data
Where can I get data?
• Public datasets
• National Institute of Health
• National Cancer Institute
• Cancer Imaging Archive
• Research Clinics
• Strategic Partners
• ClinicalTrials.gov
How can I copy data?
• Might require anonymization
• Partner will likely request you lead
• Check for existing tools & standards
• Might be distributed
• RIS, PACS, EMR
• multiple archives over years
• Are transfers a pull or push?
• Might be slow
• cloud transfer
• ship SSD

10. Defining Labeling Criteria
Is our business use-case valuable?
• Get a clinical research adviser
Is our business use-case practical?
• Get a clinical research advisor
What should our target problem be?
• Get a clinical research adviser
Find Business
Use-Case
Define
Target Problem
Define
Labeling Criteria
Collect
Labels
Train
Models
What labels should we collect?
• Get a clinical research advisor
Are we collecting good labels?
• Get a clinical research adviser
.
.

11. Getting Labelers
Where to hire labelers?
• Remote part-time / contractors
• Residency programs
How to vet labelers?
• Test against golden dataset
• Consistent over time
• Inter-labeler agreement
How to hire labelers?
• $$$, equity
• research credit, publications
• Man + Machine

12. Collecting Labels
What tools to collect labels?
• 3rd party software
• Crowdflower
• Hacks
• Spreadsheets & Screenshares
• Develop in-house tools
How many labels to collect?
• Define budget, stages, & check-ins
upfront
Are the labels good?
• Quality Assessment
• Cross-reference
• Test model impact

13. Technical Challenges

14. General Model Training
How to optimize resource usage?
• Job Scheduler (Slurm)
• Track GPU usage
• Consider on-site vs cloud
Debugging tips
• Python profilers
• Python debugger (pdb)
How to track progress?
• Centralized perf. reporting
• Archive version/iterative perf.
• Slack webhook notifications

15. Medical Model Training
How to manage big scans (CTs, MRIs)?
• Might need more RAM
• Cache data locally
• Train near storage array
How to securely contain PHI?
• Only train on non-PHI data
• Provide separate PHI & non-PHI VPNs
• Hard split on all services

16. Medical Model Testing
Are our models truly generalized?
• Test against multiple data sources
Will our model work with new partners?
• Calibrate on partner data
• Scanning procedures can differ by facility
• Diagnosis criteria can differ by facility
• Do blind tests with potential partners
Is our model output drifting?
• Check if partner’s data
distribution has changed

17. Regulatory Challenges

18. Getting Medical Classification
Do we need regulatory approval?
• Check national specifications
• Compare for each use-case
• Typically based upon patient risk
• Hire regulatory consultants
How to find good consultants?
• Ask your potential partner for
suggestions
Do we have to wait for approval?
• Ask partner for iterative phases
• Start with historical / retrospective
work
• Progress toward more invasive
integrations
• Ask partner about a clinical trial

19. Other Certifications / Red-Tape
Prerequisites to consider
• ISO
• Privacy Impact Assessments
• Threat Risk Assessments
• Penetration Testing
• Data Security Officer
• Document policies & procedures
• Incident tracking
• Monitoring compliance
Processes to consider
• HIPPA training for all employees
• Partner data sharing agreements
• Establish criteria around what a
partner is willing to share
• Prevent unexpected future pauses
• Partner contractor agreements
• Establish what internal employees
have access to which partner
facilities, why, and for how long

20. Devon Bernard
VP of Engineering @ Enlitic
devon@enlitic.com
@devonwbernard
Thank you!
Any questions?