Sepsis is one of the top causes of inpatient mortality and rapid detection presents numerous challenges. In March, 2016, an interdisciplinary team consisting of top clinicians, data scientists and machine learning experts at a large academic medical center (AMC) embarked on an innovation pilot to develop a novel machine learning model to detect sepsis. A computable sepsis definition and deep learning model were developed using a curated dataset capturing over 43,000 inpatient admissions between October 1, 2014 and December 31, 2015. Ten computable sepsis definitions were compared and our clinicians agreed on the following: >= 2 SIRS criteria, blood culture order, and end organ damage. This sepsis phenotype identified patients early in the hospital course: 38% of cases occur an average of 1.3 hours after presentation to the ED and 42% of cases occur an average of 15 hours after hospital admission. At 4 hours prior to sepsis, the best deep learning model generated 1.4 false alarms per true alarm at a sensitivity of 80%, compared to 3.2 false alarms per true alarm for National Early Warning System (NEWS). Purpose Sepsis Watch detects sepsis early, guides completion of appropriate treatment, and supports front-line providers with minimal interruption of clinical workflows. Key Performance Indicators include emergency department (ED) length of stay, hospital length of stay, inpatient mortality, intensive care unit requirement, and time to antibiotics for patients who develop sepsis. Description The core technology components of Sepsis Watch are web services to extract electronic health record (EHR) data in real-time, a data pipeline to normalize features, a computable sepsis definition, a deep learning sepsis prediction model, a web application (Figure 1), an automated report that calculates KPI performance, and a model input and output monitoring tool. A suite of education, training, communication, and workflow materials were also prepared with nurse educators and are hosted on an intranet training site. After a three-month silent period, Sepsis Watch was deployed in the ED of the 1,000 bed flagship hospital on November 5, 2018. Conclusions Sepsis Watch is the first deployment of deep learning model in real-time to detect sepsis integrated with an EHR. The tool is used by Rapid Response Team (RRT) nurses to provide proactive support to ED providers to identify and manage sepsis. A six-month clinical trial will be completed in May 2019 to rigorously assess the clinical and operational impact of the program.

1. Proprietary and ConfidentialDIHI
Sepsis Watch: a deep
learning sepsis detection
and treatment platform
DUKE University
School of Medicine
DUKE Institute for
Health Innovation
September 2019
Mark Sendak, MD, MPP
Population Health & Data Science Lead
Duke Institute for Health Innovation

2. Proprietary and ConfidentialDIHI
Development of Deep Learning Sepsis Model
• 42,000+ inpatient encounters at Duke Hospital
over 14 months, 21.3% with a sepsis event; no
specific inclusion/exclusion criteria.
• 34 physiological variables (5 vitals, 29 labs).
– At least one value for each vital in 99% of
encounters.
– Some labs rarely measured (2-4%), most
measured 20-80% of the time.
• 35 baseline covariates (e.g. age, transfer
status, comorbidities).
• 10 medication classes (antibiotics, opioids,
heparins).
• 32+ million data points: 25 million vital sign
measurements, 2 million med admins and 5.2
million labs.
Dataset Design & Implementation Team
User Interface
Design
Machine
Learning
Clinical
Informatics
Hospital
Medicine
Critical
Care
Infectious
Diseases
Data
Engineering
Emergency
Medicine
Nursing
Champions
include Hospital
Presidents,
CMOs, CIO

3. Proprietary and ConfidentialDIHI
2 or more SIRS criteria
• Temperature >38°C or <36°C (6 hours)
• HR >90 (6 hours)
• RR >20 (6 hours)
• WBC count >12, <4, or % bandemia >10% (24 hours)
Suspicion for infection • Blood culture order (24 hours)
1 element of end
organ failure
• Creatinine >2.0 (24 hours)
• INR >1.5 (24 hours)
• Total bilirubin >2/0 (24 hours)
• SBP <90 or decrease in SBP by >40 (6 hours)
• Platelets <100 (24 hours)
• Lactate ≥2 (24 hours)
Duke Consensus Adult Sepsis Definition

4. Proprietary and ConfidentialDIHI
SIRS ≥2 qSOFA ≥2
SIRS ≥2 +
any culture
ordered
SIRS ≥2 + any
culture ordered
+ element of
organ damage
SIRS ≥2 +
blood culture
ordered +
element of
organ damage
qSOFA ≥2 +
any culture
ordered
ICD
diagnosis
code
associated
with sepsis
SIRS ≥2 +
bacteremia
Total
# of encounters 32928 17423 14327 13358 9184 7110 2884 1419 43046
Median length of stay
in days (lower-upper
quartiles)
4.6
(2.8-8.1)
5.9
(3.2-10.7)
6.4
(3.7-12.1)
6.9
(3.9-12.8)
7.3
(4.1-14.6)
8.3
(4.5-16.3)
7.5
(4.1-15.4)
11.0
(5.9-23.7)
4.0
(2.4-7.0)
Inpatient mortality
rate (%)
3.7% 6.7% 6.9% 7.4% 9.7% 12.6% 16.3% 15.0% 2.9%
ICU requirement rate
(%)
21.3% 32.0% 28.7% 30.0% 34.5% 45.0% 46.4% 38.9% 18.9%
Antibiotic
administration rate
(%)
62.4% 69.0% 82.8% 83.2% 90.0% 85.5% 98.5% 97.8% 63.2%
IV fluid administration
rate (%)
38.0% 37.8% 47.4% 48.5% 56.7% 49.6% 86.7% 67.1% 42.4%
Vasopressor
administration rate
(%)
10.2% 17.1% 15.0% 16.0% 19.4% 27.3% 32.8% 28.8% 9.6%
Balancing Disease Severity & Opportunity

5. Proprietary and ConfidentialDIHI
Distribution of sepsis events by
number of hours after admission
Distribution of sepsis events per day by
ED (40%) vs inpatient (60%) setting
Hours Before or After Admission that
Patients Develop SepsisFrequency
Frequency
Number of Sepsis Events per Day
Setting
Inpatient
ED
Deeper Dive Into Data – Sepsis at DUH

6. Proprietary and ConfidentialDIHI
Duke Raleigh
Hospital
Duke Regional
Hospital
Duke University
Hospital
Beds 186 369 957
Encounters 22987 32082 42806
Sepsis Cases 571 (2.5%) 870 (2.7%) 2674 (6.2%)
Daily Sepsis Cases 3.1 cases / day 4.7 cases / day 14.5 cases / day
Sepsis Cases in ED 391 (68.5%) 585 (67.2%) 1241 (46.4%)
Time to Sepsis in ED 1.92 hours 1.83 hours 2.01 hours
adult encounters to 3 hospitals between March 1, 2018 – August 31, 2018
Rapid Identification Key Across System

7. Proprietary and ConfidentialDIHI
Labs & Vitals
Multitask
Gaussian
Process
Smoothed &
imputed labs
& vitals on
regular grid
Deep Recurrent
Neural Network
MGP params
Network params
Meds
Baseline
Covariates
Predicted
probability of
sepsis
Actual sepsis
indicator
Model loss function
all
encounters
End-to-end learning!
Deep Learning Model Architecture

8. Proprietary and ConfidentialDIHI
Deep Recurrent Neural Network
RNN params
: Lab 1
: Lab 2
: Baseline
: Medication
: Grid Time
Current
risk
X X
Gaussian process
imputes & interpolates,
maintaining uncertainty
Powerful deep learning
prediction model
Combining Multimodal Data and Deep Learning

9. Proprietary and ConfidentialDIHI
Deep Recurrent Neural Network
RNN params
: Lab 1
: Lab 2
: Baseline
: Medication
: Grid Time
Current
risk
X X
Gaussian process
imputes & interpolates,
maintaining uncertainty
Powerful deep learning
prediction model
Combining Multimodal Data and Deep Learning

10. Proprietary and ConfidentialDIHI
Deep Learning Model Operating Performance
upcom
ing
publication

11. Proprietary and ConfidentialDIHI
Deep Learning Model Operating Performance
upcom
ing
publication

12. Proprietary and ConfidentialDIHI
ØSupport primary providers
…without causing alarm-fatigue
ØImprove patient care over entire cycle
high risk through treatment
Epic
Deep
Learning
Model
User Interface
ØLeverage Sepsis Care Team
RRT nurses and hospitalists
Custom Workflow to Reduce Alarm Fatigue

13. Proprietary and ConfidentialDIHI
*An RRT or
patient
consult is the
top priority of
the RRT
nurse

14. Proprietary and ConfidentialDIHI
Triage
• Sepsis identified every 5 minutes and
sepsis risk computed every hour
• System normalizes data, groups clinically
related concepts into meaningful features,
and ensures valid inputs to deep learning
model
• Deployed on-premise cloud with Docker
containers
• Black = meets sepsis criteria
• Red = high risk of sepsis
Sepsis Watch User Interface

15. Proprietary and ConfidentialDIHI
Triage Monitor
Sepsis Watch User Interface

16. Proprietary and ConfidentialDIHI
Triage Monitor Treat
Sepsis Watch User Interface

17. Proprietary and ConfidentialDIHI
5%
22%
24%
36%
17%
20%
28%
37%
37%
50%
39%
60%66%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Q1,
2016
Q2,
2016
Q3,
2016
Q4,
2016
Q1,
2017
Q2,
2017
Q3,
2017
Q4,
2017
Q1,
2018
Q2,
2018
Q3,
2018
Q4,
2018
Q1,
2019
SepsisWatchGoLive
Sepsis Treatment Compliance

18. Proprietary and ConfidentialDIHI
• Sepsis Definition
– Poster Presentation, American Thoracic Society 2018: “What is Sepsis: Investigating the
Heterogeneity of Patient Populations Captured by Different Sepsis Definitions.”
– Manuscript pre-print: https://www.biorxiv.org/content/10.1101/648907v1
• Deep Learning Model
– Plenary Presentation, Society of Hospital Medicine 2018: “Deep Personalized Medicine: Bringing
Deep Learning to Sepsis Care”
– Manuscript, International Conference on Machine Learning 2017: “Learning to Detect Sepsis with a
Multitask Gaussian Process RNN Classifier”
– Manuscript, Machine Learning in Health Care 2017: “An Improved Multi-Output Gaussian Process
RNN with Real-Time Validation for Early Sepsis Detection”
– Manuscript under review
• Implementation
– Oral Presentation, Machine Learning in Health Care 2018: “Leveraging Deep Learning and Rapid
Response Team Nurses to Improve Sepsis Management”
– Poster Presentation, Society of Hospital Medicine 2019: ““Integration of Sepsis Watch, a Deep
Learning Sepsis Detection and Treatment Platform, Into Clinical Care”
– Manuscript under review (pre-print: https://www.jmir.org/preprint/15182)
Peer Review Evidence

19. Proprietary and ConfidentialDIHI
• Sepsis Definition
– Poster Presentation, American Thoracic Society 2018: “What is Sepsis: Investigating the
Heterogeneity of Patient Populations Captured by Different Sepsis Definitions.”
– Manuscript pre-print: https://www.biorxiv.org/content/10.1101/648907v1
• Deep Learning Model
– Plenary Presentation, Society of Hospital Medicine 2018: “Deep Personalized Medicine: Bringing
Deep Learning to Sepsis Care”
– Manuscript, International Conference on Machine Learning 2017: “Learning to Detect Sepsis with a
Multitask Gaussian Process RNN Classifier”
– Manuscript, Machine Learning in Health Care 2017: “An Improved Multi-Output Gaussian Process
RNN with Real-Time Validation for Early Sepsis Detection”
– Manuscript under review
• Implementation
– Oral Presentation, Machine Learning in Health Care 2018: “Leveraging Deep Learning and Rapid
Response Team Nurses to Improve Sepsis Management”
– Poster Presentation, Society of Hospital Medicine 2019: ““Integration of Sepsis Watch, a Deep
Learning Sepsis Detection and Treatment Platform, Into Clinical Care”
– Manuscript under review (pre-print: https://www.jmir.org/preprint/15182)
Peer Review Evidence

20. Proprietary and ConfidentialDIHI
• Sepsis Definition
– Poster Presentation, American Thoracic Society 2018: “What is Sepsis: Investigating the
Heterogeneity of Patient Populations Captured by Different Sepsis Definitions.”
– Manuscript pre-print: https://www.biorxiv.org/content/10.1101/648907v1
• Deep Learning Model
– Plenary Presentation, Society of Hospital Medicine 2018: “Deep Personalized Medicine: Bringing
Deep Learning to Sepsis Care”
– Manuscript, International Conference on Machine Learning 2017: “Learning to Detect Sepsis with a
Multitask Gaussian Process RNN Classifier”
– Manuscript, Machine Learning in Health Care 2017: “An Improved Multi-Output Gaussian Process
RNN with Real-Time Validation for Early Sepsis Detection”
– Manuscript under review
• Implementation
– Oral Presentation, Machine Learning in Health Care 2018: “Leveraging Deep Learning and Rapid
Response Team Nurses to Improve Sepsis Management”
– Poster Presentation, Society of Hospital Medicine 2019: ““Integration of Sepsis Watch, a Deep
Learning Sepsis Detection and Treatment Platform, Into Clinical Care”
– Manuscript under review (pre-print: https://www.jmir.org/preprint/15182)
Peer Review Evidence

21. Proprietary and ConfidentialDIHI
• Sepsis Definition
– Poster Presentation, American Thoracic Society 2018: “What is Sepsis: Investigating the
Heterogeneity of Patient Populations Captured by Different Sepsis Definitions.”
– Manuscript pre-print: https://www.biorxiv.org/content/10.1101/648907v1
• Deep Learning Model
– Plenary Presentation, Society of Hospital Medicine 2018: “Deep Personalized Medicine: Bringing
Deep Learning to Sepsis Care”
– Manuscript, International Conference on Machine Learning 2017: “Learning to Detect Sepsis with a
Multitask Gaussian Process RNN Classifier”
– Manuscript, Machine Learning in Health Care 2017: “An Improved Multi-Output Gaussian Process
RNN with Real-Time Validation for Early Sepsis Detection”
– Manuscript under review
• Implementation
– Oral Presentation, Machine Learning in Health Care 2018: “Leveraging Deep Learning and Rapid
Response Team Nurses to Improve Sepsis Management”
– Poster Presentation, Society of Hospital Medicine 2019: ““Integration of Sepsis Watch, a Deep
Learning Sepsis Detection and Treatment Platform, Into Clinical Care”
– Manuscript under review (pre-print: https://www.jmir.org/preprint/15182)
Peer Review Evidence

22. Proprietary and ConfidentialDIHI
Team Data Science
2
Students
Masters, Statistics
Brian Cozzi
Medical Students
Nathan Brajer
Staff Clinicians
Cara O’Brien, MD
Armando Bedoya, MD,
MMCi
mark.sendak@duke.edu