Amit Sheth, Pramod Anantharam, Krishnaprasad Thirunarayan, "kHealth: Proactive Personalized Actionable Information for Better Healthcare", Workshop on Personal Data Analytics in the Internet of Things at VLDB2014, Hangzhou, China, September 5, 2014.
Accompanying Video: http://youtu.be/pqcbwGYHPuc
Paper: http://www.knoesis.org/library/resource.php?id=2008
kHealth: Proactive Personalized Actionable Information for Better Healthcare
1. kHealth: Proactive Personalized Actionable Information for
Better Healthcare
Put Knoesis Banner
PDA@IoT, in conjunction with VLDB, September, 2014
Amit Sheth, Pramod Ananthram, T.K. Prasad
The Ohio Center of Excellence in Knowledge-enabled Computing (Kno.e.sis)
Wright State, USA
2. 2
A Historical Perspective on Collecting Health Observations
Imhotep
Laennec’s stethoscope
Image Credit: British Museum
2600 BC ~1815 Today
Diseases treated only
by external observations
First peek beyond just
external observations
Information overload!
Doctors relied only on
external observations
Stethoscope was the
first instrument to go
beyond just external
observations
Though the stethoscope
has survived, it is only one
among many observations
in modern medicine
http://en.wikipedia.org/wiki/Timeline_of_medicine_and_medical_technology
3. “The next wave of dramatic Internet growth will come through the confluence of
people, process, data, and things — the Internet of Everything (IoE).”
- CISCO IBSG, 2013
Beyond the IoE based infrastructure, it is the possibility of developing applications that spans
Physical, Cyber and the Social Worlds that is very exciting.
3
http://www.cisco.com/web/about/ac79/docs/innov/IoE_Economy.pdf
What has changed now?
4. ‘OF human’ : Relevant Real-time Data Streams for Human Experience
Petabytes of Physical(sensory)-Cyber-Social Data everyday!
More on PCS Computing: http://wiki.knoesis.org/index.php/PCS
4
5. 6
MIT Technology Review, 2012
The Patient of the Future
http://www.technologyreview.com/featuredstory/426968/the-patient-of-the-future/
6. ‘FOR human’ : Improving Human Experience (Smart Health)
Weather Application
Asthma Healthcare
Application
Personal
Public Health
Detection of events, such as wheezing
sound, indoor temperature, humidity,
dust, and CO level
Close the window at home
during day to avoid CO in
gush, to avoid asthma attacks
at night
7
Population Level
Action in the Physical World
Luminosity
CO level
CO in gush
during day time
7. ‘FOR human’ : Improving Human Experience (Smart Energy)
Weather Application
Power Monitoring
Application
Personal Level Observations
Electricity usage over a day, device at
work, power consumption, cost/kWh,
heat index, relative humidity, and public
events from social stream
8
Population Level Observations
Action in the Physical World
Washing and drying has
resulted in significant cost
since it was done during peak
load period. Consider
changing this time to night.
8. kHealth
Knowledge-enabled Healthcare
Four current applications:
To reduce preventable readmissions of patients with
ADHF and GI; Asthma in children; patients with Dementia
9
10. Empowering Individuals (who are not Larry Smarr!) for their own health
Through physical monitoring and
analysis, our cellphones could act as
an early warning system to detect
serious health conditions, and
provide actionable information
canary in a coal mine
kHealth: knowledge-enabled healthcare
11
11. What?
• kHealth is a knowledge-based
approach/application for patient-centric
health-care that exploits:
(a) Web based tools and social media,
(b) Mobile phone technology and wireless sensors,
(c) For synthesizing personalized actions from
heterogeneous health data
(i) For disease prevention and treatment
(ii) For health, fitness and well-being
12
13. kHealth Kit for the application for Asthma management
Sensordrone
(Carbon monoxide,
temperature, humidity)
Node Sensor
(exhaled Nitric
Oxide)
15
Sensors
Android Device
(w/ kHealth App)
Total cost: ~ $500
*Along with two sensors in the kit, the application uses a variety of population level signals from the web:
Pollen level Air Quality
Temperature & Humidity
14. Why?
• “Unintelligible” health data deluge due to
– Continuous monitoring of patients using passive and
active sensors
– Continuous monitoring of environment using sensors
– Public health reports
– Population level information
– Social media conversations
– Personal Electronic Medical Records (EMRs)
– Wide use of affordable mobile/wireless technologies
19
15. Why?
• Empowering patients to improve health by
– Abstracting and integrating low-level sensor data
to more meaningful health signals
– Recommending personalized actions
• Ubiquitous, timely and effective health
management and telemedicine
– Involve patient and health-care team without
causing “interaction fatigue”
20
16. kHealth: Health Signal Processing Architecture
Personal level
Signals
Public level
Signals
Population level
Signals
Domain
Knowledge
Risk Model
Events from
Social Streams
Take Medication before
going to work
Contact doctor
Avoid going out in the
evening due to high pollen
levels
Analysis
Personalized
Actionable
Information
Data Acquisition &
aggregation
21
17. How?
• Data collection from various sources
– Active and passive sensing devices
– Social media crawling
– EMR
• Syntactic and semantic integration
– Qualitative/imprecise citizen observations
– Quantitative/precise sensor observations
• Provide complementary and collaborative information
• Using Semantic Web technologies, e.g., SemSOS
22
18. How?
• Semantic Perception: Reasoning for decision
making and action generation
– Perception cycle
– Personalized action recommendation using
• Patient health score (linear scale, RYG-abstraction)
• Patient vulnerability score (personalization)
– Qualify vs quantify
• Domain (e.g. disease) specific knowledge
23
19. 24
Asthma Domain Knowledge
Asthma Control
and Actionable Information
Domain
Knowledge
Asthma Control
à
Daily Medication
Choices for starting
therapy
Not Well Controlled Poor Controlled
Severity Level
of Asthma
(Recommended Action) (Recommended Action) (Recommended Action)
Intermittent Asthma SABA prn - -
Mild Persistent Asthma Low dose ICS Medium ICS Medium ICS
Moderate Persistent
Asthma
Medium dose ICS alone
Or with
LABA/montelukast
Medium ICS +
LABA/Montelukast
Or High dose ICS
Medium ICS +
LABA/Montelukast
Or High dose ICS*
Severe Persistent Asthma High dose ICS with
LABA/montelukast
Needs specialist care Needs specialist care
ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist ;
*consider referral to specialist
20. 25
Patient Health Score (diagnostic)
How controlled is my asthma?
Risk assessment
model
Semantic
Perception
Personal level
Signals
Public level
Signals
Domain
Knowledge
Population level
Signals
GREEN -- Well Controlled
YELLOW – Not well controlled
Red -- poor controlled
21. 26
Patient Vulnerability Score (prognostic)
How vulnerable* is my control level today?
Risk assessment
model
Semantic
Perception
Personal level
Signals
Public level
Signals
Domain
Knowledge
Population level
Signals
Patient health
Score
*considering changing environmental conditions and current control level
22. Background
Knowledge
31
Health Signal Extraction to Understanding
Physical-Cyber-Social System Observations Health Signal Extraction Health Signal Understanding
Personal
Population Level
Acceleration readings from
on-phone sensors
Wheeze – Yes
Do you have tightness of chest? –Yes
Risk Category assigned by
doctors
<Wheezing=Yes, time, location>
<ChectTightness=Yes, time, location>
<PollenLevel=Medium, time, location>
<Pollution=Yes, time, location>
<Activity=High, time, location>
PollenLevel
Wheezing
ChectTightness
Pollution
Activity
PollenLevel
Wheezing
ChectTightness
Pollution
Activity
RiskCategory
<PollenLevel, ChectTightness, Pollution,
Activity, Wheezing, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
.
.
.
Expert
Knowledge
Sensor and personal
observations
tweet reporting pollution level
and asthma attacks
Signals from personal, personal
spaces, and community spaces
Qualify
Quantify
Enrich
Outdoor pollen and pollution
Public Health
Well Controlled - continue
Not Well Controlled – contact nurse
Poor Controlled – contact doctor
23. 36
How are machines supposed to integrate and interpret sensor data?
RDF OWL
Semantic Sensor Networks (SSN)
24. 39
W3C Semantic Sensor Network Ontology
Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K.,
Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).
25. 41
What if we could automate this sense making ability?
… and do it efficiently and at scale
26. SSN
Ontology
2 Interpreted data
(deductive)
[in OWL]
e.g., threshold
1 Annotated Data
[in RDF]
e.g., label
0 Raw Data
[in TEXT]
e.g., number
Levels of Abstraction
3 Interpreted data
(abductive)
[in OWL]
e.g., diagnosis
Intellego
Hyperthyroidism
… …
Elevated
Blood
Pressure
Systolic blood pressure of 150 mmHg
“150”
42
28. People are good at making sense of sensory input
What can we learn from cognitive models of perception?
• The key ingredient is prior knowledge
44
29. Semantic Perception : Perception Cycle
Semantic perception in kHealth involves:
• Abductive reasoning to derive candidate
explanations for sensor data, and
• Deductive reasoning to disambiguate among
multiple explanations with patient inputs and
additional targeted sensor observations.
Intellego
45
30. Observe
Property
* based on Neisser’s cognitive model of perception
Perceive
Feature
Explanation
Discrimination
1
2
Perception Cycle*
Translating low-level signals
into high-level knowledge
Focusing attention on those
aspects of the environment that
provide useful information
Prior Knowledge
46
31. To enable machine perception,
Semantic Web technology is used to integrate
sensor data with prior knowledge on the Web
47
32. Prior knowledge on the Web
W3C Semantic Sensor
Network (SSN) Ontology Bi-partite Graph
48
33. Prior knowledge on the Web
W3C Semantic Sensor
Network (SSN) Ontology Bi-partite Graph
49
34. Explanation is the act of choosing the objects or events that best account for a
set of observations; often referred to as hypothesis building
Observe
Property
Perceive
Feature
Explanation
1
Explanation
Translating low-level signals
into high-level knowledge
50
35. Discrimination is the act of finding those properties that, if observed, would help distinguish
between multiple explanatory features
Observe
Property
Perceive
Feature
Explanation
Discrimination
2
Focusing attention on those
aspects of the environment that
provide useful information
Discrimination
51
37. Semantic Perception : Abstraction
• Mapping low-level sensor values to coarse-grain
abstract values
– E.g., Blood pressure: 150/100 => High bp
• Extracting signatures for high-level human
comprehensible features from low-level
sensor data stream.
– E.g., Parkinson disease : unsteady walk, fall,
slurred speech, etc.
53
38. How do we implement machine perception efficiently on a
resource-constrained device?
Use of OWL reasoner is resource intensive
(especially on resource-constrained devices),
in terms of both memory and time
• Runs out of resources with prior knowledge >> 15 nodes
• Asymptotic complexity: O(n3)
57
39. Approach 1: Send all sensor observations
to the cloud for processing
Approach 2: downscale semantic
processing so that each device is capable
of machine perception
intelligence at the edge
Henson et al. 'An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices,
ISWC 2012.
58
40. Efficient execution of machine perception
Use bit vector encodings and their operations to encode prior knowledge and
execute semantic reasoning
010110001101
0011110010101
1000110110110
101100011010
0111100101011
000110101100
0110100111
59
41. Evaluation on a mobile device
Efficiency Improvement
• Problem size increased from 10’s to 1000’s of nodes
• Time reduced from minutes to milliseconds
• Complexity growth reduced from polynomial to linear
O(n3) < x < O(n4) O(n)
60
42. Semantic Perception for smarter analytics: 3 ideas to takeaway
1 Translate low-level data to high-level knowledge
Machine perception can be used to convert low-level sensory
signals into high-level knowledge useful for decision making
2 Prior knowledge is the key to perception
Using SW technologies, machine perception can be formalized and
integrated with prior knowledge on the Web
3 Intelligence at the edge
By downscaling semantic inference, machine perception can
execute efficiently on resource-constrained devices
61
43. 62
thank you, and please visit us at
http://knoesis.org
Hinweis der Redaktion
Starting slide
Various Big data problems – Traditional examples vs what we are doing examples.
Variety and Velocity than Volume. kHealth problem. People will be interested in Smart Data.
Traditional ML techniques, High Performance Computing, Statistics. Human level of Abstraction is Smart data.
"2600 BC – Imhotep wrote texts on ancient Egyptian medicine describing diagnosis and treatment of 200 diseases in 3rd dynasty Egypt.”
Sir William Osler, 1st Baronet, was a Canadian physician and one of the four founding professors of Johns Hopkins Hospital. He was called the father of modern medicine. Sir William Osler called Imhotep as the true father of medicine.
There are over 99.4% of physical devices that may one day be connected to
The Internet still unconnected.
- CISCO IBSG, 2013
All the data related to human activity, existence and experiences
More on PCS Computing: http://wiki.knoesis.org/index.php/PCS
TKP: Should not knowledge be used to bridge the gap between data and decision and action?
Or are we saying we need to glean knowledge?
- Larry Smarr is a professor at the University of California, San Diego
And he was diagnosed with Chrones Disease
What’s interesting about this case is that Larry diagnosed himself
He is a pioneer in the area of Quantified-Self, which uses sensors to monitor physiological symptoms
Through this process he discovered inflammation, which led him to discovery of Chrones Disease
This type of self-tracking is becoming more and more common
Actionable information example:
In Asthma use case we have a sensor – sensordrone which records luminosity and CO levels
A high correlation between CO level and luminosity is found
This is an actionable information to the user interpreting it as CO in gush during day time
=> Mitigating action can be “closing the window” during day
Also, we have weather application which performs abstraction on weather sensory observations to identify blizzard conditions (food for actions!!) :
-- 20,000 weather stations (with ~5 sensors per station)
-- Real-Time Feature Streams
- live demo: http://knoesis1.wright.edu/EventStreams/
- video demo: https://skydrive.live.com/?cid=77950e284187e848&sc=photos&id=77950E284187E848%21276
- With this ability, many problems could be solved
- For example: we could help solve health problems (before they become serious health problems) through monitoring symptoms and real-time sense making, acting as an early warning system to detect problematic health conditions
ADHF – Acute Decompensated Heart Failure
1)www.pollen.com(For pollen levels)
2)http://www.airnow.gov/(For air quality levels)
3)http://www.weatherforyou.com/(For temperature and humidity)
Data overload in the context of asthma
“
Research on Asthma has three phases
Data collection: what signals to collect?
Analysis: what analysis to be done?
Actionable information: what action to recommend?
In the next slide, we take a peek into the analysis that we do for Asthma
What is the current state of a person/patient? => Summarizing all the observations (sensor and personal) into a single score indicating health of a person
Instead of presenting all the raw data (often to much e.g., Asthma application we have developed collects CO, temperature, and humidity every 10 seconds resulting in 8,640 observations/day) which may not be comprehensible to the patient, we empower them by providing actionable summaries.
What is the likely state of the person in future? => Given the current state and the changing environmental conditions, estimate the state of the person by summarizing it into a number which is actionable.
For example, vulnerability score for a person with Asthma is computed with environmental factors (pollen, air quality, external temperature and humidity) and current state of the patient.
Intuitively, a person with well controlled asthma should have a lower vulnerability score than a person with poorly controlled asthma both being in a poor environmental state.
In the absence of declarative knowledge in a domain, we resort to statistical approaches to glean insights from data
Even if there is declarative knowledge of a domain, it may have to be personalized
The CO level may be related to the luminosity as observed by the sensordrone – as it gets brighter the CO level also increases => high CO level in daytime
If such an insight is provided to a person, the interpretation can be:
Some activity inside the house leads to high CO levels
Outside activity leads to high CO levels inside the house
Since the person knows that he/she is absent in the house during mornings, it has to be something from outside.
- Person narrows down to a possible opened window at home (forgot to close more often)
There are two components in making sense of Health Signals:
Health signal extraction – processing, aggregating, and abstracting from raw sensor/textual data to create human intelligible abstractions
Health signal understanding – derive (1) connections between abstractions and (2)
Action recommendation:
Continue
Contact nurse
Contact doctor
Only score based structure extraction is presented here. Other popular structure extraction techniques include constraint based approaches which finds independences between random variables X1, …, Xn
I-Map => different structures result in the same loglikelihood score. Thus recovering the original structure of the graph generating data using data alone is considered impossible! We go the the rescue of declarative knowledge to: (1) choose promising structures and (2) to break ties when two structure results in the same score
Massive amount of data is collected by sensors and mobile devices yet patients and doctors care about “actionable” information.
This data has all the four Vs of big data and we used knowledge enabled techniques to transform it into value
In the context of PD, we analyzed massive amount of sensor data collected by sensors on a smartphones to understand detection and characterization of PD severity.
- what if we could automate this sense making ability?
- and what if we could do this at scale?
sense making based on human cognitive models
perception cycle contains two primary phases
explanation
translating low-level signals into high-level abstractions
inference to the best explanation
discrimination
focusing attention on those properties that will help distinguish between multiple possible explanations
used to intelligently task sensors and collect additional observations (rather than brute force approach of blindly collecting all observations)
compute machine perception inferences -- i.e., explanation and discrimination -- of high-complexity on a resource-constrained devices in miliseconds
Difference between the other systems and what this system provides
Intelligence at the age. Shipping computation and domain models to the edge (Distributed)
More at: http://wiki.knoesis.org/index.php/PCS
And http://knoesis.org/projects/ssw/