1. 18 July 2013
Ho-Jin Choi
Dept. of Computer Science, KAIST
Systems Biomedical Informatics Research
Center (SBI-NCRC), SNU College of Medicine
Personalized Context-Aware Health Avatar in
Smart Phone Environment
The 2013 STI Semantic Summit
Suzdal, Russia, July 17-19, 2013
1

2. Outline
 SBI-NCRC – A brief introduction
 Research Topics
 Activity Recognition for Personalized Life-Care
 Healthcare Service Framework for Continuous Context
Monitoring
 Text Mining for Extracting Knowledge from Web Contents
 Personalized Bio and Medical Data Analysis
2

3. Systems Biomedical Informatics National
Core Research Center (SBI-NCRC)
- A brief introduction -
3

4. SBI-NCRC
 NCRC (National Core Research Center)
 Government initiative to support interdisciplinary research & education
 Since 2004, one or two centers newly selected each year
 Funding scale, 2 million USD/year * 6.5 years
 Systems Biomedical Informatics (SBI) Research Center
 An NCRC established jointly by SNU Hospital and KAIST Computer Science
 Born in September 2010
 24 professors/researchers participating from 4 organizations
 SNUH, KAIST, Ajou University, ETRI
 Goals for SBI-NCRC
 To define and realize “Digital Self” or “Health Avatar” prescriptive medicine
 To integrate clinical information and bio-information using IT
 To launch an interdisciplinary program in Biomedical Informatics
 To collaborate towards Joint KAIST-SNUH BIT Campus in Inchon area
4

5. 4P Medicine
 Preventive medicine
 Predictive medicine
 Personalized medicine
 Participatory medicine
Tests for early detection
Risk evaluation Prevention
Targeted
monitoring
Diagnosis Treatment
Results
monitoring
Caring Diseases Caring Health
5

6. Middleware
Integrative analyses
OCS PACS EMR LIS Seq. Exp. Prot. Tissue CGH
HL-7
DICOM
CDA
LOINC
BSML
MAGE
MIAPE
TMA
?
SNP
HapMap
2DPage
ProtChip
Tissue
MA
BAC
Chip
Phenomic Self Extractor Genomic Self Extractor
Clinical Genomics
BioData Acquisition
Pattern
Recognizer
DNA
Chip
Transformer
XML Binder
DBConnector
HL-7 / CDA Protocols
Hospital
caBIO
EVS caDSR
CRF CDE/CTEP
caCORE
IDE
Foundation Self
Warehouse
유방암
Application
폐암
Application
혈액암
Application
Legacy System
Authentication/Authorization
Interface
자료 수용기
CGI-
gateway
WebServer
XML-Validation
Ontology-Enhancement
Data Indexing
Clinical Trials Knowledge
Base
XML
CGI-gateway
Retrieval
engine
Query
Constructor
Clinical Research
& Clinical Trial KB
Application
Processor
Search engine
Statistical analysis
Visualization
Simulation
Communications
Workflow
Middleware
Ontology Server
Vocabulary Server
Taxonomy Server
Public Bio-DBs
Digital Self
Simulated Self
Individuated Second Self
Foundation Self
Molecular & Cellular Foundations of Self
Ubiquitous Self
Life Logs and Distributed Collaborations
Genomic Self: Translational Bioinformatics
for Genomic Health and Molecular Medicine
Phenomic Self: Data and Measurement driven
Discovery and Understanding of Human Disorders
Physiomic Self: Multi-scale Modeling of Physical and
Physiological Systems of Human Body
Semantic Self: Ontological Representation
and Engineering of Health Avatar
Augmented Self: Multi-modal Assessment and
Treatment to Retain and Enhance Human Performance
Connected Self: Life Logs and Stream-
Type Data Mining for Health Protection
Distributed Self: Customized and Context-aware
Healthcare Service Agents in Smart Phone Environment

7. Teaming
7
Group/Project Title PI’s Major
Group 1
Foundation Self: Molecular and Cellular Foundations
of Self
SNUH, Ajou U.
Project 1-1
Genomic Self: Translational Bioinformatics for Genomic
Health and Molecular Medicine
Psychiatry(1), Surgery(1),
Bioinformatics(1), Statistics(1)
Project 1-2
Phenomic Self: Data and Measurement driven
Discovery and Understanding of Human Disorders
Pathology(2), Bioinformatics(1)
Project 1-3
Physiomic Self: Multi-scale Modeling of Physical and
Physiological Systems of Human Body
Biomedical Engineering(2),
Neurosurgery(1), General
Practice(1)
Group 2 Simulated Self: Individuated Second Self SNUH, KAIST
Project 2-1
Semantic Self: Ontological Representation and
Engineering of Health Avatar
Nursing Informatics(2), Internal
Medicine(1), Pathology(1)
Project 2-2
Augmented Self: Multi-modal Assessment and
Treatment to Retain and Enhance Human Performance
NLP(1), Graphics(1), Image
Processing(1), Psychiatry(1)
Group 3
Ubiquitous Self: Life Logs and Distributed
Collaborations
KAIST, ETRI
Project 3-1
Distributed Self: Customized and Context-aware
Healthcare Service Agents in Smart Phone Environment
AI(1), Software Engineering(1),
Bioinformatics(1)
Project 3-2
Connected Self: Life Logs and Stream-Type Data Mining
for Health Protection
Information Systems(1), Data
Mining(1)

8. Life-style
Logging
Medical
Logging
Genetic
Logging
Virtual Self
Supercomputer
(Collective
Intelligence, Data
Mining)
Healthainment
Smart EMR
SNS
Health
Avatars
*
*Autonomous Health Avatar
Virtual Self and Health Avatar
8

9. 9
 Target healthcare domains
 Obesity, diabetes, dementia
 On-going research topics
 Activity Recognition for Personalized Life-Care
(Prof. Ho-Jin Choi)
 Healthcare Service Framework for Continuous
Context Monitoring (Prof. Jun-Hwa Song)
 Text Mining for Extracting Knowledge from Web
Contents (Prof. Key-Sun Choi)
 Personalized Bio and Medical Data Analysis (Prof.
Gwan-Su Yi)
Research Topics

10. Activity Recognition for Personalized
Life-Care
Prof. Ho-Jin Choi
Dept. of Computer Science
KAIST
10

11. Multi-Sensor Surveillance for Elderly Care
11
“Patient #1234 is
in a risky situation”
Data observed from microphones helps
the system detect the potentially risky
situations .
The agent estimates patient #1234’s behaviors.
When preliminary conditions of
dangerous situations are occurred
to the patient, the agent alarms to
the caregiver.

12. 12
Activity Recognition from Video Image with Depth Sensor
 Action Recognition with Automatically Detected Essential Body Joints

13. Technologies Involved
Understand Image Data
- RGB images (camera) and
depth images (depth sensor)
are sent to the system
- System then do
-Find a patient in a scene
-Track the patient
-Understand behaviors of
the patient
★ Issues to challenge
- The level of complexity
of scenes and behaviors
- Scenes may contain
various objects and
backgrounds
- Human-behaviors should
be understood as much
as possible.
13
Understand Audio Data
-Audio data (microphones)
is sent to the system
-System then do
-Detect abnormal sounds
★ Issues to challenge
- How accurate the system
detects abnormal sounds
Detect Risky Situation
-After analyzing data from
various sensors, the system
determines whether the
situation is potentially risky
-System constructs a
database for predefined
risky situations
-For every situation, the
system calculates the
likelihood of being risky
-If the likelihood scores
more than a threshold, it
alarms to the caregiver
★ Issues to challenge
-How well the system
constructs the database
-The accuracy of likelihoods
Find Patient’s Location
-Smartphone gives and
receives various signals to
update patient’s geographic
information
★ Issues to challenge
- How accurately the system
locates the patient

14. Wrist-Type Device Based Human Behavior Recognition
14
 Mediated Interface for human-robot interaction
….
Health Care
Care Services
Raw data
(Behavior pattern,
Vital Signal, etc)
Old People
How to get “Raw Data”
From Old People?
Robots
Care-giver
Activity, Gesture,
Vital signal,
Location,
Identification(MI: Mediated Interface)
Ex: Watch, Ring
Robots
Care-giver
Elderly Care Services Using Robots
Suggestion
Fall
Detection
Wandering
Monitoring
Location
Monitoring
Care Services

15. Wrist-Type Device Based Human Behavior Recognition
15
 Wrist-type and waist-type monitors
MCU Cortex-M3 (STM32F100)
RF(Zigbee) CC2520
Sensors 3-axis accelerometer(LIS331DLH)
3-axis gyro (L3G4200D)
Temperature/humidity (SHT21)
Brightness (TCS3414CS)
IR Photodiode (TSOP85238)
Emergency
button
1개 (front side)
Memory card MicroSD
Battery [Li-Ion 600mAh]
Recharger External rechager
Strap Wrist: nato band
Waist: elastic belt

16. Lifestyle Manager Using SNS and Activity Recognition
16
Life-style patterns
Clinical history
Genetic information
Server
Smartphone
users
Lifestyle
ranking
Life
log
Lifestyle
disease risk
Default
behavior
registration
Location - time
elapse
threshold
Localization by
Wifi signal

17. 17
 Analysis of life log and SNS
 Lifestyle = Eating habit(timing and food types) + CAR(Circadian activity rhythm)
Server
Many smart phone users
Location dimension
Sleep : My room, Park, Motel
Rest : TV room, Living room, Lounge
Work or study : Work place, Study place
Enjoy : Shopping place, Cultural place, Attractions
Usual food : Restaurant
Exercise : Exercise place
Religious activities : Church, Buddhist temple
Fast food or snack : Fast food place, Mc. Donald,
Convenience store
Sugar-sweetened beverage : Cafeteria,
Convenience store
Smoking : Smoking place, Convenience store,
Alcohol : Alcohol place, Bar
Drug : Drug store
UNKNOWN
1. Lifestyle
recommendation
2. Measurement of
Lifestyle Metric
Data matrix
Tries for users’ visiting patterns on
location & time dimension
People
Healthcare organizationsProactive/Reactive Services
Lifestyle Manager Using SNS and Activity Recognition

18. Lifestyle Manager Using SNS and Activity Recognition
18
- Home, Hotel
- Smoking place
- Drinking place
-Working place, Studying place
- Restaurant
- Cafeteria, Coffee shop
- Exercising place
- Religious place
-Attractions, Shopping place,
Cultural place, Enjoy place
- Hospital, Pharmacy
- Unknown
Location manager
- Facebook :
contains more
daily lives than
others
( e.g.,
“I ate a hamburger,
so cool.“ 11:45AM )
Social Network
Services
-Walking
- Exercise / Sport
- Running
- Riding an automobile
- Riding a bicycle
- No activity
Activity Recognition
-Accelerometer
- Illuminance sensor
-WiFi
- GPS
Sensor Handler
- Name
- SNS information
-Address ( GPS )
- BMI
- …
User profile
- Lifelog
- Carlorie counting
(day, week, …)
- Lifestyle disease risk
Service Provider

19. 19
Reuse
Experience Base
Developer
Community
Social
Network
Service
Extracted
Experiences
(XML-based)
Experience
Structure
Problem
Solution
Environments
Flag
(success, fail)
Problem
Topics
Solution
Topics
Document
Parser
Social
Network
Analyzer
Topic
Miner
Relationship
Creator
Co-related
Problem-Solution
Topic Pairs
Constructing case-base Reusing the cases as experience
Experience Mining for Healthcare
Social Network

20. 20
Problem Document1
Solution Document1
Original Document1
Original Document2
Original Document3
Original Document4
Original Document10
...
Problem
Document10. . .
Solution
Document10. . .
Split to problem and
solution documents
Topic Mining for Problems and Solutions

21. 21
Keywords
window
button
taskbar
bitmap
tiff
hchuldwnd
msdn
hwnd
wparam
...
Rank Doc #Words Ratio
1 D07 99 80.5%
2 D10 71 67.0%
3 D08 24 24.2%
4 D02 23 29.5%
5 D06 18 13.0%
6 D05 12 13.6%
7 D01 12 14.6%
8 D04 5 27.8%
9 D03 3 8.1%
10 D09 0 0.0%
ProblemTopic1
Keywords
http
bit
alpha
wrp
aspx
en
work
system
trusted
...
Rank Doc #Words Ratio
1 D02 54 69.2%
2 D08 42 42.4%
3 D01 39 47.6%
4 D05 34 38.6%
5 D07 15 12.2%
6 D06 12 8.7%
7 D10 10 9.4%
8 D09 5 71.4%
9 D04 4 22.2%
10 D03 4 10.8%
ProblemTopic2
Keywords
windows
vista
microsoft
logo
partner
program
application
support
certification
...
Rank Doc #Words Ratio
1 D06 108 78.3%
2 D05 42 47.7%
3 D08 33 33.3%
4 D01 31 37.8%
5 D03 30 81.1%
6 D10 25 23.6%
7 D07 9 7.3%
8 D04 9 50.0%
9 D09 2 28.6%
10 D02 1 0.0%
ProblemTopic3
Keywords
office
run
microsoft
community
aurigma
test
en
prevent
channel
...
Rank Doc #Words Ratio
1 D04 20 50.0%
2 D01 17 27.4%
3 D05 15 0.0%
4 D03 9 0.0%
5 D02 7 25.9%
6 D10 0 14.0%
7 D09 0 44.4%
8 D08 0 30.0%
9 D07 0 0.0%
10 D06 0 0.0%
SolutionTopic1
Keywords
window
bitmap
net
http
feedback
error
read
link
application
...
Rank Doc #Words Ratio
1 D05 22 50.0%
2 D01 15 24.2%
3 D03 12 50.0%
4 D02 11 40.7%
5 D04 7 0.0%
6 D10 0 20.6%
7 D09 0 15.6%
8 D08 0 40.0%
9 D07 0 0.0%
10 D06 0 0.0%
SolutionTopic2
Keywords
windows
taskbar
system
ws
msft
www
dp
question
bob
...
Rank Doc #Words Ratio
1 D05 36 50.0%
2 D04 13 50.0%
3 D01 11 17.7%
4 D03 5 50.0%
5 D02 5 18.5%
6 D10 0 33.6%
7 D09 0 28.9%
8 D08 0 16.7%
9 D07 0 50.0%
10 D06 0 50.0%
SolutionTopic3
Keywords
button
app
win
msdn
cp
support
creating
usi
bao
...
Rank Doc #Words Ratio
1 D05 34 0.0%
2 D01 19 40.7%
3 D04 5 0.0%
4 D03 4 0.0%
5 D02 4 14.8%
6 D10 0 31.8%
7 D09 0 11.1%
8 D08 0 13.3%
9 D07 0 50.0%
10 D06 0 50.0%
SolutionTopic4

22. Healthcare Service Framework for
Continuous Context Monitoring
Prof. Jun-Hwa Song
Dept. of Computer Science
KAIST
22

23. Context-Aware Healthcare Service Scenarios
23
 Example Scenarios
 Obesity monitoring
 Continuously monitors people’s activity level
and consumed calories, and suggests proper
exercises to the people.
 Elderly people monitoring
 Continuously monitors an elderly people’s
emergency situation such as slipping down on
a wet floor, and expedites an emergency call.
 Cardiac patient monitoring
 Continuously monitors a cardiac patient’s ECG,
and expedites an emergency call.

24. DietSense: Smartphone-Based Diet Monitoring for Enhancing
Obesity Self-Care
24
Comparing diet and physical activity
Monitoring Physical ActivityCapturing Diet
Calories consumed
from food
Calories burned during
physical activity
camera
microphone
accelerometer

25. Activity Log
Smartphone
1. Collecting activity data from the patient
2. Training ML algorithms for analyzing activity patterns
3. Figuring out the right does time without interrupting
the current work activity
4. Notify the does time and subsequent reminder
MedicineTaker
Motion Sensor
Activity Log
Place A Place A
Task 1 Task 2 Task 3
Action 1-1
Action 1-2
Action 1-3 Action 2-1
Action 2-2
Action 3-1
Type 1
Type 2
Smart Alarming for Long-Term Medicine Adherence
25

26. Continuous Context Monitoring
26
 Continuous monitoring of user’s context
 A key building block for personal context-aware applications
 Often requires complex, multi-step, continuous processing for multiple devices
 E.g., Running situation -> sensing in three 3-axis accelerometers, FFT processing,
recognition
Location-based
Services
HealthMonitoring
U-Trainer
U-Secretary
U-Reminder
Dietdiary
U-Learning
Behaviorcorrection
S
F C
S
S
F
F
C
SS
F
C S
S
F
C
S
F
C
Sensing
Feature extraction
Context recognition
PAN-scale dynamic
distributed computing
platform
Context monitoring
(e.g., sensing, feature extraction,
recognition)
Application logic
Location-based
Services
HealthMonitoring
U-Trainer
U-Secretary
U-Reminder
Dietdiary
U-Learning
Behaviorcorrection
A A
A App logic

27. Mobile Healthcare Service Framework
27
 Develop a healthcare
service framework
 To support multiple and long
running healthcare services with
highly scarce and dynamic
resources
 Efficient resource utilization
 Longer lasting operation (and
service) under highly scarce
resource situation
 Quick and efficient abnormal
situation detection
 Seamless (stable) operation
even under high resource
dynamics
 Challenges
 Limited battery power due to
mobility
 Scarce computing resources of
mobile devices
 Dynamic join/leave of
heterogeneous sensors
 Multiple healthcare services
share highly limited resources
Resource Manager
Policy
Manager
Energy Manager
Sensor Broker
Sensor Detector
Communication
Manager
… …Heartbeat monitoringFall monitoring
Healthcare services
API
Message
Interpreter
Sensor Data Processor
Resource
Monitor
Network protocols (e.g., ZigBee, BT)
Health Context Monitor
Sensor data
Requests
Sensor availability/status
Results
Sensor detection/control, Data/status report
Sensors in BAN/PAN
Application Broker
Application
Interface
Result Manager Message Parser
Diet diary
GPSBVP/GSR Accelerometers
……
Anomaly Detector
Feature Extractor
Resource Coordinator

28. Healthtopia – Healthcare Platform
 Providing API
for utilizing
health sensors
 Saving power
consumption
for concurrent
multiple apps
28

29. Text Mining for Extracting Knowledge
from Web Contents
Prof. Key-Sun Choi
Dept. of Computer Science
KAIST
29

30. Food Ingredients and Recipe Advice for Controlling Obesity
Web Environment
Ontology
Mobile Environment
Recipe
Extraction
Web /Wikipedia
Automatic
User Experience
Extraction
Target Food/Dish
Recognition
Manual
User Experience
Input
Web Log / SNS
Equipment
How-to
Food/Dish
Restaurant
DB
Scenario 2.
New Recipe (Low calories)
Suggestion w/ same Ingredients
Scenario 1.
Food-NutritionAssociationVisualization
Nutrition
Ingredients
Food-Nutrition
Extraction
30

31. Mining Connections between Multiple Sources
31
Literature Web Clinical Data
HeterogeneousTextual Sources
-Textbook
- PubMed
- Blogs
-Wikipedia
- Personal health record
 Medical information sources
 Literature contents affect the Web contents
 As background factual knowledge
 Web contents have other benefits
 Wide coverage
 Huge collaborators (confidence)
 Aggregating information from multiple sources
 Analysis of trend evolving on literature/Web to identify factors that will improve the
quality of patient care
 Reliability: Literature > News > Web (Wikipedia, blog, SNS)
 Accessibility: Web ≥ News > Literature

32. Detecting MeSH Keywords from Web Pages
32
 Medical Subject Headings (MeSH)
 NLM controlled vocabulary thesaurus used for indexing articles for PubMed
 Tree structure (http://www.nlm.nih.gov/mesh/trees.html)
 Provide an efficient way of accessing and organizing biomedical information
 Examples of MeSH Headings
 Body Weight, Kidney, Dental Cavity Preparation, Self Medication, Brain Edema
Extracting Candidates
Matching
Obesity is a medical condition in which excess body fat has a
ccumulated to the extent that it may have an adverse effect on
health, leading to reduced life expectancy and/or increased he
alth problems.[1][2] Body mass index (BMI), a measurement
which compares weight and height, defines people as overweig
ht (pre-obese) if their BMI is between 25 and 30 kg/m2, and o
bese when it is greater than 30 kg/m2.
• Hyperinked terms are extracted as term candidatesLanguage Handling

33. Detecting MeSH Keywords from Web Pages
33
Extracting Candidates
Matching
Obesity
Medical
condition
Body
fat
Body Mass Index
weight
Dieting
Obesity
Medical
condition
Body
fat
Body Mass Index
Body
Weight
Diet
Link
Structure
MeSH
term
Language Handling
 Language Handling
 Polysemy and homonymy problem

34. 34
Semi-Automatic Infobox Construction for
Korean Wikipedia

35. 35
맛있는 감자탕 1그릇을 먹을경우 177Kcal를 섭취하게 된다고 합니다.
Parsing the sentence
Extracting knowledge
<감자탕 1그릇, 열량, 177Kcal>
Infobox DB
Extracting Knowledge from Unstructured
Texts using Infobox DB

36. Personalized Bio and Medical Data Analysis
Prof. Gwan-SuYi
Dept. of Bio and Brain Engineering
KAIST
36

37. Personalized Diseases Risk Analysis
User
Agent
Disease risk
Prediction model
Personal genome
Data processing model
Drug response
Prediction model
Disease risk info.
(SNP-Disease)
Drug response info.
(SNP-Drug)
Personal genome
info.
Personal sequence
data
New info. on
disease risk
New info. on drug
response
Update Update
request
result
Personalized
disease risk
Genome
profile
Personalized drug
response
Storage
Build database
Personalized Personalized
Drug
response
Diseases
risk
Obesity, Diabetes Obesity, Diabetes
37

38. Constructing Databases for Diseases Risk and Drug
Response
38
184(Type I Diabetes), 203(Type II Diabetes), 82(Obesity) entries for diseases related SNP markers
collected
228 drugs, 830 SNP markers, 1341entries for drug-SNP related information collected
Diseases
Drugs
Diseases risk info.
SNP ID
Gene
Gene Region (Locus)
Risk Allele
Odds Ratio
P-value
Reference
…
PharmGKB
Drug Bank
Drug response info.
SNP ID
Gene
Gene Region (Locus)
Drug
Condition
Reference
…
Integrated database for
diseases risks and drug responses
Public database
Drug related info.
23andMe
Navigenics
Pathway Genomics
Gene sequencing
service drug related
info.
23andMe
Navigenics
deCODEme
Gene sequencing
service GWAS info.
HugeNavigator
GAD
NCBI (HapMap &
NHGRI catalog)
Public database
GWAS info.

39. Developing Methods for Analyzing Diseases Risk and
Drug Response
39
Diseases
OMIM
PharmGKB
DrugBank
Drugs
PharmGKB
DrugBank
SNPs
dbSNP
HapMap
Genome type
WTCCC
Genome body
UCSC
Ensembl
AceView
Genome
Entrez
Bio. pathway
KEGG
Reactome
NCI pathway
Panther
SNPs
dbSNP
HapMap
Genome type
WTCCC
Biological information
SNP
SNP
Analysis tech. for diseases risk
Analysis tech. for drug response
Obesity, Diabetes
• Extraction of drug response related SNP’s
• Drug targeting and biological pathway based function analysis
• Drug response prediction
• Obesity (Diabetes) related SNP or SNP combinations info.
• Genomic and biological pathway based function analysis
• Diseases risk prediction

40. Service Platform for Personalized Information about
Diseases Risk and Drug Response
Agent
User
Plug-ins from
life-logging team
Diseases risk
prediction model
Personal genome info.
data processing model
Drug response
prediction model
Diseases risk info.
(SNP-Disease)
Drug response info.
(SNP-Drug)
Diseases DrugsGenome
body
SNP PathwayGenome
type
Life-logging
database
Personal genome
info.
40

41. Thanks for Attention
Contact:
hojinc@kaist.ac.kr
41