Keynote: SECOND INTERNATIONAL WORKSHOP IN MULTIMEDIA PRAGMATICSMMPrag 2019, San Jose, California, 28-30 March 2019 http://mipr.sigappfr.org/19/keynote-speakers/ The Holy Grail of machine intelligence is the ability to mimic the human brain. In computing, we have created silos in dealing with each modality (text/language processing, speech processing,image processing, video processing, etc.). However, the human brain’s cognitive and perceptual capability to seamlessly consume (listen and see) and communicate (writing/typing, voice, gesture) multimodal (text, image, video, etc.) information challenges the machine intelligence research. Emerging chatbots for demanding health applications present the requirements for these capabilities. To support the corresponding data analysis and reasoning needs, we have to explore a pedagogical framework consisting of semantic computing, cognitive computing, and perceptual computing (http://bit.ly/w-SCP). In particular, we have been motivated by the brain’s amazing perceptive power that abstracts massive amounts of multimodal data by filtering and processing them into a few concepts (representable by a few bits) to act upon. From the information processing perspective, this requires moving from syntactic and semantic big data processing to actionable information that can be weaved naturally into human activities and experience (http://bit.ly/w-CHE). Exploration of the above research agenda, including powerful use cases, is afforded in a growing number of emerging technologies and their applications - such as chatbots and robotics. In this talk, I will provide these examples and share the early progress we have made towards building health chatbots (http://bit.ly/H-Chatbot) that consume contextually relevant multimodal data and support different forms/modalities of interactions to achieve various alternatives for digital health (http://bit.ly/k-APH). I will also discuss the indispensable role of domain knowledge and personalization using domain and personalized knowledge graphs as part of various reasoning and learning techniques.

1. ON EXPLOITING MULTIMODAL INFORMATION
FOR MACHINE INTELLIGENCE AND NATURAL INTERACTIONS
WITH EXAMPLES FROM HEALTH CHATBOTS
Keynote: SECOND INTERNATIONAL WORKSHOP IN MULTIMEDIA PRAGMATICS
MMPrag 2019, San Jose, California, 28-30 March 2019
Amit Sheth
LexisNexis Ohio Eminent Scholar
The Ohio Center of Excellence in Knowledge-enabled Computing & BioHealth Innovations (Kno.e.sis)
Wright State, USA
Icon source used in the entire presentation - https://thenounproject.com
Presentationtemplateby SlidesCarnival
Photographsby Unsplash

2. ▰ “But most
importantly, by
freeing physicians
from the tasks that
interfere with
human connections,
AI will give doctors
the gift of time—to
restore care in
healthcare.”
2
▰ “…virtual
assistants,
powered by
personalized AI,
can provide us
with coaching to
promote our
health, shape our
diet, and even
prevent illness.”

3. 3
Outline
❖ Humans benefit from consuming data in the form of various modalities
(text, speech, and visual).
❖ Multimodal information are essential and together, they provide nuances
that a single modality can’t.
❖ For a machine to attain intelligence, it requires comprehensive
understanding of the environment that it is in.
❖ And to develop natural interactions with human, a machine needs to
develop understanding of the data it consumes.
❖ This talk will focus on different data modalities and examples on how a
machine (chatbot) can use such information to provide intelligent
assistant and natural communication in the health domain.

4. 4
Source: https://www.businessinsider.com/amazon-reveals-alexa-sales-2019-1
Voice Assistants on the Rise

5. 5
Machine-centric to
Human-centric Computing
Artificial
Intelligence
Ambient
Intelligence
Augmenting
Human Intellect
Human-Computer
Symbiosis
Computing for
Human Experience
Machine-centric Human-centric
John McCarthy Mark Weiser Douglas Engelbart Joseph C.R. Licklider
Figure: Views along the spectrum of machine-centric to human-centric computing.
At the far right is our work on Computing for Human Experience, which explores paradigms such as
Semantic, Cognitive, and Perceptual Computing. http://bit.ly/SCP-Magazine
Kno.e.sis Center
http://bit.ly/k-Che,
http://slidesha.re/k-che

6. Eric Topol:
The World of Shallow Medicine
▰ “(Doctors) have not had access to
patients in their real-world, on the
go, at work, while asleep. The data
doctors access is from the
contrived setting of the medical
office, constrained by the temporal
lists of the visit itself….(EHR info)
is remarkably incomplete and
inaccurate.”
▰ “Patients exist in
the world of
insufficient data,
insufficient time,
insufficient
context, and
insufficient
presence.”
7

7. 8
Using Smart Chatbots to Go Help Escape
Shallow Medicine
Socio-
economic
Demo-
graphic
Family &
social
Psychological
Environment
Genetic
Susceptibility
Source: Why do people consult the doctor?
- Stephen M Campbell and Martin O Roland
Decision
Making
Can voice assistant (chatbot) technology
substantially improve monitoring of
patient’s conditions and needs?
Simple Tasks
● Appointment scheduling
● Information retrieval
● Scripted-automation
Complex & Demanding Tasks
● Multimodal input and output
● Natural communication
● Augmented Personalized Health
(serving different levels of health needs)
Contextualization
Personalization
Abstraction
Different modality of data
ImagesText Speech Videos IoTs

8. Figure source: https://www.aarp.org/health/conditions-treatments/info-2017/
bronchitis-and-pneumonia-symptoms.html
A machine may recognize the picture as
“a woman is coughing”.
As human, we immediately conjecture and relate to
many phenomena with different contexts.
Semantic
Association
(Label picture
as coughing)
Cognitive
(Look at additional
background information
& interpret in different
context, ie: cough vs
wheezing cough
Perception
(Has the patient condition worsen?
How well is the patient doing?)
Paradigms
that Shape
Human
Experience

9. AUGMENTED PERSONALIZED HEALTH
EXPLOITING MULTIMODAL INFORMATION FOR:
SELF-MONITORING
SELF-APPRAISAL
SELF-MANAGEMENT
INTERVENTION
DISEASE PROGRESSION AND TRACKING

10. 11
This not only prevent the disease, but also enhances the patient’s health.
BariatricsAsthma
Use Cases: APH for Asthma and Bariatrics: Patient-centric drivers

11. “
12
The Holy Grail of machine intelligence is the ability to
mimic the human brain. However, the human brain’s
cognitive and perceptual capability to seamlessly
consume, abstracts massive amounts of multimodal data,
and communicate information challenges the machine
intelligence research. Growing number of emerging
technologies such as chatbots & robotics present the
requirements for these capabilities.

12. What is Modality
GENERAL
A particular mode in
which something exists
or is experienced or
expressed.
A particular form of
sensory perception: ‘the
visual and auditory
modalities’.
HEALTHCARE MODALITY
Modality (medical imaging), a
type of equipment used to
acquire structural or
functional images of the body,
such as radiography,
ultrasound, nuclear medicine,
computed tomography,
magnetic resonance imaging
and visible light.
IN HCI
A modality is the classification
of a single independent channel
of sensory input/output
between a computer and a
human.
Multiple modalities can be
used in combination to provide
complementary methods that
may be redundant (or
complementary) but convey
information more effectively.
13

13. 14
Machine Intelligence for Chatbot:
Incorporating Diverse Streams
& Modalities
Figure: Chatbot exploiting
multimodal information for machine
intelligence and natural interactions
From simple informational
interface (text, speech) to
intelligent assistant

14. USE CASES & PROTOTYPES
Examples and early progress on ongoing collaborative
healthcare (chatbot) projects
@ KNO.E.SIS
Three varieties:
 Data for patient’s real world
 Virtual Medical Coach
 Smart Nutrition

15. 16
Health Related Studies at KNO.E.SIS
[Overview]
HealthChallenges
(Also Dementia,
Obesity,
Parkinson’s, Liver
Cirrhosis, ADHF)
Public Policy/ Population Epidemiology Personalized Health
PCS + EMR + Multimodal
(Speech + Image)
kHealth
Asthma in Children
Bariatric Surgery
Nutrition
Physical(IoT)/Cyber/
Social (PCS)+ EMR
Marijuana Social
Drug Abuse Social
Mental Health
Depression & Suicide Social + Public + EMR
Health Knowledge
Graph Services
Social + Clinical Data
...and infrastructure
technologies: Context-aware
KR (SP), KG Development, Smart
Data from PCS Big Data, Twitris

16. 3 Chatbots (Alpha/Beta Stage)
1. NOURICH: A Google Assistant based
Conversational Nutrition Management
System
2. kBOT: Knowledge-enabled (kHealth)
Personalized ChatBot for Asthma:
Contextualized & Personalized
Conversations involving Multimodal
data (IoT & Devices)
3. ReaCTrack: Personalized Adverse
Reaction Conversation-based Tracker for
Clinical Depression
17
HCI: Mobile Applications & Chatbots
@ KNO.E.SIS
kHealth
Asthma
kHealth
Bariatrics
Depression
Active (Subset)
Healthcare Projects
@ KNO.E.SIS with
mApps/chatbot
kHealth Framework: a knowledge-enhance AI learning
platform that captures the data and analyzes it to produce
actionable information.

17. 18
Physical-Cyber-Social (PCS) Data
Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1),
peak expiratory flow (PEF), indoor temperature, indoor humidity,
particulate matter, volatile organic compound, carbon dioxide,
air quality index, pollen level, outdoor temperature, outdoor
humidity, number of steps, heart rate and number of hours of
sleep. Also clinical notes.
kHealth Asthma Nutrition
Depression
Active Healthcare Projects
in Kno.e.sis (Subset)
Modality of Data
kHealth Bariatrics
For monitoring asthma control and predict vulnerability
Pre and Post Surgery monitoring and self adherence
Mobile app Q/A (tablet), weighing scale, pill bottle sensor, water
bottle sensor for reminder to drink water, number of steps, heart
rate and number of hours of sleep. Also clinical notes.
Q/A, diet, food profile, food images, nutrition
knowledge bases, user knowledge graph.
For nutrition tracking and diet monitoring
Modeling Social Behavior for Healthcare Utilization in Depression
Q/A, social media profile (Twitter, Reddit).

18. 19
Modalities in Select mApps

19. 20
Chatbots for
Healthcare KNO.E.SIS
Overview

20. 21
Use Case 1: ASTHMA
Many Sources of Highly Diverse Data
(& collection methods: Active + Passive):
Up to 1852 data points/ patient /day
kBot with screen interface for
conversation
Images
Text
Speech
*(Asthma-Obesity)
★ Episodic to Continuous Monitoring
★ Clinician-centric to Patient-centric
★ Clinician controlled to Patient-empowered
★ Disease Focused to Wellness-focused
★ Sparse data to Multimodal Big Data
http://bit.ly/kHealth-Asthma

21. Data Collection So Far
110
patients
30
parameters
1852
data points per
patient per day
63%
kit compliance
● Data Collection: Since Dec 2016
● Active sensing: 18 data
points/day (Peak flow meter and
Tablet)
● Passive sensing: 1834 data
points/ day (Foobot, Fitbit,
Outdoor environmental data)
5-17
years of age
1 or 3
months of
monitoring
22

22. 23
Utkarshani Jaimini, Krishnaprasad
Thirunarayan, Maninder Kalra, Revathy
Venkataramanan, Dipesh Kadariya, Amit
Sheth, “How Is My Child’s Asthma?”
Digital Phenotype and Actionable
Insights for Pediatric Asthma”, JMIR
Pediatr Parent 2018;1(2):e11988, DOI:
10.2196/11988.

23. 24
Use Case 2: NOURICH
(diet management chatbot)

24. 25
Use Case 3: Elder Care Intelligent Assistant to support elderly with
Heart Failure (HF),
Chronic Obstructive Pulmonary Disease (COPD) or
Type 2 Diabetes Mellitus (T2DM).

25. “To support the (chatbots’) data analysis and reasoning
needs, we use a pedagogical framework consisting of
Semantic computing, Cognitive computing,
and Perceptual computing
This requires moving from syntactic and semantic big
data processing to actionable information that can be
weaved naturally into human activities and experience.
26

26. SEMANTIC-COGNITIVE-PERCEPTUAL
COMPUTING
Knowledge-Infused AI with Contextualization
(Knowledge Graphs), Personalization & Abstraction

27. 28
Semantic Browsing
Extraction
Data Integration and Interlinking
Entity
Complex Extraction
Aberrant
Drug-related
Behaviour
Neuro-Cognitive
Symptoms
Adverse
Drug
Reaction
Relation Event Severity
Personal Sensor Data De-identified EMR Blog Post
Context Representation Relevant Subgraph Selection
Semantic Search
Disease-specific
Chatbot
Visualization
Health
Knowledge Graph
Intent
Open Health Knowledge Graph

28. 29
SOCIAL -MEDIA TEXT
(July 12,2016)
EVENT-SPECIFIC
SCHEMA-BASED
KNOWLEDGE

29. 30
Application: Evolving Patient Health Knowledge Graph (PHKG)
Figure: A healthcare intelligent assistant interacts with the patient via various conversational interfaces
(voice, text, and visual) to acquire and disseminate information, and provide recommendation (validated
by physician). The core functionalities of the chatbot (Component C boxed in blue) are extended with a
background HKG (Component A boxed in green) and a evolving PKG (Component B boxed in orange).
★ Smarter & engaging agent
★ Minimize active sensing
(Questions to be asked)
★ Ask only informed & intelligent
questions
★ Relevant & Contextualized
conversations
★ Personalized & Human-Like

30. 31
ONE SLIDE TO SHOW HOW
PHKG EVOLVES OVER TIME
Knoesis Alchemy API
KHealth Project (IoT) datasets (e.g., asthma, obesity, Parkinson)
Reasoning mechanisms
Enriching KG
Enriching KG
In-built rule-based
inference engine
Machine
Learning
Updating the KG
with more triples
Analyzing datasets
Executing reasoning
Ontology Catalogs:
● BioPortal
● Linked Open Vocabularies (LOV)
● Linked Open Vocabularies for
Internet of Things (LOV4IoT)
Linked Open Data (LOD):
● UMLS
● SNOMED-CT
● ICD-10
● Clinical Trials
● Sider
Personalized Health
Knowledge Graph
(PHKG)
Personal
Sensor Data
Electronic Medical
Records (EMR)
Figure: How a PHKG evolves with multimodal information

31. 32
Contextualization and
Personalization
kBOT initiates greeting
conversation.
Understands the patient’s health
condition (allergic reaction to high
ragweed pollen level) via the
personalized patient’s knowledge
graph generated from EMR,
PGHD, and prior interactions with
the kBot.
Generates predictions or
recommended course of actions.
Inference based on patient’s
historical records and
background health knowledge
graph containing contextualized
(domain-specific) knowledge.
Figure: Example kBot conversation which
utilizes background health knowledge graph
and patient’s knowledge graph to support
intelligent interactions including
individualized recommendation.
★ Conversing only information relevant
to the patient
Context enabled by relevant
healthcare knowledge including
clinical protocols.

32. GENERIC CHATBOT VS
INTELLIGENT CHATBOT
With Examples of Contextualization, Personalization, and
Abstraction

33. 34
Contextualization
refers to data interpretation in terms of knowledge (context).
Without Domain Knowledge With Domain Knowledge
Chatbot with domain (drug)
knowledge is potentially more
natural and able to deal with
variations.

34. 35
Personalization
refers to future course of action by taking into account the contextual factors such as
user’s health history, physical characteristics, environmental factors, activity, and lifestyle.
Without
Contextualized Personalization
With
Contextualized Personalization
Chatbot with contextualized
(asthma) knowledge is
potentially more personalized
and engaging.

35. 36
Abstraction
A computational technique that maps and associates raw data to action-related
information.
With AbstractionWithout Abstraction
.

36. 37
Smarter Chatbot with
Semantically-Abstracted Information
Smarterdata
Data Sophistication
Smart (semantically-abstracted)
data should answer:
★ What causes my disease severity?
★ How well am I doing with respect to
prescribed care plan?
★ Am I deviating from the care plan? I am
following the care plan but my disease
is not well controlled.
★ Do I need treatment adjustments?
★ How well controlled is my disease over time?
Example of Abstraction

37. 38
Semantic, Cognitive, Perceptual Computing:
Paradigms That Shape Human Experience
http://bit.ly/SCPComputing
Humans are interested in high-level concepts
(phenotypic characteristics).
Semantic Computing: Assign labels and associate
meanings (representation & contextualization).
Cognitive Computing: Interpretation of data with
respect to perspectives, constraints, domain knowledge,
and personal context.
Perceptual Computing: A cyclical process of
semantic-cognitive computing for higher level of
perception and reasoning (abstraction & action).

38. Knowledge-Infused
Learning with
Semantic,
Cognitive,
Perceptual
Computing
Framework
39
THE BABY STEPS:
MACHINE / DEEP LEARNING INFUSED WITH
PERSONALIZED HEALTH KNOWLEDGE GRAPH
Knowledge
Domain (Ontology)
Personalized HKG
Multisensory
Sensing &
Multimodal
Data Interactions
ImagesText Speech Videos
IoTs
Natural Language
Processing,
Machine with
Deep Learning
AUGMENTED PERSONALIZED
HEALTH (APH)
Modeling broader disease context, and
personalized user behavior
Reasoning & decision-
making framework To achieve ABSTRACTION and minimize
data overload, assist in making choices,
appraisal, recommendations

39. 40
In short,
❖ Multimodal information are essential and can
be exploited for machine intelligence and
natural interactions.
❖ Knowledge-infused learning could give us the
power need to match complex requirements.
❖ Semantic-Cognitive-Perceptual Computing
enables contextualization, personalization, and
abstraction for Augmented Personalized Health.

40. 41
Special Thanks
Hong Yung (Joey) Yip
(Graduate Student)