1. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Activity and Emotion Recognition
to Support Early
Diagnosis of Psychiatric Diseases
Pervasive Health Conference
Tampere, 30th January 2008
B. Arnrich Tampere, January 30th

2. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Paper Contributors
David Tacconi, Oscar Mayora
CREATE-NET
Paul Lukowicz
University of Passau
Bert Arnrich, Cornelia Setz, Gerhard Tröster
ETH Zurich
Christian Haring
PSHT
PSYCHIATRIC STATE HOSPITAL TIROL PSYCHIATRIC STATE
HOSPITAL TIROL
B. Arnrich Tampere, January 30th

3. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Outline
Introduction and Motivation
Bipolar disorder
Pervasive computing to support diagnosis of
Bipolar Disorder
A proposed System Architecture
Discussion and Future Work
B. Arnrich Tampere, January 30th

4. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Introduction
Global Burden of Disease
Mental illness accounts for over 15% of the burden of diseases in
established market economies1
Disability Adjusted Life Years (DALYs) measure the lost years of
healthy life due to premature death or disability
Depression
is the most common psychiatric disorder, accounting for 50.8
million DALYs or 10.7% of the global burden of disease
It is ranked fourth among all causes of DALYs and is the leading
nonfatal condition globally
Mental disorders like the bipolar disorder
account for another 14.1 million (3.0%) DALYs
1World Health Organization, World Bank, Harvard University
B. Arnrich Tampere, January 30th

5. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Motivation
Extend psychotherapy beyond the therapy hour
State of the Art: computer-aided between-session
therapy
Online questionnaires
Automatic scheduling
Our proposal: Activity and Emotion Recognition
as a specific contribution to therapy
B. Arnrich Tampere, January 30th

6. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Challenges
Few technological solutions exist to aid people
affected by mental illness
Obvious reasons are:
people affected by mental illness are more likely to
have problems dealing with complex technology
providing behavioral assistance is much more difficult
than providing physical assistance
solutions require considerable amount of domain
specific knowledge
B. Arnrich Tampere, January 30th

7. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Bipolar Disorder
Characterization
repeated relapses of depression and mania
Recurrence rates are high at around 50% to 70%
Treatment of Bipolar disorder
Main: Pharmacotherapy
Alternative: teach the patients to recognize and manage
Early Warning Signs (EWS)
Diagnosis through patient questionnaires
Depression: Hamilton Depression Scale (HAMD)
Mania: Bech-Rafaelsen Mania Scale (BRMS)
Both contain a series of questions related to patient’s state,
activities and feelings
B. Arnrich Tampere, January 30th

8. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Bipolar Disorder: the HAMD
1. Depressed Mood 11. Anxiety Somatic
2. Feelings of Guilty 12. Somatic Symptoms
3. Suicide (Gastrointestinal)
4. Insomnia (early) 13. Somatic Symptoms
General
5. Insomnia (middle) 14. General Symptoms
6. Insomnia (late) 15. Hypocondriasis
7. Work and Activities 16. Loss of Weight
8. Retardation: 17. Insight
Psychomotor
9. Agitation 18. Diurnal Variation
10. Anxiety (Psychological) 19. Depersonalization and
Derealization
20. Paranoid Symptoms
B. Arnrich Tampere, January 30th

9. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Bipolar Disorder: the BRMS
1. Motor activity 7. Self-esteem
2. Verbal activity 8. Contact
3. Flight of thoughts
9. Sleep
4. Voice/Noise level
10. Sexual interest and
5. Hostility activity
6. Mood and feelings of
well-being 11. Work level
B. Arnrich Tampere, January 30th

10. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Contributions
We identify Bipolar Disorder as a condition that can realistically
benefit from behavioral monitoring
We identify support in early detection of imminent transitions
between normal, manic and depressed states as the specific
contribution to therapy
We identify specific behaviors that need to be detected by the
proposed system, using the HAMD and the BRMS
Based on literature study and previous work by the authors, we argue
that detecting these specific behaviors is feasible
We propose an appropriate system architecture based on existing
devices and previous systems implemented by the authors groups
B. Arnrich Tampere, January 30th

11. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Insomnia and Sleep disorders
HAMD 4-6, BRMS 9
”Gold standard” (laboratory settings):
polysomnographic monitoring of sleep time
physiological parameters (e.g. respiration, heart rate variability) and sleep
motion
Alternative On-body sensors
unobtrusively embedded into biomedical clothes or mattresses
allow to obtain preliminary diagnosis and to perform more frequent tests
under real-life conditions
Alternative sensor mats placed under the mattress
thin film, dynamic quasi-piezoelectric sensors
capacitive pressure sensor mat
B. Arnrich Tampere, January 30th

12. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Verbal activities and Conversations
BRMS 2, 4 and BRMS 8
Spoken messages convey non-textual characteristics like
intonation, speaking rate or emotional state
Automatic speech character identification would allow to
extract features describing contextual side information
Emotion recognition can give the therapists information
about variation of the patient’s mental state
B. Arnrich Tampere, January 30th

13. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Emotion Recognition
Restriction to a set of
basic emotional states
Feasibility study:
10 subjects
6 emotions
Recognition rates
comparable to humans
B. Arnrich Tampere, January 30th

14. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Activity Recognition
HAMD 7-11 and BRMS 1
Several past works on activity recognition
Based on previous experience, we target
systematic real life trials to:
quantify Work and Activities (HAMD 7 and BRMS 1)
detect Agitation (HAMD 9) and Anxiety (HAMD 10, 11)
measure Psychomotoric Retardation (HAMD 8)
B. Arnrich Tampere, January 30th

15. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Activity recognition
Main sources of activity
information
Worn combination of
accelerometer and microphone
Location information
Previous experiments
Spotting complex activities is
feasible
Recognition directly on wrist
worn device
B. Arnrich Tampere, January 30th

16. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
System Architecture
Challenges to be considered:
Patients are likely to reject pervasive computing technology in
principle
Target devices should be as less obtrusive as possible
Patients cannot be asked to perform any training of devices
Activity and emotion recognition is targeted to medium and
long term behavior
Higher errors in single activity recognition are allowed
Focus on average behaviors rather than in instantaneous activity
pattern or emotions
Behaviors that are repeated in time and that can be symptoms of
disease’s relapse
B. Arnrich Tampere, January 30th

17. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
System Architecture
The User Interfaces module:
•present persuasive feedback to the users for
motivating healthier patients’ behavior
The Context Acquisition module gathers data
from Sensors and is driven by:
•Emotion Recognition Manager that selects
sensors for emotion recognition
•Activity Recognition Manager that selects
sensors for recognizing user’s activity
•User model manager gives proper inputs
The Content Manager module is responsible:
•For uploading the data to the EMR through the
Data Upload module
•For presenting information to the patient
through the Feedback Manager module
The User Model includes all patient’s
characteristics, disease’s peculiarities and his
preferences. Information stored in:
•User Profile (UP)
•Disease Description (DD)
•Patient Description (PD)
B. Arnrich Tampere, January 30th

18. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Conclusion and Future work
Conclusion
Concept on applying existing pervasive computing
techniques to support the early diagnosis of bipolar
disorder
Proposal of a system architecture designed to monitor
patient’s behavior
Future work:
Integrating the currently available technology
Laboratory testing
Field test at the Psychiatric Hospital in Tirol, Austria
B. Arnrich Tampere, January 30th

19. Activity and Emotion Recognition to Support Early
Diagnosis of Psychiatric Diseases
Thank you for your attention.
B. Arnrich Tampere, January 30th