This is a presentation on Sensor Based Ambient Assisted Living architecture and approaches developed by the Multimedia Knowledge and Social Media Analytics Lab of CERTH-ITI. It includes sensors used for monitoring Activities of Daily Living of elders and persons with mild Dementia at home. Visual and sensor data analytics are combined with formal representations (ontology), fusion, reasoning techniques and visualizations in order to provide an objective view of everyday activities. Example projects and pilots are included. Clinical assessment show improvement in cognitive abilities of participants.

1. Sensor-based AAL
ΕΛΕΒΗΤ 2019
CERTH-ITI MKLab Group
Dr Ioannis Kompatsiaris*
Researcher Grade A, MKLab Director
Dr Thanos Stavropoulos
Postdoc Associate
Dr Spiros Nikolopoulos
Senior Researcher
Mrs Ioulietta Lazarou
Clinical Researcher

2. The problem: Dementia Care
• Inaccessible – Unaffordable - Inefficient Care of Dementia
• No specific treatment – lifestyle/non-pharmaceutical
• High 1 Nurse per 1 Patient Effort
• High cost
• Lack of objective information
Yearly cost of dementia
care in 2030
People living with dementia in
2050
People living with
dementia now
46M 135.5M 2T $
1 in 2 over 80 (US)
Growing Numbers for Dementia

3. Overall approach
Enhance current clinical workflow:
 Continuous, comprehensive monitoring of PwD, condition and
progression
 Objective multi-sensor measurements (limiting interpretation
subjectivity)
Connect monitoring results with dementia staging and
assist diagnosis
Provide PwD with regular personalized feedback, updates
and interventions
 Improving condition
 Enhancing a sense of safety and increased independence
Relieve informal carers

4. The Solution
• Existing IoT and wearable sensors provide diverse measurements
(steps, HR, presence, object usage)
• Intelligent analysis turns them into meaningful and useful behaviors
and symptoms (cooking, chores, TV, sleep, stress)
… in order for clinicians to provide care more effectively and efficiently

5. Technological Approach
Sensors, Analysis & Integration

6. • 2013 - Use of prototypes
• Philips
• 2015 - Emergence in the
market
• FitBit
• Jawbone
• MS Band
• Withings
• 2019 - Growing capabilities
• Empatica Embrace
• Epilepsy FDA-app.
• Omron HeartGuide
• 24/7 BPM
• Withings Move / Move ECG
• 1 year battery life + ECG
Evolution of IoT Devices

7. Device/Hardware Layer
• A moving target; Need for future-proof
modular support
• Service-oriented architecture
• Components need to support data retrieval
• Streaming
Real-time transfer e.g. over Smartphone
• Use of Manufacturer SDKs to build
smartphone apps for storage or upload
• MS Band, Empatica
• Data-logging
storage in the device or on 3rd party cloud
• Use of 3rd party (provider)
cloud API for retrieval
• Fitbit, Withings, Jawbone
Our App
via SDK
Our
Cloud
Device
Device
Cloud
Device
App
Device
Our
Cloud

8. Device variety and modalities in the platform
Sleep Sensor
IR Presence
Object Movement
Door Sensor
Wearables
• Beddit
• Withings Aura
• Philips DTI2
• Jawbone UP24, UP3
• Fitbit Zip, Charge HR
• Empatica E4
• MS Band
• Wireless Sensor Tags
Appliance Usage
• Plugwise

9. Wearable Cameras
• Go Pro Hero
For more sophisticated
functions such as:
• Activity Recognition
• Object Recognition
• Room Recognition

10. Depth Cameras
• Microsoft Kinect
• ASUS Depth Camera
• Activity Recognition
• Indoor localization

11. Architecture for
Integration
• Integration of
• Device Layer to retrieve data
• Sensor & Image processing
layer to analyze them
• Store unanimously in a
Knowledge Base using
semantic web technologies
• Ontology with context & clinical
information
• Semantic Interpretation
• Provides activities, behavior
using ontology + reasoning
• Provides symptoms and
problems using rules
• User interfaces to present
Stavropoulos, T. G., Meditskos, G., & Kompatsiaris, I. (2017). DemaWare2:
Integrating sensors, multimedia and semantic analysis for the ambient care
of dementia. Pervasive and Mobile Computing, 34, 126-145.

12. Visual Analytics for
Activity Recognition
12
• Detailed activity and location detection and recognition at home and lab
environments.
Crispim-Junior, C. F., Buso, V., Avgerinakis, K., Meditskos, G., Briassouli, A.,
Benois-Pineau, J., ... & Bremond, F. (2016). Semantic event fusion of
different visual modality concepts for activity recognition. IEEE transactions
on pattern analysis and machine intelligence (IEEE TPAMI), 38(8), 1598-
1611.

13. Signal Analysis
Raw skin conductance signal
Baseline histogram for a week Filtering and segmentation in 5 Stress levels
Stress level signal
Developed with Philips NL, evaluated in Lulea Technical University
Kikhia, B., Stavropoulos, T., Andreadis, S., Karvonen, N., Kompatsiaris, I., Sävenstedt,
S., ... & Melander, C. (2016). Utilizing a wristband sensor to measure the stress level
for people with dementia. Sensors, 16(12), 1989.

14. Semantic Knowledge Structures (OWL 2)
• Formal vocabularies for capturing context in different levels of
granularity
• Low-level observations (e.g. objects, locations)
• Complex activity models (e.g. tea preparation)
• Clinical knowledge (e.g. problems, monitoring parameters)
14

15. Context-based Multi-sensor
Fusion and Analysis
15
• Objective: Fusion of information coming from heterogeneous sources in
order to derive high-level interpretations of the behaviour of the person
• Our approach: Knowledge-driven semantic segmentation and
classification of context
• Combination of SPARQL and OWL 2 meta-modelling
ADL REC PRE
Prepare Drug Box 92.00% 88.46%
Make Phone Call 89.29% 96.15%
Watch TV 84.00% 95.45%
Water the plant 80.00% 95.24%
Read Article 95.83% 85.19%
Meditskos, G., Dasiopoulou, S., & Kompatsiaris, I.
(2016). MetaQ: A knowledge-driven framework for
context-aware activity recognition combining SPARQL
and OWL 2 activity patterns. Pervasive and Mobile
Computing, 25, 104-124.

16. End-result: Signals + Activities

17. Applications & Pilots
Use Cases, Clinical Approaches, Interventions

18. EU FP7 ICT Project
• Dementia Ambient Care 2011 – 2015
• Multimodal sensing
• 3 use cases
• 300+ participants

19. @Lab
• Controlling equipment
• Dementia Assessment in a
controlled environment

20. @Lab Results • 290 participants Nice & Thessaloniki
• 80% accuracy Healthy - MCI - AD

21. @Home
• System-supported interventions
• 4 in Dublin, 6 in Thessaloniki for 4 – 12 months
• Improvement or non-deterioration in cognitive state
• Compared to non-system-supported interventions or regular care

22. Apps for self-management • Confidence, inclusion

23. Clinical Observations - Correlation between
metrics
Increased Physical Activity –
Decreased Sleep Latency
Increased Physical Activity –
Increased Sleep Duration

24. Clinical-Neuropsychological Assessment
• 6 Users
• 4 with Mild Cognitive Impairment
(MCI)
• 2 with Alzheimer’s Disease (AD)
• Neuropsychological and Clinical
Evaluation
• Baseline (1st half) and Follow-up
Assessment (2nd half)
• Standardized and Validated
Cognitive Test
• Tailored non-pharmaceutical
interventions (4-12 months)
i.e., Cognitive Behavioral
psychotherapy, reminiscence, memory
exercises etc
Tests Assessment Time
Users 1-6
M SD
MMSE
1st half 26.50 3.56
2nd half 28.33 1.86
NPI
1st half 1.67 0.52
2nd half 0.83 1.33
FRSSD
1st half 3.83 3.13
2nd half 2.67 2.66
FUCAS
1st half 48.67 8.48
2nd half 44.00 2.45
RAVLT - copy
spelling
1st half 4.83 1.72
2nd half 5.50 2.43
RAVLT - learning
1st half 7.67 3.01
2nd half 9.00 4.05
RAVLT - recall
1st half -3.50 1.05
2nd half -0.50 5.17
RAVLT total
1st half 38.67 13.53
2nd half 45.83 15.94

25. Clinical Observations
User 1 User 2 User 3 User 4 User 5 User 6
Total Time
Asleep (hours)
1st half 7.24 6.4 8.09 6.39 5.98 6.24
2nd half 8.14 7.36 8.62 6.47 7.1 7.58
P 0.001 0.0001 0.17 0.09 0.02 0.003
Number of
Interruptions
1st half 2.32 5.52 5.8 3.74 3.72 3.67
2nd half 2.13 5.85 3.8 2.25 2.4 4.75
P 0.44 0.57 0.001 0.02 0.0001 0.001
Shallow Sleep
(hours)
1st half 4.36 2.31 3.78 3.88 3.00 2.67
2nd half 3.88 3.04 3.67 4.37 3.35 3.1
P 0.01 0.02 0.64 0.36 0.0001 0.02
Sleep Latency
(min)
1st half 6.83 0.47 8.8 10.11 5.81 8.8
2nd half 3.33 0.46 8.4 5.1 5.5 8.4
P 0.009 0.95 0.88 0.03 0.59 0.02
Deep Sleep
(hours)
1st half 1.61 0.9 1.83 1.83 1.35 1.13
2nd half 1.94 1.22 2.13 2.13 1.42 1.54
P 0.02 0.0001 0.04 0.13 0.54 0.0001
Physical Activity
(min)
1st half 56.07 43.31 68.62 33.91 13.1 109.7
2nd half 57.16 43.17 74.25 35.13 43.26 112.94
P 0.22 0.91 0.0001 0.29 0.04 0.72
• Improvement in Sleep Parameters and Physical Activity in the 2nd half of observational
period

26. EU H2020 IoT LSP Project
• EU IoT Large Scale Pilots 2017 – 2020
• 9 Deployment Sites
All Partners 25.772.829 €
MEDTRONIC (coord) & 40 partners
• Integrate major open IoT platforms
• Active and Healthy Ageing (AHA) applications
• 9 Use Cases
• Mobility & Transport, emergency, home
activity & behavioral monitoring etc.
• A Marketplace to discover and install apps

27. Activage IoT Ecosystem Suite - AIOTES
• Not only dementia but any scenario of Active and Healthy Ageing
• Allows you to build and monetize eHealth apps over 9 open
European platforms
• 2nd open call to be launched soon

28. National Project
• EU-funded Erevno-Kainotomo-Dimiourgo 2018 – 2020
• 75 Home Participants
• Wearable & Apps to support Alzheimer
• Minimum equipment to support easy deployment and maintenance
• Promoted to national Telecom providers
All Partners 587.450 € Role
CERTH Thessaloniki IoT Integration, AI Analysis, RnD
ARX .NET Thessaloniki Mobile Apps, Business Dev
Frontida Zois Patras Clinical Pilots & Evaluation

29. IMI2 RADAR AD
• EU and EFPIA (Pharma) co-funded and co-developed 2019 –
2021
• Platform for Alzheimer’s “digital biomarkers” brought closer
to medical practice
• Building over RADAR-CNS platform and trials for multiple
diseases
• 3-Tiers of Clinical Trials
• 1) Small-scale Homes 2) Large-scale Homes
3) CERTH Smart Home - multiple equipment and incubator
environment

30. CERTH-ITI Smart Home
• Rapid-prototyping & demonstration for actual living scenarios
• 1st near Zero Energy Building in Greece
• 2-story, 4-bedroom, 3-bathrooms, living room, kitchen
• one of the most important pillars of the Digitise European Industry effort
• Equipment
• Smart Home
• Energy – Solar Panel
• Open to any choice of RADAR AD Tier 3
ROBOTICS & AI HEALTH
ENERGY BIG DATA

31. User-acceptance and Perspective
• A Patient-Advisory Board (PAB) to select devices
• Game with cards to extract preferences (1st meeting in Luxembourg)
• Anonymously select device representatives and aspects
• Top voted aspects are
• Appearance and Style, Weight, Water-proof, Emergency button feature and
Battery life

32. Other Projects - Wearable Cameras for the Blind
• eVision
• National Pilot Project
• Wearable Cameras for the blind
• SUITCEYES
• EU Pilot Project
• Suit & Wearable Cameras for
the deafblind
All Partners 660.075 €
CERTH Thessaloniki
TETRAGON LTD Thessaloniki
PRISMA Alexandroupoli
MASOUTIS Thessaloniki
THESSALONIKI
MUNICIPALITY
Thessaloniki
All Partners 2.359.963 €
CERTH Thessaloniki
UNI BORAS Sweden
UNI OFFENBURG Germany
UNI LEEDS UK
UNI EIDHOVEN NL
LES DOIGTS France
HARPO SP Poland

33. Conclusions
• IoT Constantly evolving – especially wearables
• New modalities, reliability, acceptance and user-related parameters
• Platform flexibility, also to pilot – deployment scale and complexity
• Sensors, analytics, visualization, interpretation can assist
staging and interventions
• Difficult to differentiate between actual contribution and enhanced
social activity
• Large-scale pilots and security aspects
• Big Data – Machine Learning approaches
• Emphasis on physical and cognitive interventions
• EEG mobile devices
• New applications
• Outdoor environment, smart cities
• Working environments, e.g. Mental Health of Employers

34. Thank You
Email : ikom@iti.gr
Links
Projects: demcare.eu activageproject.eu radar-ad.org
Lab & all other scientific activity mklab.iti.gr
Videos
Lab Trials - https://www.youtube.com/watch?v=AEuX58HLIDo
Home Monitoring - https://www.youtube.com/watch?v=0JNlaM6BpMA
CERTH-ITI Smart Home Video - https://www.youtube.com/watch?v=8pcw1Xhk240