Semantic and Fuzzy Modelling for Human Behaviour Recognition in Smart Spaces. A case study on Ambient Assisted Living. 24.4.2015 Åbo Akademi University, Finland

1. Semantic and Fuzzy Modelling for Human Behaviour Recognition in Smart
Spaces. A case study on Ambient Assisted Living
1Doctoral Defense 24th April 2015,Turku, Finland
Natalia A. Díaz Rodríguez
Supervisors: Prof. Johan Lilius and Prof. Miguel Delgado Calvo-Flores. Advisor: Prof. Manuel Pegalajar Cuéllar
Turku Centre for Computer Science (TUCS), Dept. of Information Technologies, Åbo Akademi University (Finland)
Dept. of Computer Science and Artificial Intelligence, University of Granada (Spain)

2. 2
SPAIN: 15m of elders in 2049 (1/3 of the population) (INE)
FINLAND population 65+ years: 18.14% [1]
• [1] http://www.finnbay.com/media/news/government-prepares-to-set-out-new-requirements-for-senior-caretakers/

3. 3Ros et. Al. 2011

4. OBJECTIVES
 Understand Smart Spaces
–Human Activity Modelling
and Recognition
 Program Smart Spaces
4

5. Ambient Assisted Living (AAL): usage of
technology to provide assistance to people who
needs it in their daily activities, in the less
obstrusive way
Aim: Independent living, safety, support
older/disadvantaged people
Includes: methods, systems, products and
services
5
Background

6. Activity Recognition in Smart Spaces
6
[Image: http://www.businesskorea.co.kr/sites/default/files/field/image/smart%20home.jpg + The noun project]

7. Human Activity Recognition
7

8. Handling uncertainty, vagueness
and imprecision
 Broken/ missing sensors
 Incomplete data, vagueness
etc.
 Different ways of perfor-
ming activities
– Different object usage
 Behaviour change
8

9. 9
Tools

10. Methods
WHY Semantic Technologies & Ontologies?
 Semantic Web: well-defined meaning
 Ontology:
– In Philosophy: study of entities and their
relations
– In Artificial Intelligence: “Explicit specification of
a conceptualization” [Gruber, 93]
– Web Ontology Language (OWL)
10

11. 11[CONON Context Ontology]
Methods: Ontologies

12. 12
Methods: Ontologies
JULIOANA MARIA
NATALIA
Has Brother
Has Mother

13. 13
JULIOANA MARIA
NATALIA
Has Brother
Has UncleHas Mother
Methods: Ontologies

14. Methods: Fuzzy Logic
WHY fuzzy (description) logics and fuzzy
ontologies?
 Real life is not black & white
– Classical (Crisp) Logic: True/False
– Fuzzy Logic: [0, 1]
• e.g. blond, tall
 For automatic reasoning about uncertain,
vague or imprecise knowledge
 For natural language expressions
14

15. 15
Case study on Ambient Assisted Living:
A fuzzy ontology for activity modelling and recognition
[Image: http://www.harmonizedsystems.co.uk/]
Example: Take Medication

16. 16
Case study on Ambient Assisted
Living: A fuzzy ontology for activity modelling and
recognition
Classes, Individuals, Data Properties and Object Properties
SUBJECT PREDICATE OBJECT
User performs activity Taking medicine =
(0.3 User performs sub-activity reach Cup or Medicine Box)
(0.3 User performs sub-activity move Cup or Medicine Box)
(0.1 User performs sub-activity place Cup or Medicine Box)
(0.1 User performs sub-activity open Medicine Box)
(0.1 User performs sub-activity eat Medicine Box)
(0.1 User performs sub-activity drink Cup)

17. 17
Hybrid activity modelling and
recognition with fuzzy ontologies
2 phased algorithm:
1. Sub-activities (data-driven phase)
2. High-level activities (knowledge-based phase)
Validation: CAD-120 dataset:
10 sub-activities, 10 activities, 10 objects, 4 users

18. Cornell Activity Dataset
[Koppula et al. 2013]
18

19. Hybrid data-
driven and
knowledge-
based
activity
recognition
19

20. ACTIVITY prediction:
accuracy results
20

21. Activity recognition
- comparison with state-of-the-art
21

22. 22Ros et. Al. 2011
Programming Smart Spaces
Implementation

23. A visual language to
configure the Smart Space behaviour
 TARGET USER: non-technical
background
 AIM:
– Rapid & easy programming of applications/
services
– Improve interoperability and usability
23

24. PROPOSAL: Smart Space visual programming
24

25. Main contributions
1. A set of ontologies to model human behaviour and tackle
uncertainty and vagueness inherent to real life
2. An architecture that integrates Semantic Web and
Fuzzy Logic for interpretable activity recognition
3. A hybrid knowledge-based and data-driven algorithm for
real-time, effective and robust activity recognition (84.1%
precision)
4. Design and development of a toolbox for non-expert
users and rapid and easy programming of Smart Spaces
[4 Journals -3 on 3rd Q1-, 9 conference papers, Google Anita Borg,
Nokia and HLF scholar. Entrepreneurship award]
25

26. Technology transfer:
26

27. Future Challenges
 Multiple humans sensing
 Parallel/interleaved activities
 Automatic ontology learning and
evolution
 Lifestyle modelling (Philips Research)
 Start-up
27

28. Thank you for your attention
Natalia A. Díaz Rodríguez
https://research.it.abo.fi/personnel/ndiaz
ndiaz@abo.fi
Embedded Systems Lab. Dept. of Information Technologies
Åbo Akademi University, Turku, Finland
TUCS (Turku Centre for Computer Science)
Dept. of Computer Science
and Artificial Intelligence
University of Granada, Spain
28

29. Appendix
 DTW: O(mn) (m,n length of the time
series)
 PAA compression size = 2
 Take out food is a subsequence of
microwaving (Subsumption heuristic/filter)
29

30. SUB-ACTIVITY
prediction: accuracy results
30

31. 31
Modelling activities with
fuzzy ontologies
Classes, Individuals, Data Properties and Object Properties
SUBJECT PREDICATE OBJECT
Filomena is a Person
Filomena has heart rate 60
Filomena performs sub-activity Reach glass
Filomena performs sub-activity Move medicine
Filomena performs sub-activity Pour water in glass
Filomena performs sub-activity Eat medicine
Filomena performs sub-activity Drink from glass

32. A crucial but challenging task in Ambient
Intelligence and AAL. Requires:
Context-awareness and heterogeneous data
sources
Training data: examples
Common-sense knowledge
Adaptation of behaviours
Alzheimer, Parkinson
32
Human Activity Recognition

33. 33
Cornell Activity Dataset
[Koppula et al. 2013]

34. 34
Cornell Activity Dataset
[Koppula et al. 2013]