A walk through my PhD projects, internship at Philips Research and research visit at UC Santa Cruz

1. CAN SUPERVISED AND UNSUPERVISED
LEARNING MARRY HAPPILY?
USE CASES ON HUMAN ACTIVITY RECOGNITION
Natalia Díaz Rodríguez
Visiting scholar
University of California, Santa Cruz
nadrodri@ucsc.edu
4th August 2015, Mobilize Center, Stanford University

2. OUTLINE
•ABOUT ME & MY RESEARCH
•RECENT PROJECTS
•SUPERVISED AND UNSUPERVISED MACHINE
LEARNING
•Use Cases on Activity Recognition

3. ABOUT ME: What excites me in AI?
•Semantic computing
•Interpretable, intuitive, human-readable knowledge representation
•Unsupervised and Deep Learning (e.g. ComputerVision)
•Automatic and powerful statistical learning
•Can semantics and statistics blend together?
• Cognitive neuroscience
•Can the brain’s learning mechanism inspire machine learning?
•Quantified Self
•Actionable, personalized well-being

4. RECENT ACTIVITIES
•Google Anita Borg Scholar
•/SYS/TUR Global
•Women in Math and CS
•India collab.NIT Meghalaya:
Wearables activity recognition

5. RECENT ACTIVITIES: India collab: IIT Kharagpur
Virtual tutor: India folklore dance:
Kinect touch free interaction

6. RECENT ACTIVITIES: SICN 2015 Fellow
Summer Institute of Cognitive Neurosciences
http://sicn.cmb.ucdavis.edu/
Program http://sicn.cmb.ucdavis.edu/SI_2015_Week_1_Schedule_1_1_15.pdf http://sicn.cmb.ucdavis.edu/SI_2015_Week_2_Schedule_1_2_15.pdf

7. RECENT ACTIVITIES:
Smart Dosing (with Nursing Science Dept., Finland)
Medication tray filling and dispensing in hospital wards

8. RECENTACTIVITIES:
First PersonVision for Activity Recognition
Unsupervised annotation (collab. with A. Betancourt, Netherlands)
• Recording public Autographer dataset
Betancourt, A., Morerio, P., Barakova, E. I., Marcenaro, L., Rauterberg, M., & Regazzoni, C. S. (2015). A Dynamic Approach and a New Dataset for Hand-
Detection in First Person Vision. In International Conference on Computer Analysis of Images and Patterns. Malta.

11. Currently:Visiting Scholar
HYBRID POSSIBILISTIC AND PROBABILISTIC
SEMANTIC MODELLING OF UNCERTAINTY FOR
SCALABLE HUMAN ACTIVITY RECOGNITION
•Prof. Lise Getoor
•Probabilistic Soft Logic (PSL)

12. RESEARCH AGENDA
• Supervised and unsupervised learning USE CASES:
1. Remote rehabilitation with Kinect
2. Human activity recognition in Ambient Intelligence
3. Semantic lifestyle profiling with wearables
4. Conciliating probabilistic and possibilistic Activity
Recognition with PSL
• Mentally stimulating future challenges

13. Classical data-driven Machine Learning

14. Knowledge-based Machine Learning
•Event calculus
•Situation calculus
•Rule-based systems
•Fuzzy logic

15. Knowledge-based Machine Learning
WHY Semantic Technologies & Ontologies?
•Semantic Web: well-defined meaning
•Ontology:
•In Philosophy: study of entities and their relations
•In Artificial Intelligence: “Explicit specification of a
conceptualization” [Gruber, 93]
• Web Ontology Language (OWL)

16. Background:
Ubiquitous computing and Ambient Intelligence
•Smart Space (SS)
•Context-awareness:
•Infrastructure and architectures
•End-user programming frameworks for AmI
16

17. Background: Ambient Assisted Living
•Usage of technology to provide assistance to people
who needs it in their daily activities, in the less
obstrusive way
•Aim: support older/disadvantaged people,
independent living, safety
•Includes: methods, systems, products and services
17

18. Case study 1: A Kinect ontology for physical
exercise annotation and recognition
•Active Healthy Ageing project (EIT Digital)
with Philips Personal Health Labs (PHL)
•Sensor data aggregation platform
18

19. 19

20. Kinect for remote rehabilitation DEMOS
•Sit-to-stand test
https://www.youtube.com/watch?v=g8HOtFTk80c
• Remote monitoring of post-surgery
rehabilitation exercises on shoulder, knee, hip
https://www.youtube.com/watch?v=XL4JexDNs-Q

21. Ontology features
Skeleton tracking (bone joint rotations + bone orientations)
21

22. Exercises &Workouts Ontology
22

23. Kinect ontology: examples of use
23
• Example 1: Defining basic movement (Stand, BendDown,
TwistRight, MoveObject, etc.)
• Example 2: Provide workout feedback (# series in time, quality
comparison with medical guidelines)

24. Kinect ontology: examples of use
24
• Example 3: Historic analysis can monitor posture quality in time. E.g. having back
less straight than 1 year ago-> notify to correct/prevent on time.
• Example 4: Notifications for office workers sitting too long/ not properly

25. Case Study 2: PhD (Cum laude, Finland-Spain):
Semantic and fuzzy modelling for human behaviour recognition in
Smart Spaces, a case study on Ambient Assisted Living
Ros et. Al. 2011

26. 26
SPAIN: 15m of elders in 2049 (1/3 of the population) (INE)
FINLAND population 65+ years: 18.14% [1]
• [1] http://www.finnbay.com/media/news/government-prepares-to-set-out-new-requirements-for-senior-caretakers/

27. PHD OBJECTIVES
27
•Understand Smart Spaces
•Human Activity Modelling and Recognition
•Program Smart Spaces

28. PHD OBJECTIVES
•Understand Smart Spaces
•Human Activity Modelling and Recognition
•Program Smart Spaces
28

29. Activity Recognition in Smart Spaces 29
[Image: http://www.businesskorea.co.kr/sites/default/files/field/image/smart%20home.jpg + The noun project]

30. Human Activity Recognition
30

31. Handling uncertainty, vagueness and imprecision
•Broken/ missing sensors
•Incomplete data, vagueness
•Different ways of performing activities
• Different object usage, duration, etc.
•Behaviour change
31

32. Tools
32

33. 33
[CONON Context Ontology]
Methods: Ontologies

34. 34
JULIOANA MARIA
NATALIA
Has Brother
Has Mother
Methods: Ontologies

35. 35
JULIOANA MARIA
NATALIA
Has Brother
Has UncleHas Mother
Methods: Ontologies

36. Methods: Fuzzy Logic
WHY fuzzy (description) logics and fuzzy ontologies?
•Real life is not black & white
•Classical (Crisp) Logic:True/False
•Fuzzy Logic: [0, 1]
• e.g. blond, tall
•For automatic reasoning about uncertain, vague or imprecise
knowledge
•For natural language expressions
[Bobillo 2008 fuzzyDL:An Expressive Fuzzy Description Logic Reasoner:
http://gaia.isti.cnr.it/straccia/software/fuzzyDL/intro.html] 36

37. 37
[Image: http://www.harmonizedsystems.co.uk/]
Example:Take medication

38. A fuzzy ontology for activity modelling and recognition
38
Classes, Individuals, Data Properties and Object Properties
SUBJECT PREDICATE OBJECT
User performs activity Taking medicine =
(0.3 User performs sub-activity reach Cup or Medicine Box)
(0.3 User performs sub-activity move Cup or Medicine Box)
(0.1 User performs sub-activity place Cup or Medicine Box)
(0.1 User performs sub-activity open Medicine Box)
(0.1 User performs sub-activity eat Medicine Box)
(0.1 User performs sub-activity drink Cup)

39. 39
Surveying
Activity
Recognition
techniques

40. 40
Surveying
Activity
Recognition
techniques

41. Surveying
ontologies for
activity
modelling
41

42. AR Ontologies ranking: domain coverage evaluation
42

43. OOPS! (OntOlogy Pitfall Scanner!)
evaluation
43

44. Case study 1:
A fuzzy ontology for
AR in the office/work
environment
44

45. 45

46. 2-phased algorithm:
1. Sub-activities (data-driven phase)
2. High-level activities (knowledge-based phase)
Validation: CAD-120 dataset:
•10 sub-activities, 10 activities, 10 objects, 4 users
46
Hybrid activity recognition with fuzzy ontologies

47. Cornell Activity Dataset [Koppula et al. 2013]
47

48. Hybrid data-driven and
knowledge-based
activity recognition
48

49. a) Data-driven sub-activity recognition phase
49

50. b) Knowledge-driven sub-activity recognition phase
50
Ontological
definitions

51. Ontological definitions: object interaction
51

52. Ontological definitions: object affordances
52

53. 10
semantic
rules
53

54. SUB-ACTIVITY prediction accuracy
54

55. 55
ACTIVITY prediction accuracy

56. Activity recognition
- comparison with state-of-the-art
56

57. Activity recognition
times (ms)
57

58. PHD OBJECTIVES
•Understand Smart Spaces
•Human Activity Modelling and Recognition
•Program Smart Spaces
58

59. Deployment: Programming Smart Spaces
59
Ros et. Al. 2011

60. A visual language to configure the Smart Space behaviour
•TARGET USER:
a) Developer
b) Non-technical background
•AIM:
•Rapid & easy programming of applications
•Improve interoperability and usability
60

61. Programming environments for novice programmers
61
[Scratch] [IFTTT]

62. PROPOSAL: SS visual language mapping to OWL 2
62

63. PROPOSAL: Smart Space visual programming
63

64. PhD main contributions
1. A set of ontologies to model human behaviour and tackle
uncertainty and vagueness inherent to real life
2. An architecture that integrates Semantic Web and Fuzzy Logic
for interpretable activity recognition
3. A hybrid knowledge-based and data-driven algorithm for real-
time, robust activity recognition (84.1% prec.)
4. Design & development of a toolbox for non-expert users and
rapid programming of Smart Spaces
[4 Journals -3 on Q1-, 9 conf. Papers. Google Anita Borg, Nokia and HLF scholar. University entrepreneurship award]
64

65. Startup technology transfer:
65

66. USE CASE 3:
Semantic lifestyle profiling with wearables
Can we recognize lifestyle patterns automatically?
1. Provide meaning to large heterogeneous data
-Interpretable, actionable insights
2. Knowledge-based methods & uncertainty handling
-Behavior vs Profile recognition
Day routines and lifestyles:
•Work/shop-aholics, Gym addicts
•Pet/ Partner/ Kids
•Retired/ Worker/ On holiday

67. USE CASE 4:
Activity recognition with Probabilistic Soft Logic
•Can manual work be automated?
•Can we improve…
•Model & rule learning
•Accuracy
•Scalability
•Genericity

68. Probabilistic Soft Logic (PSL) :
Expresses collective inference problems
mapping logical rules to convex functions
(defining a hinge-loss Markov Random field).
• FOL Predicate: relationship, property or role
• Atom: (continuous) random variables
• Rule: dependencies or constraints
• Set: aggregates
PSL Program = Rules + Input DB
[PSL (open-source): psl.umiacs.umd.edu]

69. PSL advantages against fuzzy OWL
•Statistical relational learning
•Adds probabilistic component to possibilistic one
•Captures cyclic dependencies
•Rule weight & latent variable learning
•Scalable (convex optimization) learning
•Most probable explanation (MPE) inference

70. Current work @LINQS Lab
•Can we seamlessly blend…
•Knowledge-based and data-driven mechanisms
•Supervised and unsupervised learning
for a general activity recognition framework?
•CanTIMED streams be handled naturally?
•Cost-sensitive, Online model learning and evolution
•Can models balance
•Flexibility and reproducibility?
•AccuracyVS deviation/anomaly detection?

71. General activity recognition framework
AIM:
Automate heuristics while maintaining rich semantic granularity:
•Context-awareness
•Object interaction, cardinality, recursion, rule subsumption
•Unordered/ordered (sub)sequences of sub-activity-object pairs
•Min/ max pattern sequential repetition in Δt

72. Activity rule examples (PSL)
performsSubActivity(Object, ObjPosition,Time)
m.add rule:
performsMove(MedicineBox, P, T1) &
PerformsDrink(WaterGlass, P, Tn)) >>
PerformsTakingMedicine(Tn),
weight : 10;

73. Future Challenges in Activity Recognition
•Multiple human sensing
•Parallel/interleaved activities
•Automatic ontology learning and evolution
•Reduce manual work
•(FOL/DL) Logics support for temporal constraints
73

74. • Unsupervised activity modelling
• First camera visionAR
• Automatic dataset annotation
• Wearables sparsity and uncertainty
• Scaling and real-timeness
74
Future Challenges in Activity Recognition

75. TAKE HOME MESSAGE
•Supervised ML: Statistical and probabilistic approaches
•Unsupervised ML: Interpretable knowledge-based techniques
Both are needed!
Domain expert knowledge and common sense knowledge:
Classical ML & Deep Learning: unable to exploit it!

76. THANKYOU!
Brainstorming/collaboration ideas, tips, pointers are welcome! 
Natalia Díaz Rodríguez nadrodri@ucsc.edu
https://research.it.abo.fi/personnel/ndiaz
COST Action on Architectures, Algorithms and Platforms for Enhanced Living
Environments: aapele.eu and Finnish Foundation for Technology Promotion

77. Kinect for remote rehabilitation DEMOS
•Sit-to-stand test
https://www.youtube.com/watch?v=g8HOtFTk80c
• Remote monitoring of post-surgery
rehabilitation exercises on shoulder, knee, hip
https://www.youtube.com/watch?v=XL4JexDNs-Q

78. Comparison with existing state-of-the-art (sub-activity and
activity recognition modules)
78

79. Human Activity Recognition
A crucial but challenging task in Ambient Intelligence
and AAL. Requires:
Context-awareness and heterogeneous data sources
Training data: examples
Common-sense knowledge /domain experts
Adaptation of behaviours
Alzheimer, Parkinson
79

80. ACTIVITY prediction accuracy (ideal situation)
(100% accurate sub-activity prediction) 80

81. SUB-ACTIVITY
prediction: accuracy results
81

82. 82
Fuzzy KB and rules in fuzzyDL

83. Smart Space Architecture: Smart-M3
83

84. 84

85. Equivalent SPARQL Query
85
Each rule is converted into a SPARQL query, which can be
transformed into a Smart-M3 subscription.

86. Crisp to fuzzy OWL query mapping to improve
semantics and usability
86

87. Handling uncertainty reasoning when
programming Smart Spaces
• Fuzzy reasoners: expressivityVS
computational requirements and
platform versatility:
• Best compromise: fuzzyDL
87

88. Activity recognition
Algorithm

89. 89

90. Ontology classes, data & object properties
90