With increasing ubiquity of wearable sensor based devices, smartphones and social applications, rich streams of personal data are being generated with phenomenal volume and variety. The quantified-self movement is increasingly becoming mainstream with easy availability of numerous commercial devices and growing interest in gaining self-knowledge. This presents an opportunity to create a contextual model of a person helpful to gain actionable insights at both individual and society level. But a single device or source may be insufficient to collect data on all aspects of a person's life, thus, multiple data streams need to be combined to detect and store meaningful life events. This poses several challenges related to selection, nature, granularity, reliability and representation. This work presents a unified framework for aggregating streams of heterogeneous spatio-temporal data, to create, visualize and analyze a chronicle of life events, called Personicle. A formal algorithm for collecting and pre-processing multi-dimensional data has been presented, to abstract high-level context from data streams. A concept lattice based algorithm for life-event recognition has also been explored. A web-based prototype as an application of the framework has been presented, which collects real life data from Google Calendar, Moves application, and Nike+ Fuelband, to create Personicle. An empirical evaluation of the prototype was done to understand the usability and measure the performance. Eight participants evaluated the web-based prototype for its usability. The user evaluation garnered 75 out of 100 points on the standard System Usability Scale (SUS) scale, thus grading the prototype usability as good.

1. 3/12/2014 ©2014 Parul Seth 1
PERSONICLE
Contextual and Actionable Chronicle of a Person’s Life
THESIS DEFENSE
By
PARUL SETH
parulseth@gmail.com | parul.seth@uci.edu | http://parulseth.com
PROFESSOR
RAMESH C. JAIN
(Co-Chair)
PROFESSOR
GLORIA MARK
PROFESSOR
YUNAN CHEN
(Co-Chair)
THESIS COMMITTEE

2. Evaluation & Conclusion
Applications
Proposed Solution
Opportunity
Motivation
Background
3/12/2014 ©2014 Parul Seth 2
Outline

3. P
e
r
s
o
n
i
c
l
e
C
o
n
t
e
x
t
u
a
l
A
c
t
i
o
n
a
b
l
e
C
h
r
o
n
i
c
l
e
P
e
r
s
o
n
’
s
L
i
f
e
3/12/2014 ©2014 Parul Seth 3
Let’s walk through the title…

4. L I F E
3/12/2014 ©2014 Parul Seth 4

5. P
e
r
s
o
n
i
c
l
e
C
o
n
t
e
x
t
u
a
l
A
c
t
i
o
n
a
b
l
e
C
h
r
o
n
i
c
l
e
P
e
r
s
o
n
’
s
L
i
f
e
3/12/2014 ©2014 Parul Seth 5
Let’s walk through the title

6. Person & Persona
3/12/2014 ©2014 Parul Seth 6
Person
Intents
Identity
Interests
Persona
At the micro level a person may be defined by her Identity, Interests and Intents (3Is).

7. P
e
r
s
o
n
i
c
l
e
C
o
n
t
e
x
t
u
a
l
A
c
t
i
o
n
a
b
l
e
C
h
r
o
n
i
c
l
e
P
e
r
s
o
n
’
s
L
i
f
e
3/12/2014 ©2014 Parul Seth 7
Let’s walk through the title…

8. 3/12/2014 ©2014 Parul Seth 8
From Calendar to Chronicles
Chronicles are chronologically ordered, reported facts and details about the
past events, arranged in a structure such as a timeline.
Calendar has a plan for future events

9. P
e
r
s
o
n
i
c
l
e
C
o
n
t
e
x
t
u
a
l
A
c
t
i
o
n
a
b
l
e
C
h
r
o
n
i
c
l
e
P
e
r
s
o
n
’
s
L
i
f
e
3/12/2014 ©2014 Parul Seth 9
Let’s walk through the title…

10. 3/12/2014 ©2014 Parul Seth 10
Actionable Goals
I want to sleep well, I have been to stressed out!
I want to exercise to loose weight! I want to be happy!
Of course, I want to live longer!

11. P
e
r
s
o
n
i
c
l
e
C
o
n
t
e
x
t
u
a
l
A
c
t
i
o
n
a
b
l
e
C
h
r
o
n
i
c
l
e
P
e
r
s
o
n
’
s
L
i
f
e
3/12/2014 ©2014 Parul Seth 11
Let’s walk through the title…

12. Events
Situations
Contexts
3/12/2014 ©2014 Parul Seth 12
Contexts, Situations, Events

13. 3/12/2014 ©2014 Parul Seth 13
Contexts, Situations, Events Exemplified

14. Life Events
3/12/2014 ©2014 Parul Seth 14
CORE LIFE EVENTS CONTEXTUALIZED LIFE EVENTS DOMAIN SPECIFIC LIFE EVENTS
eating at a restaurant
watching movie in a theatre
exercising at gym
sowing
irrigating
harvesting
Life Events are in which the object of interest, i.e. the person participates in the occurrence of the event.

15. …
3/12/2014 ©2014 Parul Seth 15
Motivation

16. 3/12/2014 ©2014 Parul Seth 16
Mobile Connectivity
+ Bandwidth
Internet
Social Networking
Increasing Computing Power
+ Data Universe
Wireless Sensors &
Actuators
Imaging
Genomics
Information Systems
Growing Dimensions of Ubiquitous Computing
Image Source: http://eecatalog.com/IoT/files/2014/04/Freescale-
Internet-of-Things-Tree.jpg?file=2014/04/Freescale-Internet-of-
Things-Tree.jpg

17. Growing Interest in Empowering the SELF through
Quantification
3/12/2014 ©2014 Parul Seth 17
Image Source: http://cdn.static-
economist.com/sites/default/files/imagecache/290-
width/images/print-edition/20120303_TQD003_0.jpg

18. 3/12/2014 ©2014 Parul Seth 18
This Growth is Generating More & More BIG DATA

19. 3/12/2014 ©2014 Parul Seth 19
Ok but is More Data, More Devices = More Answers?
More Data gives us the ability to ask more
Questions, only when this Data is
 Processed into Meaningful Information,
 Information is questioned to create Knowledge,
 Knowledge is capable of supporting Wisdom
to Think and Act
Data
Information
Knowledge
Wisdom

20. …
3/12/2014 ©2014 Parul Seth 20
Opportunity

21. 3/12/2014 ©2014 Parul Seth 21
Aggregation and interpretation of microscopic data
Knowledge about each individual to get a macroscopic view of the whole society
Uncover the reality of need, availability of resources and the possibility of service
To create a unified framework for:
Processing data and generating information
Gaining insights about an individual and the surrounding context
Detecting the occurrence of an event, to create a chronicle of past events
Understanding individual’s behavior, habits, thoughts, real-world interactions and actions.
Understanding Individual at Micro Level and Society at the Macro
Level
MACRO-LEVEL
To address the challenge of “connecting people to resources effectively, efficiently and promptly” by
identifying context, situation and events that help in exploring the needs.
MICRO-LEVEL

22. Central Objectives
3/12/2014 ©2014 Parul Seth 22
Given multiple multi-dimensional heterogeneous data sources:
• detect and create a collection of life events
• create a contextualized and actionable chronicle for a person
• store and display in a timeline based structure.
To shift the focus from data to people by carving Little Data from Big Data.

23. P
e
r
s
o
n
i
c
l
e
C
o
n
t
e
x
t
u
a
l
A
c
t
i
o
n
a
b
l
e
C
h
r
o
n
i
c
l
e
P
e
r
s
o
n
’
s
L
i
f
e
3/12/2014 ©2014 Parul Seth 23
Let’s walk through the title…one last time.

24. 3/12/2014 ©2014 Parul Seth 24
Personicle
Personicle (Person’s or Personal Chronicle) represents a chronicle of
a person’s life, containing a collection of Life Events
Each epoch of time can be mapped to a recognizable life event helpful for retrieval, processing, future
decision making and action taking.

25. …
3/12/2014 ©2014 Parul Seth 25
Proposed Solution

26. Layers of the Framework
3/12/2014 ©2014 Parul Seth 26
Data Collection
Information Creation
Meaning Making
Action Taking
OPPORTUNITY PROPOSED SOLUTION

27. Design Challenges, Considerations and Decisions
3/12/2014 ©2014 Parul Seth 27
CHALLENGES DESIGN DECISIONS

28. 3/12/2014 ©2014 Parul Seth 28
Schematic Diagram of Personicle Framework

29. …
3/12/2014 ©2014 Parul Seth 29
Data Collection

30. Contexts, Sensors, Data Sources
3/12/2014 ©2014 Parul Seth 30

31. …
3/12/2014 ©2014 Parul Seth 31
Information Creation

32. 3/12/2014 ©2014 Parul Seth 32
Personicle Builder Engine

33. 3/12/2014 ©2014 Parul Seth 33
Context and Feature Extraction Algorithm for Pre-processing
‘N’ Heterogeneous Streams
Let Si
t denote the data stream from ith source at equal time slot t, then,
Si
t ≡ {SAt
i,1,SAt
i,2, SAt
i,3,… SAt
i,N},
PC ≡ {PC1, PC2, PC3, … PCR},
where PCi contains attributes such as PAr,1, PAr,2, PAr,3,…
Aφ(wi) is defined as a map,
Aφ(wi): Si
t  {PAt,i
1,1(wi
1,1), PAt,i
1,2(wi
1,2), PAt,i
1,3(wi
1,3), …
PAt,i
2,1(wi
2,1), PAt,i
2,2(wi
2,2), PAt,i
2,3(wi
2,3), …
…
PAt,i
R,1(wi
R,1), PAt,i
R,2(wi
R,2), PAt,i
K,3(wi
R,3), … PAt,i
R,M(wi
R,M)},
where, PAt,ir,m(wir,m) represents the extracted mth Personicle
Attributes (PA) (that belongs to rth Personicle Context, PC) from ith
source along with its weight (wir,m).
Extracted attributes from N heterogeneous sources are merged together
using operator φ. Thus, the combined mth Personicle Attribute of rth
Personicle Context (PC) for time slot t, PAt
r,m, is computed as,
PAt
r,m = PAt,1
r,m(w1
r,m) φ PAt,2
r,m(w2
r,m) φ PAt,3
r,m(w3
r,m) φ …
PAt,N
r,m(wN
r,m)
PAt represents a set of Personicle Attributes extracted at time slot t,
PAt = {PAt
1,1, PAt
1,2,PAt
1,3 … ,
PAt
2,1, PAt
2,2, PAt
2,3 … ,
…
PAt
R ,1, PAt
R ,2,PAt
R ,3,… }.

34. 3/12/2014 ©2014 Parul Seth 34
Source Capability Descriptor

35. 3/12/2014 ©2014 Parul Seth 35
Contexts Descriptor

36. Locomotion: {motion-type: sitting}
Physiological: {activity level: low, steps: 0, calories:0}
Calendar: {scheduled-event-type: meeting, scheduled-event-duration: 60}
Location: {location-name: work}
Social: {participant: john doe, participant-person-relation: supervisor}
t = 5 minutes
Jan, 12, 2014 11:00 AM Jan, 12, 2014 11:05 AM
Personicle
Contexts
Personicle
Attributes
A Single Time Bucket within the Temporal Stream
3/12/2014 ©2014 Parul Seth 36

37. 3/12/2014 ©2014 Parul Seth 37
Life Event Recognition
For detecting life events in the Personicle Builder Engine,
a hybrid of the specification and learning based
approaches may be employed to take advantages of both
the approaches.

38. 3/12/2014 ©2014 Parul Seth 38
Some Definitions
“Formal Concept Analysis is a principled way of deriving a concept hierarchy or formal
ontology from a collection of objects and their properties.”
“Formal context in FCA is a triple, K = (G, M, I) where G is a set of objects, M is a set of
attributes and the binary relation I ⊆ G × M shows which objects possess which attributes.”
“Formal concept is a pair (G, M) of set of objects (the extent) and a set of attributes (intent).”
A Cross Table is a tabular representation of formal context, it contains object names in rows
to rows and corresponding attribute names in columns, of the table, and a checkmark (a
Boolean value) showing the binary relation between objects and attributes.
A Concept Lattice constructed using the cross table, by extracting all the possible concepts.
A concept lattice is a partial ordered set with binary joins and meets. With the partial order
relationship ≤ between two concepts (G1, M1) ≤ (G2, M2)  (G1 ⊆ M2) and (M2 ⊆ M1)

39. 3/12/2014 ©2014 Parul Seth 39
Cross Table for Lattice Construction
Home
Work
Restaurant
Mall
Unknown
Walking
Running
Cycling
Transportation
Unknown
Sedentary
Low
Medium
High
VeryHigh
Meeting
Class
Other
6AM-12PM
12PM-3PM
3PM-6PM
6PM-12AM
12AM-6AM
<=10Minutes
10-30Minutes
30-60Minutes
1-2Hours
>2Hours
Meeting X X X X X X X X X X X X
Walking X X X X
Running X X X X
Cycling X X X X X
Working X X X X X X X
Eating Breakfast X X X X X X X
Eating Lunch X X X X X X X
Eating Dinner X X X X X X X
Shopping X X X X X
Driving X X X
Excercising X X X X
Sleeping X X X X X X
Attending
Class/Seminar X X X X X X X X X X X
DurationCalendar TimeLocation Locomotion Activity LevelContexts
Life
Events
Attributes
(Intent)
Objects
(Extent)

40. 3/12/2014 ©2014 Parul Seth 40
Concept Lattice & Traversal for Life Events

41. Creating the Stream of Life Events
3/12/2014 ©2014 Parul Seth 41
The temporal collection of life events {LE1, LE2, LE3, LE4…} over a period of
time, after semantically identifying and merging the life event buckets, is the final
Personicle.
The occurrence of each life event, LE, is a set of {Object, Context, [Time]}
• Object may represent any real-world object, in this case, a person.
• Context is represented by the set of personicle attributes (PA) classified by the
high level personicle contexts (PC)
• Time represents the temporal interval t.

42. …
3/12/2014 ©2014 Parul Seth 42
Meaning Making & Action Taking

43. Meaning Making
3/12/2014 ©2014 Parul Seth 43
Operator Type Operators
Arithmetic +, -, *, /, %
Character ||
Comparison
=, !=, ^=, <>, >, <, >=, <=, IN, ANY/SOME, NOT IN, ALL,
[NOT] BETWEEN x and y, EXISTS, x [NOT] LIKE y [ESCAPE z]
Logical NOT, AND, OR
Set UNION, INTERSECTION, MINUS
Geospatial WITHIN, INTERSECTS, NEAR
Projection FIRST, MATCH, SLICE
Disjunction X OR Y EXISTS WHEN X OR Y EXISTS
Conjunction X AND Y EXISTS WHEN X AND Y EXISTS
Sequence
(X>Y OR Y>X), (X MEETS Y, OR Y MEETS X), (X OVERLAPS
Y or Y OVERLAPS X), (X STARTS Y OR Y STARTS X), (X
DURING Y), (X FINISHES Y OR Y FINISHES Y), (X=Y)

44. Sample Queries
3/12/2014 ©2014 Parul Seth 44
•How many physiotherapy visits (or other health visits) has a person made in the
past year or past five years? What does it say about the person’s health?
•How much did the person exercise (walking, cycling, running, swimming, gym) vs
travel vs walk, for the last three months of this year. How does this compare with
the same three months from the previous year? What caused the difference?
•Is the deterioration of person’s health related to the places she has visited in the
past six months?
•How often does this person eat at restaurants vs at home?
Spatio-temporal Queries
•Which days of the week the person is most active?
•What kind of cuisine does the person mostly prefers while eating out?Continuous Events Queries
•Is the person’s sleeplessness on particular days related to the number or nature of
events on those days?
•Is this person usually late for scheduled events?
•How does the person spend his/her days?
•How the person’s activity patterns relate to his/her parent(s) or friends.
•How does a person’s colleague spend her day for the same time they both spent at
work??
Pattern Queries
•How does weather affect the activity level of the person?
•Does walking fast relate to being late for a meeting?
•Analyze the amount of time spent with the family.
•Time spent outdoors vs indoors in summers and winters.
•Does the person attends social events, or is mostly busy with work related events?
Event Association Queries
•Create new event called “sleeplessness”. By assimilating sleep data and correlating
it with food intake and number of emails/meetings/phone calls, plot the affects.
•Create new event called “activeness”. By assimilating “high” and “very high”
activity levels for a time period then create activeness events.
Event Composition Queries

45. Applications
3/12/2014 ©2014 Parul Seth 45
Predictive
Intelligent
Calendar
Smart & Sensitive
Healthcare
Fitness
Elderly Care Child Care
Enterprise
Applications
Lifestyle/Interest/
Capability
Matching
Surveillance/
Sousveillance
Multimedia
Gaming
Lifestyle
Management
Secondary
Research

46. …
3/12/2014 ©2014 Parul Seth 46
Prototype Application: Lifestyle Management, Fitness, Retrospection

47. Iterative Design and Development of the Personicle
Prototype
3/12/2014 ©2014 Parul Seth 47
Analysis
Design
ImplementationEvaluation
Feedback

48. Functional Requirements
3/12/2014 ©2014 Parul Seth 48

49. Non-functional Requirements
3/12/2014 ©2014 Parul Seth 49

50. Tools & Technology
3/12/2014 ©2014 Parul Seth 50

51. Architecture: Structural Viewpoint
3/12/2014 ©2014 Parul Seth 51

52. Architecture: Functional Viewpoint
3/12/2014 ©2014 Parul Seth 52

53. Nike +
Moves
Google Calendar
Personicle
Personicle Timeline Visualization

54. I took 832 steps to walk
from APS to my Health
Informatics class in SSL!
Nike +
Moves
Google Calendar
Personicle
Personicle Timeline Visualization

55. 3/12/2014 ©2014 Parul Seth 55
Analysis & Query Engine Interface

56. 3/12/2014 ©2014 Parul Seth 56
Life Events Calendar

57. …
3/12/2014 ©2014 Parul Seth 57
Evaluation and Discussion

58. Life Event Recognition Accuracy
3/12/2014 ©2014 Parul Seth 58
Data Collection
• Participant 1, Five months data
• Participant 2, One month data
Chosen Life Events
• 12 Life Events-
• Working, Walking, Meeting, Sleeping, Cycling, Running, Exercising, Shopping, Transporting, Eating
Dinner, Eating Lunch, Attending Class/Seminar
Methodology
• Set of tasks in the task sheet for recording real-life activity data in their natural setting
Results

59. Usability Study
3/12/2014 ©2014 Parul Seth 59
Verify the
applicability and
existence of Design
Decisions
To unveil any existing
issues or areas of
improvement.
To generate new
ideas.
To collect information
to make the future
system more effective
and usable.
GOALS METHODS
Participants filled the Usability Questionnaire
(10 SUS Usability Questions, 7 Subjective Questions)
Prototype and its Screencast was evaluated
8 Participants Recruited (snowball sampling)
RESULTS
The final SUS score for the first ten questions in the questionnaire was 75 points out of
100.
Participants provided a positive feedback for the prototype, along with valuable
suggestions for improvements

60. Selected Feedback
3/12/2014 ©2014 Parul Seth 60
“Being able to see my active data
history within a calendar is very
helpful”
LIFE EVENTS CALENDAR TIMELINE VISUALIZATION
“Yes, can’t think of any other way
(for visualizing Personicle) that
would be better”
QUERY & ANALYSIS
“It looks great, but if I choose to
geek-out more during analyzing
my data, it would be nice to be able
to build my own queries”
APPLICABILITY & IMPROVEMENTS
“I feel that the map should scale to
relevant locations. For example, the
map is zoomed out to California for
a query about walking across
campus”
“I think, it could be incredibly
helpful for managing fitness data…
(The system is useful) anywhere
that (sic) peoples’personal data can
be used for marketing, advertising,
predicting future behavior. Courts
could use it to prove where people
are”…”Child care”
“Privacy controls. This is a very
powerful too, but could also be
abused by parties interested in
our personal data…I would
strongly recommend extensive
privacy/disclosure controls. This
sort of tool is potentially
powerful for end-users, but also
suggests a lot of
surveillance/tracking abuse
issues”
“I would like the actual system to
be as responsive as the prototype”
“I would like to have this on a
mobile device as well”

61. Enhancements & Future Work
3/12/2014 ©2014 Parul Seth 61
Seeded Life
Event
Recognition
Enhancing User
Control
Standardization
of Data Storage
Model
Real-time
Feedback
Predictive
Analytics
Improving User
Experience
Ethnographic
Field Study

62. Contributions
3/12/2014 ©2014 Parul Seth 62
Defined the unified framework for collecting and processing heterogeneous data streams to
identify and create a stream of events that can be analyzed to offer actionable insights.
Presented and formalized the algorithm for collecting and pre-processing multi-dimensional data
by identifying high-level contexts from multiple heterogeneous sources.
Presented an algorithm for life event recognition and creation of a unified spatio-temporal
representation i.e. a chronicle of life events.
Identified the kinds of queries that people may ask the framework for analysis, reflection and
action.
Illustrated specification, development and empirical evaluation of a web-based prototype, as an
instance of the framework, for collecting data from three sources of personal data to build a
stream of life events visualized and analyzed in a spatio-temporal timeline based structure.

63. 3/12/2014 ©2014 Parul Seth 63
Thank You!
PARUL SETH
parulseth@gmail.com | parul.seth@uci.edu |
http://parulseth.com
©Bill Watterson