This talk will review the current status of human behavior monitoring methods to develop an intelligent software assistant for users’ daily lives. Google Now like existing Apps will be reviewed in comparison with our approaches. Technical aspects for system implementation will be thoroughly discussed to give a clear picture on the system including popular sensors, human activity event processing methods and behavior pattern mining approaches. Personal opinions and ideas will be freely discussed during the talk.

1. How To Monitor Human Behaviors:
Current Status & Our Approaches
October 24th, 2013
Pil Ho Kim, pilho.kim@unitn.it
University of Trento, Italy
1

2. Current Status
• What did you feel when you read the title of this talk?
• Then let’s see what is going on now. We will review three
software applications:
• Google Now
• Moves
• Our work

3. Google Now
• Objective: An Intelligent Personal Assistant (Know-it-all)
• http://www.google.com/landing/now/
Images © Google

4. Main Services (as of now)
#1. Manage your day
#2. Stay connected
#3. Be a local
4
Images © Google

5. Information Sources
• Gmail Cards (GC)
• Web History (WH)
• Synched Calendar (SC)
• Current Location (CL)
• Google Plus (G+)
• User Reminder (UR)
• Current Time (CT)
• “Remind me” Button (RB)
• Google Offer Service (GO)
• User Selection (US)
• Event Booking (EB)
• Google Finance Portfolio (GF)
• Smart TV (ST)
• Collaboration with 40 more
companies (http://
www.google.com/landing/now/
integrations.html)

6. #1. Manage Your Day
Sources: Gmail Cards (GC), Web History (WH), Synched Calendar (SC), Current Location (CL), Google Plus (G+),
User Reminder (UR), Current Time (CT), “Remind me” button (RB), Google Offer service (GO)
•
Boarding pass (GC)
•
•
• Activity summary (CL)
•
• Next appointment (SC, CL)
•
• Weather (CL)
•
• Traffic & Transit (CL, WH)
•
• Flights (GC)
•
• Hotels (GC)
•
• Restaurant reservations (GC, CL) •
Events (GC, RB)
Packages (GC)
Friends’ birthday (G+)
Your birthday (G+)
Time reminders (UR, CT)
Location reminders (UR, CT, CL)
Event reminders (RB)
Saved offers (GO)
Car rentals (GC)

7. #2. Stay Connected
Sources: Gmail Cards (GC), Web History (WH), Current Location (CL), Google Plus (G+), User Selection (US), Event
Booking (EB), Google Finance Portfolio (GF), Smart TV (ST)
•
Zillow (Search nearby real
estate listings) (WH, CL)
•
• Movies (WH, CL)
• Fandango (Movie booking)
Sports (US)
(EB)
• Concerts (WH, CL)
• Stocks (WH, GF)
• Public alerts (WH, CL)
• Developing story &
breaking news (WH)
• Research topic (WH)
• New albums (WH)
• New books (WH)
• New video games (WH)
• New TV episodes (WH)
• TV cards (ST)
• Concert ticket (GC)

8. #3. Be A Local
Sources: Current Location (CL)
• Public transit (CL)
• Places (CL)
• Nearby attractions (CL)
• Nearby events (CL)
• Nearby photo spots (CL)
• Translation (CL)
• Currency (CL)
• Time at home (CL)

9. Supported Platforms
Desktop supports are under way through Chrome notification:
source: https://support.google.com/mail/answer/1075549?hl=en
Images © Google

10. User Interfacing
• Majority of natural language interfaces through
texts and also speech.
• Cards, cards and cards (Unified representation of
user’s personal events).
• Sources from the user through interactions and
monitoring in consideration of contexts from
outer sources.
• Notification methods: App and widget for mobile
and notification bar for desktop

11. What Others Think on Google Now?
Google Now for iOS review: straddling the creepy line
(May 2nd, 2013),The Telegraph
• As more services are added, it will keep track of
more of your choices. These things are either
creepy or cool, depending on your definition of
privacy.
• Unusually for Google it’s a beautiful, graceful
interface that can present a lot of information in
a concise, aesthetically pleasing format.
• Features can be turned off with impressive
granularity. Also the battery life is issued.
Source: http://www.telegraph.co.uk/technology/mobile-app-reviews/10032788/Google-Now-foriOS-review-straddling-the-creepy-line.html

12. What Others Think on Google Now?
Review: Google Now useful as a supplement to Siri (4 May,
2013), USA Today
• Google Now’s power lies in giving you information
you need to know before you have to ask. It works
best as a supplement to Siri, rather than a
replacement.
• Google Now's judgments on what information I
need to know can be sketchy at times.
• Despite getting stray information at times, I find
Google Now useful enough to leave it on. I could
always customize the service by telling it never to
give me stock quotes, for instance.
Source: http://www.usatoday.com/story/tech/2013/05/04/google-now-siri/2129755/

13. What Others Think on Google Now?
Google’s Virtual Assistant Google Now Is Awesome - Here’s Why I Never
Want To Use It (22 June, 2013), Business Insider
• Google Now puts Siri to shame – it's more accurate and far more
versatile.
• Trouble question: It begs to be relied upon and integrated into
your lifestyle in a major way. Even a casual Google Now user is
still feeding all his data to it every time he uses the web.
• The intrinsic nature of Google Now is that it's designed to be
blindly relied upon, to be a "life aid." What happens when it goes
kaput for some reason? All those things that you were used to not
having to think about become problematic.
• I'm in charge of my phone. My phone's not in charge of itself.
Google Now nearly represents the phone coming to life, acting on
its own and without your direction. It becomes slightly less your
phone and you become a slightly less mindful.
Source: http://www.businessinsider.com/the-case-against-google-now-2013-6

14. What Others Think on Google Now?
Google Now could be the injection of intelligence smart watches
need (16 Oct, 2013), theguardian
• Google is reckoned to be working on a Google Now-
focused smart watch codenamed ‘Gem’ which could
finally make smart watches intelligent (Confirmed yesterday -- Source #2).
• Industry experts reckon none of the current smart
watches has truly "smart" functionality. And that leaves
the door wide open for something that multiple sources
are murmuring about: a Google Now-powered
wearable.
• Google started a project (Codename “Gem”) to bring
Google Now to the smart watch. FYI, Google Glass
already supports Google Now.
Source #1: http://www.theguardian.com/technology/2013/oct/16/google-now-gem-smartwatch
Source #2: http://online.wsj.com/news/articles/SB10001424052702304655104579165080029933904

15. Development History
• Late 2011: Enhancing existing Google Voice Search
(Codename: Majel)
• June 27, 2012: Google Now appeared in Android 4.1 Jelly
Bean Google I/O
• October 29, 2012: UI changed to Cards with information
pulled from Gmail account, user location, search history
and more have been added since then.
• March 21, 2013: iOS version of Google Now
• May 15, 2013: Google I/O 2013 announced the upcoming
release of Google Now on desktop platforms; the feature
will be accessible only via Google Chrome or Google
Chrome OS.

16. Trends on “Google Now”
Google Now
comes to iPhone
Images ©
Google Trend

17. Today Google Now Told Me..

18. Personal Review (I)
• Great example in merging heterogenous
sources to provide intelligent &
personalized services.
• Shows how Google monitors personal
behaviors and what it classifies.
• The concept of Google Card adds strong
flexibility in extending personalized services.

19. Personal Review (II)
• It looks clear that Google runs at least
multiple instances of real-time listeners
(CEP) for each Google Now user.
• Looks promising but not working well
for me. Accuracy (in multiple aspects)
could be an issue.
This email was exchanged on Mar
2013. The content is in fact a
customer service email asking
about my old flight travel
(11/10/2012 - 24/10/2012), not
today (24/10/2013).

20. Moves
• Tracks the amount of exercise -- activity tracking app.
• http://www.moves-app.com

21. Features
Images ©
Moves-app.com

22. Our Approaches
+
Google Now
Moves
And keep you data under your hood !
Images © Google
& Moves-app.com

23. Big Picture
Data
Semantic
27

24. Big Picture
Sources
Data
Event
Processor
27
Semantic
Knowledge

25. Events & System
• We need to develop a system for event detection.
• We mean an event as a change from multiple sources.
• Personal events are so diverse and non-static.
• So we do not define an event.
• But our system provides a way for users (or developers)
to define events of their own with good enough freedom.
• Event streams can be modeled as a pattern to build up a
personalized knowledge base (i.e. an ontology linking a
word or words in semantic relations to low-level
changes).

26. Implementation Strategies
• The amount and quality of information sources directly
affects the machine intelligence.
• The challenges are then (1) a backend to collect
enormous data from various sources and (2) methods
to mine big data.
• We have currently implemented (1), which will get
better, and are working on (2).

27. Mining Data to Events
• Many methods exist like signal processing, statistical
analysis and probabilistic state modeling methods.
• Why Complex Event Processing (CEP)?
• Rich supports for temporal logics and complex
controls.
• Encapsulate above methods as event sources using,
what are called, Platform Independent Model (PIM)
and Computational Independent Model (CIM) that
are under various testing for implementation.
• Enable continuous learning rather than offline
learning. Though, it should be noted that real-time
pattern modeling needs significant efforts.
33

28. Complex Event Processing
Video © Opher Etzion, IBM Research
CEP is a technology to help you process complex events.
So don’t be scared but start using it as a tool.
34
Source: http://www.slideshare.net/opher.etzion/event-semantics-and-modelpresented-at-the-event-based-multimedia-workshop-presentation-at-acmmultimedia-2013

29. Information Sources (as of now)
•
• Ambient Temperature (AT)
• Battery Status (BS)
• Light Level (LL)
• Gravity (GV)
• Gyroscope (GS)
• Linear Acceleration (LA)
• Orientation (OE)
Accelerometer (AC)
• Magnetic Field (MF)
• Rotation Vector (RV)
• GPS Location (GL)
• Network Location (NL)
• WIFI (WF)
• Pressure (PL)
• Relative Humidity (RH)
We need more sources to be intelligent.

30. How to Detect Events?
Method #1. User Records
36

31. How to Detect Events?
Method #2. Manual Data Analysis
37

32. How to Detect Events?
Method #3. Automated Event Detection Using CEP
38

33. Esper?
http://www.espertech.com
•
An esper refers to an individual capable of telepathy and other similar
paranormal abilities.
•
•
Sorry.
•
Esper is an extensible middleware supporting:
Esper is an open source event stream processing (ESP) and event correlation
engine (CEP).
•
•
•
•
•
•
Java, .Net, Map or XML events
Runtime statement management
API or configuration driven
Plug-in SDK for functions, aggregations, views and pattern detection extensions
Agile development and testing, integrates with Eclipse and JUnit
Adapters: Adapters to any possible transport exist through integration with
Apache Camel, ServiceMix and Mule ESB. EsperIO and Enterprise Edition
provide additional adapters.
39

34. Esper Example (First Try)
Tracking user’s
indoor &
outdoor location
40

35. Esper Example (First Try)
Tracking user’s
indoor &
outdoor location
Too complex logic in the
query statement to output
an event detection status
Stream
dependent
domain
knowledge
Sensor-specific source selection
40

36. Current Supports:
Now You Can Write Much Less
41

37. Current Supports:
Now You Can Write Much Less
An abstract event class that encapsulate heterogenous
sources to trigger next actions (ex. pushing/pull
notification, auto recording, pattern mining, etc.)
Generalized status retrieval
State changes are now modeled as a pattern.
Domain knowledge becomes
independent functions.
An abstract event class that
encapsulates heterogenous sources.
41

38. Thank You.
Any question?
43

39. References
Please refer following references for details.
• Opher Etzion, Modeling and semantics of events and
contexts, http://www.slideshare.net/opher.etzion/eventsemantics-and-model-presented-at-the-event-basedmultimedia-workshop-presentation-at-acmmultimedia-2013
• Google Now: http://www.google.com/landing/now/
• Moves: http://www.moves-app.com
44