3. 1. Determine Sensors
Sensors Critical to Determine Context
•
•
•
•
•
Hard Sensors – GPS, Accelerometer, Gyro, Temperature, Humidity, etc.
Soft Sensors – Calendar, Facebook Friends, Profile, Likes/Dislikes, History, etc.
Mobile Devices – Arrays of sensors, with new ones being introduced all the
time (e.g. fingerprint democratized with Android)
Sensor Co-processors – Sensors moving to co-processors means lower power
consumption (which means that sensors can be on all of the time)
Bluetooth LE – Greatly extends the the range and types of sensors (e.g. indoor
proximity and awareness)
3
5. 2. Sensor Aggregation
We Don’t
Hard Sensors
GPS
•
Accel.
•
Temp
Noise
•
Step
Need *All* of the Raw Sensor Data to Determine Context
Augmented Sensors – Algorithm that augments or abstracts existing sensors
(e.g. “Shaking” using the accelerometer)
Extended to Motion APIs – Google introducing standing, walking, cycling,
driving “sensors” in Android 4.3
Vendor Opportunities – Multiple vendors building SDKs for sensor aggregation
– inc. Semusi (Male/Female “sensor” from accelerometer and gyro)
Soft Sensors
Profile
Schedule
History
Likes/Dis
.
Social
etc.
5
8. 3. Send to Non-Relational Data Store
Goal is
Hard Sensors
to capture aggregated sensor data quickly and efficiently
Collections
• Schema-less– No need to care about schema or structure
Accel. Fire and Forget – No need to send confirmation back to client, or handle
Geo
•
Temp missed messages or retries
Motion
• Scale and Partitions – Data store should be a single entry point that
Noise
Environ.
automatically handles scale and partitioning
GPS
Step
etc.
Soft Sensors
Profile
Schedule
Events
History
New Like
Likes/Dis
.
Friend
Social
etc.
etc.
8
11. 4. Determine Perceived Context
Analyze
Hard Sensors
data store to determine one or more perceived contexts
Collections
Simple Matches– Single sensor data can often reveal simple matches (e.g.
Accel. movement or in/out geo-fenced location)
Geo
Non-Relational
•
Temp Combined Matches – Multiple parallel or serialized sensors to reveal more
Motion
Store
complex matches (e.g. movement with external GPS, combined with loss of
Noise
signalEnviron.walking activity might indicate parked in underground parking
and
Step structure).
etc.
• Learned Matches – Multiple sensor data used to reveal matches through
Soft Sensors
patterns (e.g. driving detected at same time every day might indicate
Profile commuting, which is turn can infer place of work)
• Timing – Context often derived from “time blocks” of sensor data – e.g. what is
Events
Schedule
happening now vs. what has happened over the last 30 minutes
GPS
•
History
New Like
Likes/Dis
.
Friend
Social
etc.
etc.
11
14. 5. Triggers on Context Change
Triggers
Hard Sensors
used to monitor change inQuery
context
Perceived
Context
Collections
Change in Context– Move from one perceived context state to another (e.g.
At Work
Accel. from in Geo
office to commuting)
Non-Relational
•
Temp External Data Source– Triggers often rely on external data source (which can
Motion
Store
Commuting
be interpreted as another soft sensor). Could include environmental data, move
Noise
Environ.
listings, weather forecasts, etc.
Step Machine Learning– Often benefits from some kind of ML to help determine
etc.
•
In Traffic
change in context across multiple sensors, especially in recommendations
Soft Sensors
space
GPS
•
At Home
Profile
Schedule
Events
History
New Like
Likes/Dis
.
Friend
Social
etc.
etc.
Watching
TV
Asleep
Make
determination
etc.
14
17. 6. Invoke Functions
Certain
Hard Sensors Triggers
GPS
•
Accel.
Query
Used to Invoke Functions
Perceived
Context
Collections
Take Action– Functions take action based At Work
on triggers.
may or Geo not involve some user input.
may
Temp
Motion
Noise
etc.
Commuting
House too
cold
In Traffic
Soft Sensors
At Home
Profile
Schedule
Events
History
Friend
Social
etc.
Timing
Loop
Flipping
channels
Better
movie
New Like
Likes/Dis
.
that
Started
commute
Environ.
Step
Non-Relational
Store
Context Change
Invoke an action
(Triggers)
etc.
Watching
TV
Asleep
Make
determination
etc.
External
Data
17
20. 7. User Interaction and Feedback Loop
Interact
Hard Sensors
Perceived
with user to notify them ofQuery action, and/or provide feedback
any
Context
GPS
Collections
Step
etc.
Context Change
function
(Triggers)
• User Notification– Notify the user of the result from a
At Work
Accel. User Interaction/Confirmation – Get confirmation from the user that a function
Geo
•
Functions
Started
shouldMotion happen, Non-Relationalrecommendation.
really
or provide a
commute
Temp
Store
Commuting
• Feedback Loop – Provide option for user to submit feedback based on the Turn on heating
House too
Noise
Environ.
cold
function (like, dislike, rating, etc.)
Adaptive application
In Traffic
Soft Sensors
At Home
Profile
Schedule
Events
History
Friend
Social
etc.
Flipping
channels
Better
movie
Make
recommendation
Proactive application
New Like
Likes/Dis
.
Watching
TV
Timing
Loop
etc.
Asleep
Make
determination
etc.
External
Data
20
24. Types of Context-Aware Applications
Adaptive
•
•
•
Act on behalf of the user
Try to adapt to the user’s context
Often based using single-trigger
•
•
•
Turn on heating when leave for home
Phone to silent in meetings
Offer music library when get on bus
Proactive
•
•
•
•
Involving the user
Often tied to a recommendation engine
Often requires user-interaction
Often uses multiple triggers
•
Recommend movie to watch when
watching TV
Display offer based on history when
walking through grocery store
•
•
Need to be careful if building adaptive
UI, especially if hiding/showing
features
•
UI needs to be unobtrusive, and easy to
dismiss
24
26. Challenges
Context Mismatch
•
•
•
Sometimes the perceived context is just wrong…
“You really think I want to do this now?!?!”
Multiple contexts help accuracy, and feedback mechanism is critical
Being Creepy
•
•
•
Sharing everything that you know about the context/trigger can appear creepy
“Your Mom’s birthday is tomorrow, and you’ve missed the last three years in a
row. Do you want to pull into the 7-Eleven ahead? They have flowers on sale…”
Reveal only enough to let the user deduce the same context, and feedback again
is critical
Being Annoying
•
•
•
Interruptions should follow normal human behaviors
“I wanted to let you know that I’ve turned the heating on” – “Not now, I’m
driving!”
Knowing the context of the recipient during the UI loop is also important
26
28. Examples of Opportunities
Retail
•
•
Opportunistic product offers while
shopping
Shopping recommendations based on
current context
Real Estate
•
Contextual awareness of prospective
buyers searching for properties
Gaming
•
Using context to enhance the gaming
experience of patrons
Field Employees
•
Context-aware applications that helps
field-based employees become more
productive
Home Automation
•
Context-aware applications that interact
with other systems in the home
Travel Applications
•
More intelligent travel applications
through context
28
30. References
•
•
•
Shilt, Adams, Want (1994): Proceedings of the Workshop on Mobile Computing
Systems and Applications
• http://www.interactiondesign.org/references/conferences/proceedings_of_the_workshop_on_mo
bile_computing_systems_and_applications.html
Swati A. Sonawane: Context-Aware Computing
• http://www.slideshare.net/swatibaiger/context-aware-computing14084995
Albrecht Schmidt (2013): Context-Awareness, Context-Aware User Interfaces, and
Implicit Interaction
• http://www.interaction-design.org/encyclopedia/contextaware_computing.html
30