A talk I gave at ICMI 2010, held in Beijing, China. The full paper reference is: El Ali, A., Nack, F. & Hardman, L. (2010). Understanding contextual factors in location-aware multimedia messaging. In Proceedings of the 12th international conference on Multimodal Interfaces, 2010, Beijing, China.

1. Understanding Contextual Factors in
Location-aware Multimedia Messaging!
Abdallah
“Abdo”
El
Ali
Frank
Nack
Lynda
Hardman

2. Outline!
I. Introduc*on
II. Prototype
III. Diary
Study
IV. Results
V. Conclusions
&
Future
Work
2 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

3. Motivation!
Ubiquitous
Compu*ng
promises
to
populate
our
daily
lives
with
specialized
‘context-­‐aware’
services
that
enhance
our
experience
easier
(Weiser, 1995)"
friendlier
efficient
Loca*on-­‐awareness
(GPS)

Context-­‐awareness
(human
intent?)
(Dey, 2002)"
3 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

4. ! Case Study: Location-aware
Multimedia Messaging (LMM)!

Geo-­‐tagged
mul*media:
photos,
text,
video,
audio

Made
at
a
loca*on
and
viewed
at
that
loca*on
Assump:on:
LMMs
can
reflect
cultural
aspects
of
people’s
experiences
and
make
them
visible
at
loca*ons

Window
into
experiences?
e.g.,
Photos
of
sunset,
tagged
with
sunset,
taken
at
*me
t
indicates
X
was
appreciating something
4 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

5. What is an Experience?!
Memory
View:
Result
of
an
experience
process.
Experience
memory
consists
of
one
or
more
actors,
spa*otemporal,
social,
cogni*ve,
and
affec*ve
aspects
based on idea of episodic memory "
(Tulving, 1993)

“Looking
at
these
photos
reminds
me
of
the
good
*mes
we
had
at
the
last
ICMI
conference
in
Beijing”

Post-­‐hoc
representa*on

Used
as
a
framework
to
understand
the
contextual
factors
in
LMM

Emphasis
on
message
produc*on
5 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

6. Questions!
1. What
contextual
factors
are
involved
in
using
LMM
systems?
2. Can
these
factors
inform
the
study
and
design
of
future
LMM
systems?
Exploratory
study!
6 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

7. Related Work!
GeoMedia
(Papliatseyeu
&
Mayora,
2008):
-­‐
permits
aaaching
mul*media
messages
(as
images,
audio
or
video)
to
loca*ons
-­‐
no
user
study
GeoNotes
(Persson
&
Fagerberg,
2002)
and
E-­‐graffi*
(Burrell
&
Gay,
2002):
-­‐
loca*on-­‐aware
systems
that
allow
users
to
leave
textual
messages
such
as
reminders
or
post-­‐it
notes
at
loca*ons
-­‐
extensively
studied
in
real-­‐world
usage
contexts
-­‐
focus
on
user
reac*ons
to
designed
systems
-­‐
tools
used
as
loca*on-­‐based
e-­‐mail
We
were
interested
in:
-­‐
whether
mul*media
messages
can
help
understand
experiences
7 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

8. Prototype!
LMM
prototype
on
the
Android
mobile
device
Allows
annota*on
of
loca*ons
with
mul*media
messages
(drawings,
text,
photographs)
Tool
to
acquaint
study
par*cipants
with
LMM
concept
8 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

9. Prototype: Message Creation!
Create
-­‐
Touch-­‐based
drawing
(doodles)
-­‐
Wri*ng
text
Snap
-­‐
Photographs
Engage at t0" Create at t1"
9 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

10. Prototype: Message Viewing!
Explore
When
user
is
at
the
right
posi*on
and
orienta*on,
s/he
can
view
the
message
Message
appears
as
an
Augmented
Reality
overlay
on
the
camera
view
View at t2"
10 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

11. Diary Study !
Longitudinal
(~1
week)
mul*modal
diary
method
(Amin, 2009)"
11 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

12. Categorization Task!

Needed
for
inter-­‐coder
reliability
(n=6)
Messages
categorized
according
to
both:
Domain:
What
is
the
message
about?
(e.g.,
entertainment,
architecture)
Task:
What
was
the
purpose
of
the
message?
(e.g.,
apprecia*on,
cri*cism)
12 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

13. Diary Study: Participants !
8
subjects
(6
m,
2
f)
aged
between
13-­‐27
(M=
23;
SD=
4.4)
13 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

14. Diary Study: Materials !

8
custom-­‐designed
paper
diaries,
where
each
had
a
template
asking:
1.
Ques*ons
about
the
message
2.
Ques*ons
about
the
subject
and
her
context

Post-­‐study
interview
ques*ons
14 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

15. Diary Study: Materials(1) !
1. Ques*ons
about
the
expression:
-­‐
Date
-­‐
Time
-­‐
Message
format
(drawing,
text,
photo,
-­‐
video,
audio
recording,
other)
-­‐
Title
of
message
-­‐
Public
or
private
15 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

16. Diary Study: Materials(2) !
2. Ques*ons
about
the
subject
and
her
context:
Spa*otemporal
(Q1,
Q4)
Social
(Q5)
Affec*ve
(Q3)
Cogni*ve
aspects
(Q2,
Q7,
Q8,
Q9,
Q10,
Q11)
16 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

17. Diary Study: Materials(3) !
Interview
Ques*ons:
-­‐
Difficulty
filling
in
the
diary
-­‐
Media
preference
-­‐
Awareness
and
experience
of
past
week
-­‐
Desire
to
view
and
write
message
metadata
-­‐
Willingness
to
use
a
future
mobile
applica*on
17 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

18. Procedure!
Info
brochure
LMM
Prototype
Demo
Diary
(2
mul*media
messages
per
day
for
1
week)
Post-­‐study
Interview
18 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

19. Results!
Total
message
count:
110
Vo*ng
“winner-­‐takes-­‐all”
procedure
used
to
analyze
Categoriza*on
Task
results
(n=6)
If
equal
number
of
responses
to
two
dis*nct
categories
then
expression
classified
under
both
Categoriza*on
Task
results
directly
assimilated
into
diary
study
results
19 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

20. Media Choice!
“In
the
beginning,
it
was
photos,
and
during
the
week,
because
it
wasn't
that
interes*ng,
I
used
more
text"
Text
(symbolic)
can
be
used
to
express
something
beyond
quali*es
of
the
loca*on
itself
Songs
act
as
surrogates
for
the
memory
of
a
place
20

21. Domain Ratings (N=6) for 110 Messages!
Total = 114%"
21

22. Task Ratings (N=6) for 110 Messages!
Total = 113%"
22

23. Spatiotemporal Aspects!
Most
Urban
expressions
fell
into
Aesthe*cs
(63%)
and
Apprecia*on
(49%)

*ght
correspondence
between
being
outdoors
and
aesthe*c
apprecia*on
Controlling
for
Public
Place,
many
messages
were
about
Ac*vity
Repor*ng
(39%)

microblogging
behavior
(e.g.,
Twiaer
feeds)
23

24. Social Aspects!
Difference
between
public
and
private
messages
and
messages
made
alone
or
in
the
company
of
others

Messages
made
alone
were
also
made
public
(76%)
24

25. (Russell, 1980)
Affective Aspects!
Tendency
between
being
alone
and
nega*vely
valenced
mood
(60%)

Cathar*c
outlet
typical
of
web
2.0
social
behavior?
25

26. Cognitive Aspects!
Causal
rela*on
between
prior
ac*vity
&
message
crea*on:
35%
Interview:
Awareness
of
daily
environment?
All
reported
planning
behavior,
but
not
if
the
tool
is
embedded
in
daily
life
26 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

27. Viewing Metadata(1)!
Context
metadata
(mood,
companions,
event)
desired:
6
subjects
Standard
metadata
(name,
date,
*me)
desired:
2
subjects
"Not
at
first
sight,
that
would
ruin
my
personal
view
of
their
message.
But
it
should
be
available
if
wanted...why
the
message
was
made,
what
did
the
person
want
to
express."
27 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

28. Viewing Metadata(2)!
No*fica*on:
Filter
by
context:
7
subjects
Query:
1
subject
“If
I'm
walking,
then
I'd
like
to
search
myself,
but
if
I'm
biking,
I'd
like
noEficaEon
of
what
there
is”

No*fica*on
should
depend
on
the
situa*on
subjects
are
in
to
avoid
interrup*on

Personaliza*on
best
considered
itself
context-­‐dependent?
28 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

29. Limitations!
2
expressions
per
day
for
one
week
is
unnatural
Cogni*ve
effort
Availability
of
media
capture
devices
Instruc*ons
insufficient
29 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

30. Implications!
 Predominant
domain
(Aesthe*cs,
Entertainment)
and
task
(Apprecia*on,
Ac*vity-­‐repor*ng)
categories
in
experience
capture
behavior
 Applica*on
personaliza*on
(‘when’)
should
depend
on
and
adapt
to
the
user's
context
(‘what’)
 Capturing
experiences
(memory
view)
vs.
the
experience
of
capture
(process/interac*on
view)
30 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions!

31. Conclusions!
 Paper
diary
a
useful
low-­‐fidelity
mechanism
for
understanding
contextual
factors
in
LMM
 Experience
memory
framework
not
very
useful
and
effortul
to
analyze

clear
behavioral
paaerns
emerge
with
larger
datasets?
31 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions!

32. Current / Future Work!
 Path
planning
&
POI
recommenda*on
service
based
on
social
media
content
(e.g.,
FlickR
photos)

Experience
as
Memory
View
 Mul*modal
&
Crossmodal
feedback
in
place
and
path
recommenders
to
enhance
city
explora*on

Experience
as
Interac*on
View
32 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions!

33. Thanks
for
listening.
Ques:ons?
(VISIT
ME
AT
MY
DOCTORAL
SPOTLIGHT
POSTER)
33 Website:
http://staff.science.uva.nl/~elali/

34. References!
A.
Papliatseyeu
and
O.
Mayora
Ibarra.
Nailing
the
reality
with
GeoMedia:
loca*on-­‐
aware
mul*media
tags.
In
Proceedings
of
MobiMedia’08
Conference,
Oulu,
Finland,
July
2008.
ACM.
A.
Amin,
S.
Townsend,
J.
Ossenbruggen,
and
L.
Hardman.
Fancy
a
drink
in
canary
wharf?:
A
user
study
on
loca*on-­‐based
mobile
search.
In
INTERACT
’09:
Proceedings
of
the
12th
IFIP
TC
13
Interna*onal
Conference
on
Human-­‐
Computer
Interac*on,
pages
736–749.
Springer-­‐Verlag,
2009.
E.
Tulving.
What
is
episodic
memory?
Current
Direc*ons
in
Psychological
Science,
pages
67-­‐70,
1993.
J.
Burrell
and
G.
K.
Gay.
E-­‐graffi*:
Evalua*ng
real-­‐world
use
of
a
context-­‐aware
system.
Interac*ng
with
Computers,
14(4):301–312,
2002.
K.
Dey.
Understanding
and
using
context.
Personal
&
Ubiquitous
Compu*ng,
5(1):4–7,
2001.
M.
Weiser.
The
computer
for
the
21st
century.
Human-­‐computer
interac*on:
toward
the
year
2000,
pages
933–940,
1995.
P.
Persson
and
P.
Fagerberg.
Geonotes:
a
real-­‐use
study
of
a
public
loca*on-­‐aware
community
system.
Technical
Report,
2002.
Russell
J.
A.
(1980).
A
circumplex
model
of
affect.
Journal
of
Personality
and
Social
Psychology,
39:1161–1178.
34