Slides for the talk I gave at ICMI 2012, held in Santa Monica, CA, USA. The full paper reference is: El Ali, A., Kildal, J. & Lantz, V. (2012). Fishing or a Z?: Investigating the Effects of Error on Mimetic and Alphabet Device-based Gesture Interaction. In Proceedings of the 14th international conference on Multimodal Interaction (ICMI '12), 2012, Santa Monica, California.

1. Fishing or a Z?: Investigating the Effects of Error on
Mimetic and Alphabet Device-based Gesture Interaction"
Oct.
23,
2012
Abdallah
‘Abdo’
El
Ali
Johan
Kildal
Vuokko
Lantz
h6p://staff.science.uva.nl/~elali/

2. 2

3. Outline"
I. Introduc5on
II. Methods
III. Results
IV. Discussion
V. Future
Work
3

4. Introduction
4

5. Introduction"
 Device-­‐based
gestures:
 Gesturing
by
moving
a
smartphone
device
in
3D
space
 Research
seGngs,
home
environments,
everyday
mobile
interac5on
 Alterna5ve
to
when
users
are
encumbered
(e.g.,
manual
mul5tasking)
 Natural
 Promising
alterna5ve
to
mobile
touchscreen/keyboard
input
under
situa5onal
impairments
5

6. Motivation"
 But
errors
are
an
inevitable
part
of
interac5on
with
technology…
 Many
gesture
classes
are
available
(e.g.,
iconic,
symbolic,
deic5c)
for
use
in
smartphones,
but
which
have
minimum
user
frustra5on
when
recogni5on
errors
occur?
 We
inves5gate
user
error
tolerance
for
two
iconic
gesture
sets
used
in
HCI:
mime5c
and
alphabet
gestures
6

7. Related Work"
 Gesture-­‐based
Interac5on
 Gesture-­‐Task
mapping
(Khnel
et
al.,
2011;
Ruiz
et
al.,
2011)
 Social
acceptability
(Rico
&
Brewster,
2010)
 Gesture
Taxonomies:
Deic5c,
symbolic,
physical,
mime5c/pantomimic,
abstract
(‘
m/
’)
(Rime
&
Schiaratura,
1991;
…)
 Recogni5on
Errors
 Speech:
Repeat
(Suhm
et
al.,
2001)
and
hyperar5culate
(Oviad
et
al.,
1998)
 Mul5modal:
Modality-­‐switching
(“spiral
depth”
of
6)
(Oviad
&
van
Gent,
1996)
 Touch-­‐less
Vision-­‐based
Gesture
Recogni5on:
How
many
errors
before
switching
to
keyboard
input?
40%
user
error
tolerance
before
modality
switching
(Karam
&
Schraefel,
2006)
 Lidle
work
on
which
gesture
sets
are
most
robust
to
errors
during
gesture-­‐based
interac5on!
7

8. Iconic Gestures"
 Mimetic / Pantomimic Gestures: natural, familiar, easy to learn, varied by
activities
 e.g., Fishing, calling, …
 Alphabet Gestures: familiar, easy to learn, varied by stroke
 e.g., letter “C”, letter “S”, …
8

9. Research Question"
What
are
the
effects
of
unrecognized
gestures
on
user
experience,
and
what
are
the
differences
between
mime5c
and
alphabet
gestures
(under
varying
error
rates:
0-­‐20%,
20-­‐40%,
40-­‐60%)?
Hypotheses:

Mime5c
gestures
→
users
less
familiar
with
ideal
shape
→
more
gesture
varia5on
under
high
error
rates
→
but
lower
subjec5ve
workload
due
to
higher
degrees
of
freedom

Alphabet
gestures
→
users
more
familiar
with
ideal
shape
→
more
rigid
gestures
under
increasing
error
rates
→
but
higher
subjec5ve
workload
due
to
lower
degrees
of
freedom
9

10. Methods
10

11. Mimetic Gesture Design"
11

12. Alphabet Gesture Design"
12

13. Study Design"
 Qualita5ve
Study;
Automated
Wizard-­‐of-­‐Oz
method
(Fabbrizio
et
al.,
2005)
 24
subjects
(16
male,
8
female)
aged
between
22-­‐41
(M=
29.6,
SD=
4.5)
 Mixed
between-­‐
and
within
subject
factorial
design:
2
(gesture
type:
mime5c
vs.
alphabet)
x
3
(error
rate:
low
vs.
med
vs.
high)
 Experiment
in
Presenta5on®,
Wii
Remote®
interac5on
using
GlovePie™
 Random
error
distribu5on
across
trials
(20
prac5ce,
180
test)
 Tutorial
&
videos
given
of
how
to
'properly'
perform
each
gesture
 Data
collected:
1. Modified
NASA-­‐TLX
workload
[0-­‐20
range]
ques5onnaire
data
(Hart
&
Wickens,
1990;
Brewster,
1994)
2. Experiment
logs
3. Video
recordings
of
subjects’
gesture
interac5on
13 4. Post-­‐experiment
interviews

14. !" !#
!"#$%$&'
()*$+,
8'()*$+'
89':".#),'2&"'()*$+'
!" !# !$ !% !& !'
;*<'=""*"
-./&%$
0,
'1)23#4&5'
$*67.%*6 -&7'=""*"
>.?3'=""*"'
@'()*$+,'2&"'=""*"'$*67.%*6'
A&""*"'$*67.%*6'B'4)*$+'*"7&"'$*C65&"4#)#6$&7'D'"#67*/.E#%*6F
14

15. Setup & Procedure"
≈1 hr
Reward
Exit Interview
x3
NASA-TLX
Questionnaire
Automated
Wizard-of-Oz Test [...]
Block
Practice
Block
Instructions Perform
& Tutorial Fishing
Gesture
Personal
Information
Form
Perform
Fishing
Experiment Gesture
Session
Fishing
Gesture
t Test
Block
15

16. #: ____
MENTAL DEMAND
Low High
PHYSICAL DEMAND
Low High
TIME PRESSURE
NASA-TLX
Low High
(Hart & Wickens, 1990)
EFFORT EXPENDED
Low High
PERFORMANCE LEVEL ACHIEVED
Poor Good
FRUSTRATION EXPERIENCED
Low High
 ANNOYANCE EXPERIENCED
Low High
(Brewster, 1994)
OVERALL PREFERENCE RATING
16 Low High

17. Results
17

18. Workload"
!"#$%"&'($)%*+$)
! ! !! ! !
! !! !! !!
%&
&
0$
,"
#
"
:
,%
$+/
#,
/%
../
$
"#
%#
*+
0%
0"
9&
-
()
!
01
/0
$
."
3*
'
"8
./
0"
20
"0
'
+7
'
,$
6"
!!
35
4
!
18 ;/<=-00/0=>%$" !":+31=-00/0=>%$" ?+@(=-00/0=>%$"

19. Workload"
!"#$%&'()*'+(,-'+
! !! ! !! !! !! ! !!
! !! ! !! !!
%&
%&
0$
"
#
"
:
,
$+/
#,
/%
../
$
+,
"#
%#
*
$0%
0"
9&
-
()
!
1
/0
."
3*
0
'
"8
./
0"
20
"0
'
$+7
'
!!
,
6"
35
!
4
;/<=-00/0=>%$" !":+31=-00/0=>%$" ?+@(=-00/0=>%$"
19

20. Workload"
 Mime5c
gestures
are
beder
tolerated
up
to
error
rates
of
40%
(cf.,
Karam
&
Schraefel,
2006),
compared
with
error
rates
of
up
to
only
20%
for
alphabet
gestures
 From
a
usability
perspec5ve,
mime5c
gestures
more
robust
to
recogni5on
failures
than
alphabet
gestures
20

21. Observations"
 Mime5c
gestures
evolve
into
real-­‐world
counterparts
under
error,
alphabet
gestures
tend
to
become
more
rigid
and
well
structured
 Canonical
Varia5ons
via
posi5ve
reinforcement:
Survival
of
the
fidest
gesture
varia5ons
 E.g.,
S12
exhibited
real-­‐world
varia5ons
on
both
the
Fishing
and
Trashing
gestures
 Varia5ons
develop
as
low
as
spiral
depth
of
2
(i.e.,
min.
2
recogni5on
errors)
21

22. User Feedback"
 Perceived
Canonical
Varia5ons
 S9:
“ The
shaking,
that
was
the
hardest
one
because
you
couldn’t
just
shake
freely
[gestures
in
hand],
it
had
to
be
more
precise
shaking
[swing
to
the
leA,
swing
to
the
right]
so
not
just
any
sort
of
shaking
[shakes
hand
in
many
dimensions]”
 Cultural
and
Individual
differences
 S10:
“For
the
glass
filling,
there
are
many
ways
to
do
it.
SomeFmes
very
fast,
someFmes
slow
like
beer.”
 Perceived
Performance
 Ad
hoc
explana5ons
(e.g.,
fa5gue)
given
why
there
were
more
errors
in
some
blocks
 S18:
“[Performance]
between
the
first
and
second
blocks
[baseline
and
low
error
rate
condiFons],
it
was
the
same...
10-­‐15%.”
 Social
Acceptability
 Alphabet
gestures
less
socially
acceptable
when
they
fail
 S16:
“When
it
doesn’t
take
your
C,
you
keep
doing
it,
and
it
looks
ridiculous.”
22

23. Discussion
23

24. Limitations"
 Lab-­‐study
 Task
Independence
 Random
error
distribu5on
24

25. Implications for Gesture Recognition"
 Mime5c
gestures
evolve
into
real-­‐world
counterparts
under
error,
alphabet
gestures
tend
to
become
more
rigid
and
well
structured

One
shot
recogni5on
for
mime5c
gestures
important!
 Interes5ng
explana5ons
(e.g.,
canonical
varia5ons)
and
causes
(e.g.,
fa5gue)
given
why
there
were
more
errors
in
some
blocks

Transparency
in
gesture
recogni5on
technology
may
beder
support
users
in
error
handling
strategies
25

26. Implications for Gesture-based Interaction"
 40%
error
tolerance
in
line
with
previous
work
(Karam
&
Schraefel,
2006),
which
shows
usability
of
gesture-­‐based
interac5on
 Mime5c
gestures
overall
have
beder
user
experience,
more
use
cases,
and
thus
more
suitable
for
device-­‐based
gesture
interac5on
(even
under
high
recogni5on
error!)
26

27. Future Work
27

28. Future Work"
 Quan5ta5ve
assessment
of
how
many
errors
precisely
before
canonical
variant?
 Other
classes
of
gestures
(e.g.,
manipula5ve)
 Influence
of
device
form
factor
28

29. Questions
29 h6p://staff.science.uva.nl/~elali/

30. References"
Brewster,
S.
Providing
a
structured
method
for
integra5ng
non-­‐speech
audio
into
human-­‐computer
interfaces.
PhD
thesis,
University
of
York,
1994.
Fabbrizio,
G.
D.,
Tur,
G.,
and
Hakkani-­‐Tur,
D.
Automated
wizard-­‐of-­‐oz
for
spoken
dialogue
systems.
In
Proc.
INTERSPEECH
2005
(2005),
1857–1860.
Karam,
M.,
and
Schraefel,
M.
C.
Inves5ga5ng
user
tolerance
for
errors
in
vision-­‐enabled
gesture-­‐based
interac5ons.
In
Proc.
AVI
’06,
ACM
(NY,
USA,
2006),
225–232.
Khnel,
C.,
Westermann,
T.,
Hemmert,
F.,
Kratz,
S.,
Mller,
A.,
and
Muller,
S.
I’m
home:
Defining
and
evalua5ng
a
gesture
set
for
smart-­‐home
control.
Interna5onal
Journal
of
Human-­‐Computer
Studies
69,
11
(2011),
693
–
704.
Hart,
S.,
and
Wickens,
C.
Manprint:
an
Approach
to
Systems
Integra5on.
Van
Nostrand
Reinhold,
1990,
ch.
Workload
Assessment
and
Predic5on,
257–292.
Oviad,
S.,
Maceachern,
M.,
and
Anne
Levow,
G.
Predic5ng
hyperar5culate
speech
during
human-­‐computer
error
resolu5on.
Speech
Communica5on
24
(1998),
87–110.
17.
Oviad,
S.,
and
Van
Gent,
R.
Error
resolu5on
during
mul5modal
human-­‐computer
interac5on.
In
Proc.
ICSLP
’96,
vol.
1
(oct
1996),
204–207.
Rico,
J.,
and
Brewster,
S.
Usable
gestures
for
mobile
interfaces:
evalua5ng
social
acceptability.
In
Proc.
CHI
’10,
ACM
(NY,
USA,
2010),
887–896.
Rime,
B.,
and
Schiaratura,
L.
Fundamentals
of
Nonverbal
Behavior.
Cambridge
University
Press,
1991,
ch.
Gesture
and
speech,
239–281.
Ruiz,
J.,
Li,
Y.,
and
Lank,
E.
User-­‐defined
mo5on
gestures
for
mobile
interac5on.
In
Proc.
CHI
’11,
ACM
(NY,
USA,
2011),
197–206.
Suhm,
B.,
Myers,
B.,
and
Waibel,
A.
Mul5modal
error
correc5on
for
speech
user
interfaces.
ACM
Trans.
Comput.-­‐
Hum.
Interact.
8
(2001),
60–98.
30