MAGIC: A Motion Gesture Design Tool Daniel Ashbrook, Georgia Tech and Nokia Research Center Hollywood Thad Starner, Georgia Tech http://research.nokia.com/files/2010-Ashbrook-CHI10-MAGIC.pdf Presented at the 28th Annual ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) Abstract: Devices capable of gestural interaction through motion sensing are increasingly becoming available to consumers; however, motion gesture control has yet to appear outside of game consoles. Interaction designers are frequently not expert in pattern recognition, which may be one reason for this lack of availability. Another issue is how to effectively test gestures to ensure that they are not unintentionally activated by a user’s normal movements during everyday usage. We present MAGIC, a gesture design tool that addresses both of these issues, and detail the results of an evaluation.

1. MAGIC:
A Motion Gesture Design Tool
Daniel Ashbrook Thad Starner
Nokia Research Center Hollywood Georgia Tech
Georgia Tech

2. gesture
2

3. gesture design issues
3

4. gesture design issues
support for
false positives
non-experts
3

5. Allow non-expert use…
4

6. Allow non-expert use…
…but support experts
4

7. B
Iterative design
5

8. B
Iterative design
Support extensive testing
5

9. B
Iterative design Retrospection
Support extensive testing
5

10. goal:
make gesture design easier
6

11. magic
(multiple action gesture interface creation tool)
7

12. sensing & recognition
sensor:
wrist-mounted
accelerometer
8

13. sensing & recognition
sensor:
wrist-mounted
accelerometer
microphone
“skinput”
gyroscope
muscle input
…
8

14. sensing & recognition
sensor: gesture
wrist-mounted recognition:
accelerometer DTW
microphone
“skinput”
gyroscope
muscle input
…
8

15. sensing & recognition
sensor: gesture
wrist-mounted recognition:
accelerometer DTW
microphone HMMs
“skinput” SVMs
gyroscope neural networks
muscle input $1 recognizer
… …
8

16. gesture design issues
support for
false positives
non-experts
9

17. gesture design issues
support for
false positives
non-experts
magic
user interface
9

18. Gestures to define:
• Play/Pause • Volume Up 10%
• Shuffle • Volume Down 10%
• Next Track • Next Playlist
• Previous Track • Previous Playlist
10

19. gesture classes and
examples
11

20. gesture classes and
cut
examples
11

21. gesture classes and
cut
examples
11

22. gesture classes and
cut
examples
11

23. magic usage stages
Gesture
Creation
False Positive
Gesture Testing
Testing
12

24. magic usage stages
Gesture
Creation
False Positive
Gesture Testing
Testing
12

25. gesture creation
13

26. intra-gesture consistency

27. intra-gesture consistency

28. inter-gesture differentiability

29. inter-gesture differentiability

30. 16

31. Actual Guess Actual Guess
wrong
} guesses
73% goodness 100% goodness
16

32. example
comparison
graph
17

33. example 1
comparison
graph 2
3
4
5
17

34. gesture class comparison
18

35. gesture class comparison
18

36. gesture class comparison
match box
18

37. magic usage stages
Gesture
Creation
False Positive
Gesture Testing
Testing
19

38. magic usage stages
Gesture
Creation
False Positive
Gesture Testing
Testing
19

39. gesture design issues
support for
false positives
non-experts
20

40. gesture design issues
support for
false positives
non-experts
20

41. define gesture:
delete email
21

42. false positives
define gesture: accidentally
delete email delete email
21

43. everyday gesture library
EGL
22

44. everyday gesture library
EGL
22

45. everyday gesture library
? ?
?
EGL ?
? ?
?
23

46. everyday gesture library
OK
? X
?
OK
?
EGL OK
?
X
? OK
?
X
?
23

47. EGL data collection
accelerometer contextual video
24

48. magic usage stages
Gesture
Creation
False Positive
Gesture Testing
Testing
25

49. magic usage stages
Gesture
Creation
False Positive
Gesture Testing
Testing
25

50. false positive testing
26

51. magic evaluation
experiment set up
27

52. Gestures to define:
• Play/Pause • Volume Up 10%
• Shuffle • Volume Down 10%
• Next Track • Next Playlist
• Previous Track • Previous Playlist
28

53. Gesture criteria:
• The gesture must reliably activate the desired function.
quantitative • Performing the gesture must not activate other functions.
• The functionality associated with a gesture must not be
activated by a user’s everyday movements.
• The gesture should be easy to remember.
qualitative • The gesture should be easy to perform.
• The gesture should be socially acceptable.
29

54. participants
Num Num UCD UI design Pattern rec
Condition participants Age
female experience experience experience
no EGL 7 29.0 2 5.9 6.7 4.0
EGL 7 31.6 2 6.6 5.6 3.0
(1-9) (1-9) (1-9)
recruited participants with UI and/or UCD experience
30

55. collected EGLs
EGL age gender occupation hours collected
1 32 M PhD CS 19:18
2 26 M MS CS 9:33
3 27 F Librarian 7:40
4 28 M Civil Engineer 6:53
5 30 F IT professional 4:50
6 37 M IT professional 4:08
7 28 F Writer 3:57
8 29 F Project manager 2:09
Total 29.6 4m/4f 58:28
31

56. collected EGLs
EGL age genderActivities
EGL occupation hours collected
1 32 • PhD CS
M attending conference 19:18
2 26 • MS CS
M brewing beer 9:33
3 27
• knitting
F vacationing at beach
Librarian 7:40
•
4 28 M vacationing at mountains
• Civil Engineer 6:53
5 30 • IT professional
F working at computer 4:50
6 37 • hiking
M IT professional 4:08
• cooking
7 28 F home repair
Food writer 3:57
•
8 29 F attending manager
• Project work meetings 2:09
•
Total 29.6 4m/4fmaking cheese 58:28
32

57. in-magic EGL
19:18
EGL 1
33

58. in-magic EGL
5:19 13:59
given to testing
users
EGL 1
33

59. magic evaluation
quantitative results
34

60. user performance
false positives
35

61. user performance
false positives
12
11 1
10 2
9 3
8 4
7 5
6
EGL: 1.9/gesture/hour
35

62. user performance
false positives
12 12
11 1 11 1
10 2 10 2
9 3 9 3
8 4 8 4
7 5 7 5
6 6
EGL: 1.9/gesture/hour noEGL: 52.1/gesture/hour
35

63. user performance
false positives
12 12
11 1 11 1
10 2 10 2
9
User pattern recognition experience has
9
3 3
no effect on performance!
8 4 8 4
7 5 7 5
6 6
EGL: 1.9/gesture/hour noEGL: 52.1/gesture/hour
35

64. user performance
gesture goodness
• Gesture goodness:
μ = 86% / σ = 24%
• 7 with goodness >= 97%
Actual Guess Actual Guess
wrong
} guesses
73% goodness 100% goodness
36

65. user performance
about 2h to
complete task
37

66. user performance
about 2h to
complete task
mostly linear
progression:
create, test, EGL
37

67. user performance
about 2h to
complete task
mostly linear mostly linear progression:
progression: add class, make examples,
create, test, EGL troubleshoot
37

68. magic evaluation
qualitative results
38

69. feedback
general
“This is really hard!” (but…) “I really like the
software and I really like the interface”
“Gesture creation was easy”
“It’s pretty awesome… it’s really fun”
“I liked the task” … “I think it’s really interesting”
39

70. feedback
egl
“I just kinda feared the EGL”
(regarding video) “I
didn't care why I was hitting the
[EGL]; I can't change what's in there”
“It was really useful to compare both hat-mounted
videos” (EGL video and gesture video)
The EGL tab is “very important”
40

71. feedback
visualizations
Graphs were “something a little bit complicated
for me”
“It takes a while to build up an intuition about what
[the graphs] mean”
“I thought I needed to understand [the graphs]
more”
“[The graphs] give a good reflection of what I’ve
done”
41

72. retrospection
previous playlist A previous playlist B
low goodness high goodness
“retrospective realization”
42

73. retrospection
previous playlist A previous playlist B
low goodness high goodness
“retrospective realization”
42

74. retrospection
previous playlist A previous playlist B
low goodness high goodness
“retrospective realization”
42

75. future work
• improve speed, responsiveness
• easier to understand visualizations
• test out other sensors, algorithms
• gesture & example annotations
43

76. see paper for:
• related work
• users’ gesture strategies
• statistics!

77. Thank you!
questions?

78. ISWC'10
IEEE International Symposium on Wearable
Connected Smartware - Wearable
Computers
Oct. 10th - 13th, 2010, COEX, Seoul, South Korea
Topics include
On-body, mobile Systems, usability, HCI and human factors in wearable
computing, applications of wearable systems
Submission deadline
21th, May, 2010
Papers, notes, posters, & design contest
30th, May, 2010
Videos, late breaking results, demonstrations, Ph.D.
forum, & tutorials
http://www.iswc.net

79. user performance
gesture goodness
precision · recall F-measure, or
goodness = 2 · harmonic mean of
precision + recall precision and recall
47

80. user performance
gesture goodness
precision · recall F-measure, or
goodness = 2 · harmonic mean of
precision + recall precision and recall
89% 95%
goodness goodness
p=80%, r = 100% p=100%, r = 90% 47

81. user performance
gesture goodness
precision · recall F-measure, or
goodness = 2 · harmonic mean of
precision + recall precision and recall
89% 95% 82%
goodness goodness goodness
p=80%, r = 100% p=100%, r = 90% 47 p=78%, r = 88%

82. user performance
gesture goodness
precision · recall F-measure, or
goodness = 2 · harmonic mean of
precision + recall precision and recall
89% 95% 82% 100%
goodness goodness goodness goodness
p=80%, r = 100% p=100%, r = 90% 47 p=78%, r = 88% p=100%, r = 100%

83. gesture design tool desiderata
48

84. gesture design tool desiderata
Allow non-expert use
48

85. gesture design tool desiderata
Allow non-expert use
(1-9)
Num UI design Pattern rec
Condition participants Age Num female UCD experience experience experience
noEGL 7 29.0 2 5.9 6.7 4.0
EGL 7 31.7 2
48 6.3 5.1 3.1

86. gesture design tool desiderata
Allow non-expert use
87%
mean goodness
(1-9)
Num UI design Pattern rec
Condition participants Age Num female UCD experience experience experience
noEGL 7 29.0 2 5.9 6.7 4.0
EGL 7 31.7 2
48 6.3 5.1 3.1

87. gesture design tool desiderata
Allow expert use
49

88. gesture design tool desiderata
B
Iterative design
50

89. gesture design tool desiderata
Retrospection
51

90. gesture design tool desiderata
Support further testing
52

91. gesture design tool desiderata
Support further testing
52

92. EGL limitations
• not guaranteed to span space of everyday life
• EGL is defined by
(sensor, situation, target user population)
53

93. support for non-experts
54