1. Engaging Spect-actors
with Multimodal Digital Puppetry
Céline Coutrix (presenter),
Giulio Jacucci, Anna Spagnolli, Lingyi Ma, Matti Helin,
Gabriela Richard, Lorenza Parisi, Stefano Roveda, Prayag Narula

2. Introduction
• (Digital) Puppetry
1

3. Introduction
• Field study
- Describe an example of how a computer
can be used as a medium for engagement
- Understand why would this be a promising
field for HCI applications
2

4. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
3

5. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
4

6. The system: Euclide
audience, in front of puppet
pupeteer
5

7. The system: Euclide
Euclide
system
6

8. The system: Euclide
• Has been successfully used for engagement
in the Science museum in Naples, Italy
• Why? How could it be improved? What can
we learn for future puppetry systems?
7

9. The system: Euclide
• Puppeteer
• Audience
8

10. The system: Puppeteer
puppet on distant screen
9
pupeteer

11. The system: Puppeteer
• Puppeteer
- Voice
- Face
- Body
· Global movement
· Local movement
- Scenery
10

12. The system: Puppeteer
• Voice
11

13. The system: Puppeteer
• Face
12

14. The system: Puppeteer
• Face
13

15. The system: Puppeteer
• Face
14

16. The system: Puppeteer
• Face
15

17. The system: Puppeteer
• Face
16

18. The system: Puppeteer
• Face
17

19. The system: Puppeteer
• Face
18

20. The system: Puppeteer
• Face
19

21. The system: Puppeteer
• Body: Global movement
20

22. The system: Puppeteer
• Body: Global movement
21

23. The system: Puppeteer
• Body: Global movement
22

24. The system: Puppeteer
• Body: Local movement
23

25. The system: Puppeteer
• Body: Local movement
24

26. The system: Puppeteer
• Body: Local movement
25

27. The system: Puppeteer
• Body
26

28. The system: Puppeteer
• Scenery
transparency
27

29. The system: Puppeteer
• Scenery
orientation
28

30. The system: Puppeteer
• Scenery
Zoom+Position
29

31. The system: Euclide
• Puppeteer
• Audience
30

32. The system: Euclide
distant
pupeteer
31
audience, in front of puppet

33. The system: Euclide
• Compromise between
- Museum’s requirements for cheap, robust,
generic and flexible hardware
- Puppeteers’ requirements for large number
of functionalities
- Usability
· e.g. mouth control moved from index to thumb
to avoid tiredness
32

34. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
33

35. Data Collection
• Video+audio recordings
- Over several months
- Four viewpoints
- Three different pupeteers
• Interview of head pupeteer
34

36. Data Analysis
• Constant comparison analysis
• Top-down and Bottom-up approach
- Findings originates from both the data and
our research questions
• Coding scheme built through three
steps: open, axial, selective coding
• Clips selected together, then divided
between coders
• Discussion and revision of coding until
agreements > 0.6 35

37. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
36

38. Findings: Multimodal Puppetry
• Most used features are controlled
through more accessible devices, due
to design
37

39. Findings: Multimodal Puppetry
face
body
scenery
• Mostly animation of face,
• second comes body,
• last comes scenery 38

40. Findings: Multimodal Puppetry
Simultaneous
use of modalities
Single modality
• Important multimodal use
39

41. Findings: Multimodal Puppetry
1. Speech synchronization’s problem
2. Speech+lips
3. Lips
40

42. Findings: Multimodal Puppetry
Speech durations
Silence durations
• Lot of very short speeches and silences
in order to maintain engagement
• Puppet active even when silent
41

43. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
42

44. Findings: Emergence of
Performative Structure
• Engaging the Audience Throughout the
Sessions: Evidences
- People in the audience:
· Don’t talk to each other 99% of the time
· Don’t pay attention to the outside 98% of the time
· Talk to the puppet 65% of the time
· Pleasure and Arousal annotated as positive all the
time
· Number increase, then decrease
43

45. Findings: Emergence of
Performative Structure
Number and length of sessions (mins)
10
8
6
4
2
0
< 1 min 1 min < < 2 min 2 min < < 9 min > 9 min
• Rather short, but:
- Sessions interrupted by teachers
- Group of children continue interacting
44
from the next puppet station

46. Findings: Emergence of
Performative Structure
• Structure of sessions
- Approach: enter the interaction area;
observe
- Testing: Try to interact to find out which
actions have an effect
- Playing: Interact in an aware, active,
involved way; climax
- Ending: attention diverted; leave
45

47. Findings: Emergence of
Performative Structure
• Structure of sessions
- Approach, Testing, Playing, Ending
- Long playing phases, with scenery animations
46

48. Findings: Emergence of
Performative Structure
47

49. Findings: Emergence of
Performative Structure
• the audience starts interacting happily but calm,
• then gets excited,
• finishes the interaction happily and calm again
48

50. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
49

51. Findings: Construction of
Engagement
• The Puppet’s Multimodal Resources:
Effects (face, body and scenery
animations), like natural gestures, in
relation to speech
- Pure special effects
- Non-verbal turns
- Verbal accompaniements
- Virtual gesturing
50

52. Findings: Construction of
Engagement
• Pure special effects
- Non verbal ressources are predominant
modality for interacting with the audience
- Used to attract visitors that cannot be reached
by speech (babies, far away visitors)
51

53. Findings: Construction of
Engagement
• Non-verbal turns: Contribution in an
interaction that includes speech, e.g.
- wearing accessory like glasses on special
request from the audience and subsequently
commenting on it
- Explaining a fact by displaying a picture of it
52

54. Findings: Construction of
Engagement
• Verbal accompaniements: Non-verbal
actions are accompagnied by speech, e.g.
- Acrobatic movement with onomatopeic sound
→ Communication relies on the nonverbal
resource, which dictates its structure and
meaning
53

55. Findings: Construction of
Engagement
• Verbal accompaniements: Example
Puppet: Ho una molla
I’ve got a spring
Mi serve per saltare
I need it to jump
⎡ Dong:: dong:: ⎤
Dong dong
⎣((Starts jumping))⎦
54

56. Findings: Construction of
Engagement
• Virtual gesturing: priority is given to
speech; visual features are used in
synch with speech, following its
structure and duration, e.g.
- Head movement, like pointing gesture
55

57. Findings: Construction of
Engagement
• Virtual gesturing: Example
Visitor: Dove sei?
Where are you?
Puppet: Sono⎡::((pause))⎤
I’m
⎣((Looking down))⎦
⎡qui dentro,⎤
inside here
⎣((Looking around))⎦
non mi vedi?
can’t you see me 56

58. Findings: Construction of
Engagement
• Multimodal interaction for puppeteer
is essential
- Helps avoiding the low status of a
conversational agent, e.g. verbal abuse
57

59. Findings: Construction of
Engagement
• Response to verbal abuse: Example
Child: (Scemo)
Dumb
Puppet: Vabbeh adesso basta.
All right; let’s stop this
Child: ((turns back, surprised))
Puppet: Ogni volta che scemo, ogni volta che scemo.
All the times dumb, all the times dumb.
E tu invece come sei?
And what about you instead, what are you?
Child: ((goes away))
58

60. Findings: Construction of
Engagement
• Talk = Main ressource,
Multimodal animation = instrument
• Combination attracts visitors effectively
59

61. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
60

62. Conclusion
• Why is digital puppetry a promising application
field for HCI?
- On the contrary to a fully computer-driven system
· Engagement stems from combination of multimodal
ressources by puppeteer
· Users from the audience inspire the narrative by the
puppeteer and feel like they have an active role
61

63. Conclusion
• Why is digital puppetry a promising application field
for HCI?
- Performative Structures for Brief Interactions repetitive
element of an entertainment interface could be further
taken into account
- Multimodal Use of the System by the Puppeteer
possibility to introduce advanced interface techniques
· Easing lip/speech sync,
· Mapping expressive hand gestures to puppet expressions,
· Etc.
62

64. Engaging Spect-actors
with Multimodal Digital Puppetry