This talk will focus on acoustic rendering, the rendering of scenarios and contexts from the viewpoint of sonic interaction design. Sonic interaction design sits at the crossroads between interaction design, auditory display, acoustics, interactive arts, and ubiquitous computing. The area focuses on the exploration and exploitation of sound to communicate meaning, information, and aesthetical / emotional qualities in interactive contexts. The practise and study of sonic rendering within the interaction loop between users and artefacts or environments for both functional and artistic applications will be discussed. Examples of sonic interventions and of sonic interactive installations are discussed to illustrate how human centric and experiential aspects within the interaction loop. This will show how interaction design techniques and acoustic production are used to create engaging sonic experiences. Case studies covered will include work in museum installations, auditory displays designed for public spaces, and artistic interventions. The talk will show how sonification and sonic interaction design supplements the related areas of sound production and of gestural interfaces. An overview of the techniques and methods will show how this occurs in practise. What is Sonic Interaction Design ? Examples include aesthetic and sonic quality assessment in product and interaction design: workshops and interaction with projects (e.g., SOb, CLOSED, COST SID). Creative exploration of context and opportunities for interactive sound artefacts and for acoustic interventions. Where does Sonic Interaction Design inform Acoustic Rendering ? Room acoustics and Site sound - sound within a space or interface. Context and Scenario, how these inform the rendering (how it is acoustically rendered, what is rendered, how will the rendering be interpreted) Examples includes installations and interventions from various locations including various museums, airports, train stations, and galleries. Related work in products sound design will also be covered.

1. Mixing Interaction, Sonification, Rendering and
Design - The art of creating sonic interactions
Eoin Brazil
EG_IT 22 & 23 October, Verona. Italy

3. Sonification

4. Using Sound to Effectively
Convey Information
Ben Cohen, co-
founder of Ben &
Jerry’s Ice-cream
Company
Talking against the
size and cost of the
nuclear stockpile in
the United States.

5. 203
Cartoonification
evel models: bouncing, breaking, rolling, crumpling, pouring
Maisy drinking with a straw
(From “Happy Birthday,
Maisy” by Lucy Cousins,
Candlewick Press, 1998.)

6. Area-1 Area-2 Area-3 Area-4 Area-5 Area-6 Area-7 Area-8
2.5
0.333 1 0.333 0.692 0.176 0.138 0.354 0.058
0.554 1 0.554 1.232 0.074 0.198 0.358 0.065 2.1
1.028 1 1.028 0.576 0.070 0.077 2.401 0.162
1.7
0.217 1 0.217 0.885 0.259 1.087 0.739 1.005
1.3
0.507 1 0.507 0.862 0.198 0.309 0.578 1.107
0.570 1 0.570 0.866 0.301 0.434 0.526 0.674
0.8
0.554 1 0.554 1.232 0.074 0.198 0.358 0.065
0.4
0.530 1 0.530 0.042 0.852 0.250 0.053 0.185
0.497 1 0.497 1.590 1.365 1.175 1.278 0.101 0
Area-1 Area-2 Area-3 Area-4
0.167 1 0.167 1.261 0.031 0.658 0.306 1.289
Area-5 Area-6 Area-7 Area-8
An Example of Complex datasets - Adding
sound can elaborate streams
Auditory Graphing and temporality of the data

7. Area-1 Area-2 Area-3 Area-4 Area-5 Area-6 Area-7 Area-8
2.5
0.333 1 0.333 0.692 0.176 0.138 0.354 0.058
0.554 1 0.554 1.232 0.074 0.198 0.358 0.065 2.1
1.028 1 1.028 0.576 0.070 0.077 2.401 0.162
1.7
0.217 1 0.217 0.885 0.259 1.087 0.739 1.005
1.3
0.507 1 0.507 0.862 0.198 0.309 0.578 1.107
0.570 1 0.570 0.866 0.301 0.434 0.526 0.674
0.8
0.554 1 0.554 1.232 0.074 0.198 0.358 0.065
0.4
0.530 1 0.530 0.042 0.852 0.250 0.053 0.185
0.497 1 0.497 1.590 1.365 1.175 1.278 0.101 0
Area-1 Area-2 Area-3 Area-4
0.167 1 0.167 1.261 0.031 0.658 0.306 1.289
Area-5 Area-6 Area-7 Area-8
An Example of Complex datasets - Adding
sound can elaborate streams
Auditory Graphing and temporality of the data

8. Sonic Interaction Design
• Sound and interaction are both process oriented
• Interactive systems are complex - sound is able to
convey complex information very quickly
• Interactive systems are “disappearing”, sound can tell
us about their presence and their status
• We have learned to understand complex soundscapes
-> ubiquitous, visualisation, and many areas of
computing may profit from the use of sound

9. SHAPE Exhibition Interactive Sound Object Models
for Exploration & Education
Re-Tracing the Past

10. Solids Liquids Gases
Fracture Impact Friction Fluid flowing Bubble Turbulence Explosion Low Level
Model Model Model Model Model Model Model
Models
Fracturing Crumpling Hitting Rolling Rubbing Braking Flowing Burbing Dripping Whoosing Burning Exploding Popping
Basic Events
and Textures
Pouring Filling Splashing Sucking
Derived
Walking Crushing Bouncing Dropping Breaking Sliding Squeaking
Processes
Rolling / Squeaking Rubbed Vacuum
Footsteps Can Falling
on gravel crushing coin
breaking
wheel
door glass cleaner
Simulation
Paper Struck Glass Bowed
Examples
Struck bar
crumpling string cleaning string
SOb-ifying Impact Models for Hitting,
Rolling, or Crumbling Textures /
the Exhibition
Basic Events

11. Virtual
Touch
Machine
Interacting with one of the four
`mysterious’ objects
Using a multimodal (Audio-
Haptic-Visual) interface to
explore its materiality and size
Real-Time Interaction

12. Virtual
Touch
Machine
Interacting with one of the four
`mysterious’ objects
Using a multimodal (Audio-
Haptic-Visual) interface to
explore its materiality and size
Real-Time Interaction

13. SOb-ifying the Exhibition

14. SOb - Sounding Interactive Sound Object Models
for Exploration & Education
Object Project

15. Museum Installations - Lessons & Considerations
• Enhance • Design activities /
interpretation consultations
✦ observational
• Fun yet interactive,
studies
spark
✦ consult docents /
• Active not passive experts
environment / ✦ understand
interfaces physical/spatial
constraints

16. Soundscapes & J. Tardieu, P. Susini, F. Poisson,
P. Lazareff, and S. McAdams
Train Stations France, 2008

17. Soundscapes & J. Tardieu, P. Susini, F. Poisson,
P. Lazareff, and S. McAdams
Train Stations France, 2008

18. Connecting L. Pellarin, N. Bottcher, J. M.
Olsen, O. Gregersen, S. Serafin,
Strangers at a and M. Guglielmi
Denmark, 2005
Train Station
Bouncing Frequency / Airplane Mirror
Balls Amplitude Synth Triggers

19. Ping Photon Wind Shepard
Drive Tones
Shannon Portal -
Interactive Sounds for
Auditory Display for Communication & Entertainment
Public Spaces

20. Ping Photon Wind Shepard
Drive Tones
Shannon Portal -
Interactive Sounds for
Auditory Display for Communication & Entertainment
Public Spaces

21. Ping Photon Wind Shepard
Drive Tones
Shannon Portal -
Interactive Sounds for
Auditory Display for Communication & Entertainment
Public Spaces

22. Ping Photon Wind Shepard
Drive Tones
Shannon Portal -
Interactive Sounds for
Auditory Display for Communication & Entertainment
Public Spaces

23. Ping Photon Wind Shepard
Drive Tones
Shannon Portal -
Interactive Sounds for
Auditory Display for Communication & Entertainment
Public Spaces

24. Analysis of the Sound & Space
common fate’ and ‘good continuation’ [11].

25. casual users where auditory display was a significant element to
facilitate user interaction. We used an iterative design process,
Brainstorms, Concept from simple onomatopoeicPublic Spacescomplex
Auditory representations, to
Auditory
starting
sound object models in Pure Data. The system was evaluated at
Generation each stage, from the lab to the final public setting. The problems
Linking ubiquitious auditory
addressed covered the representations of left-right, up-down,
displays design to evaluation in
Display Design
and the amount of movement by the user or groups of users. In
public environments
addition to this, it was important that the auditory display would
attract attention when users were within control range of the
system, i.e. an affordance that invited and allowed users to
Process for
Sonic Sketching sound using
discover functionality.
Mood clips (mood boards or
Boards onomatopoeia)
1. INTRODUCTION
In our Shared Worlds project we explored designing
ubiquitous computing for public environments. Part of this
Public Spaces
Figure 1: Shannon Portal installation
exploration was to design, build and evaluate installations in
Video
Bodystorming /
public environments. The work reported in this paper is about
the Shannon Portal, a complex Prototyping
interactive installation in
Roleplay the
Shannon Airport in the West of Ireland. The overall design is
Inteaction
described in [1], while in this paper we focus on the design of
the auditory display element of the installation. One of the
Evaluate
issues that we wanted to explore in the project was the use of
Evaluate large ambient displays and public interaction with such
displays. As the installation at Shannon required an interactive
image gallery to be displayed, we designed a purpose-built
back-projection system. An overhead camera Evaluate
tracked users’
Prototype using PD /
movements in front of the display, allowing the users to move a
virtual Max/MSP or similar
magnifying glass across the image gallery on screen, by
moving their body in front of the display. See Figure 1 and 2.
Over several iterations we developed hardware and software
for the display and explored different mappings of users’ h
= Pitc
actions to movement of the virtual magnifying glass on the ack tion
projection screen. For reasons outside the scope of this paper, it t-B c
F ron Dire
was decided that it was not the users’ location that would
Evaluate Left-Right Panning =
Figure 2: Back-projection, loudspeakers 2 video
and
Code interaction and
control the movement, but the users’ movement. The rationale
1 Amplitude Panning
camera
for this sound as there was only one virtual magnifying glass
was that object model
but there could be multiple simultaneous users, this mapping 3 Movement within space = Loudness
would encourage collaboration and engagement. 1.1. The challenge of used for fourth prototype public
Mappings auditory display in
While testing and evaluating the system in the lobby outside environments
our lab in the Interaction Design Centre at the University of (Shepard Tones)
Limerick, we noted that users who occasionally passed by the Is has been noted in numerous papers (e.g. [2-4]) that

26. Public Spaces - Lessons & Considerations
• Engaging (virtuality) • Design activities /
consultations
• Visitors / Working
within the space ✦ observational
studies
• Privacy / Intrusion ✦ interviews
• Masking and ✦ understanding the
soundscape issues physical /cultural /
(peak vs non-peak spatial constraints
times)

27. Hanging Gardens -
Auditory Installation
Collaboration with Jurgen Simpson
Linked two gallery spaces in Limerick and Belfast
Explore circadian rhythms of the spaces
Motion sensors and time of day, site-specific

28. Artistic Installations - Lessons & Considerations
• Interpretation vs • Design activities /
artistic consultations
• Engaging ✦ aesthetic issues
✦ distance /
• Group / Individual repairability
✦ understand
• Consider temporal
physical/spatial
constraints constraints

29. Action Sound Function
No touch N/A
Touch area outside button N/A
HCI Design and
Enter button area Tick
Interactive
Move finger on button Friction sound
Sonification for
Fingers and Ears Exit button area Tack
Lift finger off button Tock Select / Activate function

30. Where and How to start designing with Sound
• How to select and classify suitable sounds for a particular
interaction design ?
• TaDa - select sounds to communicate the dimensions and
directions with relation to their actions and to that of the
system e.g. ears-lead-eyes design pattern
• Sounds linked to physical objects as dynamic entities with
manipulation / interaction
• Multidimensional information carrier - size, shape, material,
distance, speed, and emotional expression

31. Selection of Approaches Depending on Question
• Evaluating Perceptual Qualities of Single Auditory Stimuli
• Identification Tasks, Context-Based Ratings, Attribute
Ratings
• Evaluating Relations Among Auditory Stimuli
• Discrimination Trials, Similarity Ratings, Sorting Tasks,
MDS, CA, PCA, Pathfinder Analysis
• Evaluation of an application or an interface
• Surveys, Verbal Protocols, Heuristics, Task lists, etc.

32. Sound Creation
1
Context and Auditory
Display Definition
Review of existing
Everyday Musical
Sounds Sounds
2 2
Considerations Considerations
Activity, Sound Like, Activity, Mapping,
Vocalisations / Onomatopoeia Instruments
Find Closest Match in
F/X Library / Use
Foley
2
Explore Existing
Earcon Hierarchies /
Create New Hierarchy
2
approaches,
methods, and
2 3&4
Analyse Sound and Create Earcon Family
Consider Action/s and and Test - Does it
Objects making it sound right?
Inconsistent Attributes
or Mappings
techniques.
3&4 Confused Sounds
Create the Sound and
Test - Does it sound
right?
5 6
Evaluate Scalings / Auditory Characterisation
Mappings of Story / Scene / Account
Inconsistent Attributes 7 8 9 8
Elicit Descriptors & TaDa Analysis Narrative Sound Sonic Map Creation for
or Mappings
Constructs Artefact Creation Account / Scene / Story
Confused Sounds
Sound Creation
10
Rating of Constructs & 11
Descriptor Categorisation Structuring Of Concepts
Hearsay Analysis
Personal Constructs Groups of Concepts
Descriptor Categories
Associated Ratings
17
Evaluation
Inconsistent Attributes
12 13 or Mappings
Causal Uncertainty
Measures
Structuring Of Constructs Results Confused Sounds
Confused
Sounds
Sound Analysis
Groups of Constructs
14
Definition Of Attributes
Construction Of Scales
Attribute Scales
15
Validation Of Scales
Preliminary Results
Mismatched
Sounds
16
Category Refinement
17
Evaluation
Results Sound Analysis

33. Stage 1 - Develop consensus
Select Listening Panel, Sounds Select Listeners, Sounds &
Group or
Stage 1 - Develop individual
& Attributes to explore Attributes to explore Individual
descriptive language
descriptive language
based
Play audio stimuli (~100-200) Play audio stimuli (~100-200) approaches
Determine attribute scales, Determine attribute scales,
descriptors, key audio groups descriptors, key audio groups
ranking / rating of data
Stage 2 - Participant
Stage 2 - Participant
performance data
Gather global
Gather end Participant ranks, rates, or
scores from
user groups data
naive
preference
participants
data to preferences
Stage 3 - Mapping
Stage 3 - Mapping
of data
Map preference / performance Mapping of data using
data using statistical methods statistical methods

34. CLOSED project, 2008,
Functional Artefacts Ircam, Verona, Berlin
Sonic Interaction Functional Artefacts
Design Process for Linking design to evaluation
Functional Artefacts including basic design methods
Analyse, Shape,
Describe Interaction Materials Combine
Experience Interaction Interaction
in Context Gestalt Artefact
Evaluate
Rolling
Clapping 1
Examples: Clapping 2

35. CLOSED project, 2008,
Functional Artefacts Ircam, Verona, Berlin
Sonic Interaction Functional Artefacts
Design Process for Linking design to evaluation
Functional Artefacts including basic design methods
Analyse, Shape,
Describe Interaction Materials Combine
Experience Interaction Interaction
in Context Gestalt Artefact
Rolling
Evaluate
Clapping 1
Examples: Clapping 2

36. CLOSED project, 2008,
Functional Artefacts Ircam, Verona, Berlin
Sonic Interaction Functional Artefacts
Design Process for Linking design to evaluation
Functional Artefacts including basic design methods
Analyse, Shape,
Describe Interaction Materials Combine
Experience Interaction Interaction
in Context Gestalt Artefact
Rolling
Evaluate
Clapping 1
Examples: Clapping 2

37. CLOSED project, 2008,
Functional Artefacts Ircam, Verona, Berlin
Sonic Interaction Functional Artefacts
Design Process for Linking design to evaluation
Functional Artefacts including basic design methods
Analyse, Shape,
Describe Interaction Materials Combine
Experience Interaction Interaction
in Context Gestalt Artefact
Rolling
Evaluate
Clapping 1
Examples: Clapping 2

38. Explorations and Methods
• Listening test • Causal
approaches uncertainty
• Textual • 2-D perceptual
descriptor scaling
analysis
• Repertory grid
• Sonic Mapping technique

39. Graham Coleman,
Sonic Mapping PhD Thesis
Scotland, 2008
Listening Classifying
Background Sounds Foreground Sounds
*
Ac
als
als
Ac
tio
gn
gn
tio
+
ns
?
Si
Si
ns
0
0 *
Emotions Contextual Emotions
Sounds
Written descriptions of
Sorting Visible Hidden sounds and their type
Ac
als
ti
gn
on
Si
* Everyday
s
+ Speech
0 Music
Emotions ? Abstract /
Unknown

40. Graham Coleman,
Sonic Mapping PhD Thesis
Scotland, 2008
Background Sounds Foreground Sounds
Listening Classifying
*
Ac
als
als
Ac
tio
gn
gn
tio
+
ns
?
Si
Si
ns
0
0 *
Emotions Contextual Emotions
Sounds
Written descriptions of
Visible Hidden sounds and their type
Ac
als
tio
gn
ns
* Everyday
Si
+ Speech
Sorting 0 Music
Emotions ? Abstract /
Unknown

41. High Arousal
1 2 3 4 5 6 7 8 9
9 9
Zizi - The
alarmed excited
astonished
8 afraid
close/sit
delighted
8
7 angry stroke happy
7 Affectionate
Couch
Displeasure
6 annoyed pleased 6
Pleasure
petting
5 frustrated content 5
4 miserable
none serene 4
Stephen Barrass, Linda Davy
3 depressed calm 3 & Kerry Richens
2 bored relaxed 2 Australia, 2003
1 tired sleepy
1
1 2 3 4 5 6 7 8 9
Low Arousal

42. High Arousal
1 2 3 4 5 6 7 8 9
9 alarmed excitedZizi - The
9
astonished
8 afraid Affectionate8 delighted
close/sit
7 angry stroke Couch
7 happy
Displeasure
6 annoyed 6 pleased
Pleasure
Stephen Barrass, Linda Davy
& Kerrypetting
Richens
5 frustrated content
Australia, 2003 5
4 miserable
none serene 4
3 depressed calm 3
2 bored relaxed 2
1 tired sleepy
1
1 2 3 4 5 6 7 8 9
Low Arousal

43. George Kelly,
Repertory Grid USA, 1955
Listening
Classifying
Sorting

44. Dendogram from George Kelly,
USA, 1955
Repertory Grid
Source /
Object /
Type of
Interaction
Clusters of
`related’ or
`similar’
sounds

45. Advice and tips for achieving better sound designs
Selecting and classify suitable sounds
Verify your mappings
Rapid prototyping with tools such as PD,
SuperCollider, Chuck, etc.
Contexts / Users / Environment