1. Cardiff Goldsmiths workshop
Developing Capacity in Multimodal
Research
4th -5th July 2013
Analysing multimodal data
Bella Dicks

2. What are multimodal data?
• meaning-making ‘beyond’ verbal language
• research investigates the ‘multiplicity of how
people communicate’
• how talk, facial/bodily movements and
elements of material surroundings combine
together in the production of meaning
(Bezemer and Mavers, 2011: 192)

3. Epistemological commitments
• 3 paradigms for multimodal data:
– communication (social semiotics)
– action (ethnomethodology)
– experience (sensory ethnography)
• Cross-cutting approaches:
– Multimodal ethnography
– Multimodal discourse analysis

4. Sensory ethnography
• Key term: senses, perceptions
• ‘experience’ is prime way of knowing the social world
• Objective: understand how whole, sensing self is route
into the experiential
• methodology depends on a high degree of subjectivity
and reflexivity:
– Video tours
– Sensory interviews
• Video used to represent ‘place-making encounters’
between researcher, participants and the
material/sensory environment
Sarah Pink’s 2009 Doing Sensory Ethnography

5. Ethnomethodological approach
• social action is prime means of knowing social world:
• Key terms: modes, sequences, accomplishment
• Objective: How social actors accomplish everyday practices
through practical actions and reasoning
– analyse how meaning emerges ‘from within the course of
action’
• Methodology: in-situ, close-up observation of naturally-
unfolding action
• Video: camera tightly trained on action sequences
• Transcripts: video, action, talk: disassembled
• Close analysis: turn-taking, action as accomplishment of
social practices (e.g. rebukes); gaze-direction; orientations
to objects & ‘physical ecology’ etc.
Heath and Hindmarsh’s 2010 Video in Qualitative Research

6. Social semiotics
• Key terms: modes, semiotic resources, meaning-potential
– Actors’ active choice of signifiers to communicate for specific
social purposes
– choices of mode affects kind of knowledge produced
– Relate this to changes in wider communicational landscapes
&‘knowledge-domains’ (discourse)
• objective: documenting and inventorying diverse uses of
distinct semiotic resources/modes
• Methodology: embodied interactions; semiotic inventories
(room layouts, magazines, texts, artefacts)
• Video: represent use of semiotic resources in ‘sign-making
practices’
• Analysis: interactional analysis; textual analysis
Kress and Van Leeuwen’s 2001 Multimodal Discourse

7. Research frameworks
• non-verbal domains of meaning are
approached as:
– resources (originating in the ‘interaction order’
(ethnomethodology) or ‘sign-making order’
(semiotics), respectively).
– perceptions, originating in the self’s subjective
being-in-the-world (sensory ethnography).
‘looking, touching and listening’ v. ‘seeing, feeling and hearing’

8. Example: analysis of interaction in
science discovery
• Take a social semiotic/interactional/sensory approach:
• social interaction (actors + physical ecology) can be
approached through analysis of embodied
actions, talk, use of objects, gaze-
directions, gestures, etc.
• environments can be ‘read’ as multimodal
orchestrations/assemblages (what are the semiotic
resources afforded by physical setting and how do they
interact?)
• senses: think about textures as ‘feeling’; sounds as
‘hearing’; movement as ‘routes’; ‘followings’; ‘chains of
excitement’…..

9. embodied interaction
See also: Heath, C. and vom
Lehn, D. (2008) ‘Configuring
‘Interactivity’: Enhancing
Engagement in Science Centres
and Museums’, Social Studies of
Science 38(1): 63-91
Dicks, B. (2013) Interacting with …. what?
Exploring children’s social and sensory
practices in a science discovery centre.
Education and Ethnography. forthcoming

10. environment: material modalities of ‘discovery’
in two learning spaces
Exhibits Hall Science Theatre
all the modes employed contribute to meaning…

11. methodology
• social interaction
– exhibits as ‘sites of engagement’: for interactional
talk, accent (prosody), gesture, facial expression, gaze-
direction, body posture, position & movement – but also
dress, hairstyle, body decoration
• communicative environments
– interior layouts, wall displays, instructional texts, exhibits
(colour, texture, shape, position, opaqueness/light, weight,
movement, sound, etc),
– also related texts & discourses (marketing, discourses of
science, ‘hands-on learning’ etc)
• senses
– think about how e.g. water feels on skin; listen to how
senses are verbalised, registered in face/body

12. findings
• Centre promoted as a place for ‘learning through doing’
• But children primarily engaged in
– a) sensory pleasure-taking and
– b) playing out shifting relations of affect and sociality.
• discovery centres are highly social spaces: such that twin
dimensions of the social and the sensory govern children’s
responses to exhibits, eclipsing thoughts of science
• live demonstrations (science theatre) more controlled and
school-like, more sensorially restricted: seemed to serve as
a more effective channel of scientific communication
• old-style live demonstrations are more effective than
technologically interactive exhibits in foregrounding
scientific concepts