Ventola, guijjaro 2009 the world told and the world shown multisemiotic issues

The World Told and the
World Shown
Multisemiotic Issues
Eija Ventola
and
Arsenio Jesús Moya Guijarro

The World Told and the World Shown

Also by Eija Ventola
FROM LANGUAGE TO MULTIMODALITY: New Developments in the Study of
Ideational Meaning (edited with C. Jones, 2008)
INTERPERSONAL COMMUNICATION (edited with G. Antos and T. Weber, 2008)
PERSPECTIVES ON MULTIMODALITY (edited with C. Charles and M. Kaltenbacher,
2004)
THE LANGUAGE OF CONFERENCING (edited with C. Shalom and S. Thompson,
2002)
DISCOURSE AND COMMUNITY: Doing Functional Linguistics (editor, 2000)
COHERENCE IN SPOKEN AND WRITTEN DISCOURSE: How to Create It and How
to Describe It. (edited with W. Bublitz and U. Lenk, 1999)
ACADEMIC WRITING: Intercultural and Textual Issues (edited with A. Mauranen,
1996)
A FUNCTIONAL AND SYSTEMIC LINGUISTICS: Approaches and Uses (editor, 1991)
THE STRUCTURE OF SOCIAL INTERACTION: A Systemic Approach to Semiotics of
Service Encounters (1987)
Also by Arsenio Jesús Moya Guijarro
THE TEACHING AND LEARNING OF FOREIGN LANGUAGES WITHIN THE
EUROPEAN FRAMEWORK (La enseñanza de las lenguas extranjeras en el marco
europeo) (edited with J. I. Albentosa and C. Harris, 2006)
La Enseñanza de la Lengua Extranjera en la Educación Intantil (edited with J. I.
Albentosa, 2003)
Narración Infantil y Discurso: Estudio Lingüístico de Cuentos en Castellano e
Ingles (with J. I. Albentosa, 2001)
TALK AND TEXT: Studies in Spoken and Written Discourse (edited with A. Downing
and J. I. Albentosa, 2000)
PATTERNS IN DISCOURSE AND TEXT: Ensayos de Análisis del Discurso en Lengua
Inglesa (edited with A. Downing and J. I. Albentosa, 1998)

The World Told and
the World Shown
Multisemiotic Issues
Edited By
Eija Ventola
University of Helsinki, Finland
and
Arsenio Jesús Moya Guijarro
Universidad de Castilla-La Mancha, Spain

Selection and editorial matter © Eija Ventola and
Arsenio Jesús Moya Guijarro 2009
Chapters © their individual authors
All rights reserved. No reproduction, copy or transmission of this
publication may be made without written permission.
No portion of this publication may be reproduced, copied or transmitted
save with written permission or in accordance with the provisions of the
Copyright, Designs and Patents Act 1988, or under the terms of any licence
permitting limited copying issued by the Copyright Licensing Agency,
Saffron House, 6-10 Kirby Street, London EC1N 8TS.
Any person who does any unauthorized act in relation to this publication
may be liable to criminal prosecution and civil claims for damages.
The authors have asserted their rights to be identified as the authors of this
work in accordance with the Copyright, Designs and Patents Act 1988.
First published 2009 by
PALGRAVE MACMILLAN
Palgrave Macmillan in the UK is an imprint of Macmillan Publishers Limited,
registered in England, company number 785998, of Houndmills, Basingstoke,
Hampshire RG21 6XS.
Palgrave Macmillan in the US is a division of St Martin’s Press LLC,
175 Fifth Avenue, New York, NY 10010.
Palgrave Macmillan is the global academic imprint of the above companies
and has companies and representatives throughout the world.
Palgrave® and Macmillan® are registered trademarks in the United States,
the United Kingdom, Europe and other countries.
ISBN: 978–0–230–57635–3 hardback
This book is printed on paper suitable for recycling and made from fully
managed and sustained forest sources. Logging, pulping and manufacturing
processes are expected to conform to the environmental regulations of the
country of origin.
A catalogue record for this book is available from the British Library.
A catalog record for this book is available from the Library of Congress.
10 9 8 7 6 5 4 3 2 1
18 17 16 15 14 13 12 11 10 09
Printed and bound in Great Britain by
CPI Antony Rowe, Chippenham and Eastbourne

v
Contents
List of Illustrations vii
List of Tables viii
List of Figures ix
List of Appendices xi
Acknowledgements xii
Notes on Contributors xiv
1 Introduction. The World Told and the World Shown:
Multisemiotic Issues 1
Eija Ventola and Arsenio Jesús Moya Guijarro
Part I Multimodal Theories: Coding the Visual
2 Multisemiosis and Context-Based Register Typology: Registerial
Variation in the Complementarity of Semiotic Systems 11
Christian M. I. M. Matthiessen
3 Developing Multimodal Texture 39
Martin Thomas
4 Metonymy in Visual and Audiovisual Discourse 56
Charles Forceville
5 What Makes Us Laugh? Verbo-Visual Humour in
Newspaper Cartoons 75
Elisabeth El Refaie
6 Citizenship and Semiotics: Towards a Multimodal
Analysis of Representations of the Relationship
between the State and the Citizen 90
Giulio Pagani
Part II Children’s Narratives and Multisemiotics
7 On Interaction of Image and Verbal Text in a Picture Book. A
Multimodal and Systemic Functional Study 107
Arsenio Jesús Moya Guijarro and María Jesús Pinar Sanz

vi Contents
8 The Text-Image Matching: One Story, Two Textualizations 124
María Cristina Astorga
Part III Text and Visual Interaction in
Advertising and Marketing
9 Sequential Visual Discourse Frames 139
Kay L. O’Halloran and Victor Lim Fei
10 A Systemic Functional Framework for the Analysis of
Corporate Television Advertisements 157
Sabine Tan
11 Multisemiotic Marketing and Advertising: Globalization versus
Localization and the Media 183
Anna Hopearuoho and Eija Ventola
Part IV Multisemiotics in Enacted Roles and
Virtual Identities
12 Taking the Viewer into the Field: Interaction between
Visual and Verbal Representation in a Television
Earth Sciences Documentary 207
Alison Love
13 Developing the Metafunctional Framework for Analysing
Multimodal Hypertextual Identity Construction 220
Arianna Maiorani
Part V Integrating Text, Visual and Space Multimodally
14 From Musing to Amusing: Semogenesis and
Western Museums 245
Maree Stenglin
15 Floods and Fidget Wheels: A Comparative Systemic
Functional Analysis of Slessor’s ‘Five Bells’ and
Olsen’s ‘Salute to Five Bells’ 266
Kathryn Tuckwell
Index 289

vii
Illustrations
3.1 The back of UK and Taiwan Head and Shoulders shampoo bottles 42
3.2 Three faces of a UK Sensodyne Original Toothpaste pack:
front (Face 1), side (Face 2) and back (Face 3) 43
4.1 Billboard for Interpolis Insurances, photographed in Haarlem,
Holland, summer 2006; original in colour 60
4.2 Billboard for ABN-Amro, photographed at Schiphol airport,
Holland, 2006, original in colour 62
4.3 Extreme close-up of priest mouth’s shouting in Joan’s ear
(a film still from La Passion de Jeanne d’Arc, Carl Dreyer) 65
4.4 A priest puts a pen in Joan’s hand, urging her to sign
a declaration she recants (film still from La Passion
de Jeanne d’Arc, Carl Dreyer) 66
4.5 Fontaine fiddles with the car door handle, considering escape
(a film still from Un Condamné à Mort s’est Échappé,
Robert Bresson) 67
4.6 Fontaine opens his handcuffs with a pin (a film still from
Un Condamné à Mort s’est Échappé, Robert Bresson) 68
5.1 Mac (Stan McMurty), Daily Mail, 5 November 2004, p. 17 76
5.2 Peter Schrank, Independent, 15 October 2004, p. 38 76
6.1 Co-occurrence of public sector and private sector bus liveries 98
7.1 Narrative process: symmetrical interaction 115
7.2 Narrative process: complementary interaction 116
8.1 The orientation stage in the simplified and
non-abridged versions 127
9.1 Cartier advertisement and centrefold (Time Asia, 8 May 2006) 140
11.1 Toyota advertisements in different media 191
13.1 First phase of LOTRO character creation – choice of race and
gender. Retrieved from www.youtube.com, April 2008 226

viii
Tables
2.1 Intersection of visual and aural contact values (adapted from
Martin, 1992: Table 7.3), with examples of multisemiotic
combinations; cells representing prototypical spoken
language and prototypical written language are shaded 24
2.2 The different SOCIO-SEMIOTIC PROCESS types and
PHENOMENAL DOMAIN values for written mode
with examples of multisemiotic documents 30
7.1 Visual and verbal interaction 119
9.1 SF-MDA framework for print advertisements
(based on O’Halloran, 2008a) 142
10.1 Phasal and information structure of the HSBC-text 167
10.2 Intersemiotic repetition of conceptual narrative theme 168
15.1 Functions and systems in painting, following
O’Toole (1994: Chapter 1) 273
15.2 Experiential meanings in Five Bells 278

ix
Figures
2.1 The stratification of semiotic systems: connotative and
denotative semiotic systems; context as connotative
system, stratification of denotative systems into
content plane and expression plane 13
2.2 Multisemiotic integration and diversification as a cline
represented in terms of the ordered typology of systems 14
2.3 Multisemiotic possibilities – cline of integration of
different semiotic systems 16
2.4 The system network of MOOD, with systems realized
prosodically by tone integrated systemically (illustrated
for ‘declarative’ mood) with systems realized by the
presence and relative sequence of elements of the
modal structure of the clause 17
2.5 Integration of (ideational) meanings realized by images into
linguistic semantics in WHO weekly reports 19
2.6 Integration of display (table) in multisemiotic text by
means of identifying relational clause, with display
construed as Token and interpretation of
display as Value 20
2.7 Ontogenesis – from multimodal protolanguage
(Halliday’s, 2004, phase I) via a transitional period
to language and ‘paralanguages’ (phases II and III) 22
2.8 Mode, calibration of division of labor among systems
operating in context – division of semiotic labour
between denotative semiotic systems (language
and paralanguage); division of socio-semiotic labour
between denotative semiotic systems and social systems 25
2.9 Material distance and semiotic distance (tenor) –
Edward T. Hall’s ‘distance sets’ 27
2.10 Kinds of semiotic system in relation to registerial range –
from special-purpose systems to general-purpose ones 32
3.1 Waller’s (1987) illustrations of Gestalt principles of grouping 49
8.1 System of reference in two versions of
The Sly Fox and the Red Hen 131
10.1 Soundscapes: waveform analysis of the HSBC-text 169
13.1 Construction page schematic structure 227
13.2 Systemic functional representation of the hyper-contextual
functional process of causation 228

x Figures
14.1 Activities associated with the eighteenth-century
museum 250
14.2 Participants associated with the eighteenth-century
museum 251
14.3 Activities associated with a hybrid museum 256
14.4 Participants associated with a hybrid museum 257

xi
Appendices
10.1a Excerpt of transcription template for the multimodal
analysis of dynamic moving images 172
10.1b Phasal and narrative structure in the HSBC-text 174
10.1c List of notations/notational symbols 178
13.1 Systemic functional analysis of The Fellowship of the Ring 231
15.1 ‘Five Bells’ analysis 283
15.2 Olsen’s mural, Salute to Five Bells 285
15.3 A detail of Olsen’s mural 285

xii
Acknowledgements
The editors of this book want to thank all the authors for embarking upon
this project with us when it was first announced during two symposia
on multisemiotics held at the University of Castilla-La Mancha and the
University of Helsinki. We thank the authors for the co-operation and for
their patience during the production process of the book.
We would also like to thank the Departments of English of our corre-
sponding universities for providing us the necessary financial means for the
working meetings we have had during the preparation of this book and for
financing some editorial help for us.
We are particularly grateful to Tuomo Hiippala for revising the references
included in each chapter and doing the index and to Edie Cruise for her sty-
listic suggestions and helpful advice.
Last but certainly not least, we wish to express our gratitude for permis-
sion to reproduce both visual and written material in this book to the fol-
lowing companies and institutions:
● Procter and Gamble for kind permission to reproduce Illustration 3.1, The
back of UK and Taiwan Head and Shoulders shampoo bottles.
● GlaxoSmithKline for permission to use Illustration 3.2, Three faces of a
UK Sensodyne Original Toothpaste pack: front, side and back.
● R. Waller for permission to use the material shown in Figure 3.1, Waller’s
(1987) illustrations of Gestalt principles of grouping.
● Interpolis insurances for Illustration 4.1 from Billboard for Interpolis,
photographed in Haarlem, Holland, 2006.
● ABN-Amro for Illustration 4.2 from Billboard for ABN-Amro, photo-
graphed at Schiphol airport, Holland, 2006.
● GAUMONT for permission to reproduce Illustration 4.3, Extreme close-up
of the priest mouth’s shouting in Joan’s ear (a film still from La Passion de
Jeanne d’Arc, Carl Dreyer, France © 1928 GAUMONT, and Illustration 4.4,
A priest puts a pen in Joan’s hand, urging her to sign a declaration she
recants (a film still from La Passion de Jeanne d’Arc, Carl Dreyer, France ©
1928 GAUMONT).
● GAUMONT/NOUVELLES ÉDITIONS DE FILMS for permission to repro-
duce Illustration 4.5, a film still from Un Condamné à Mort s’est Échappé,
Robert Bresson, France © 1956) and Illustration 4.6, a film still from Un
Condamné à Mort s’est Échappé, Robert Bresson, France © 1956.
● Cartoonists Stan McMurty and Peter Schrank for their kind permission
to reprint their work in Illustration 5.1, mac (Stan McMurty), Daily Mail,

Acknowledgements xiii
5 November 2004, p.17 and in Illustration 5.2, Peter Schrank, Independent,
15 October 2004, p.38.
● John Law for permission to use the photograph in Illustration 6.1,
Co-occurrence of public sector and private sector bus liveries.
● Walter Books Ltd, London SE11 5HJ for permission to reproduce
Illustrations 7.1 and 7.2 from GUESS HOW MUCH I LOVE YOU by Sam
McBratney, illustrated by Anita Jeram. Illustrations © 1994.
● Ladybirds Books Ltd for the kind permission to use Illustration 8.1, from
The Sly Fox and The Little Red Hen retold by Joan Stimson, illustrated
by Brian Price Thomas © Ladybird Books Ltd 1993, and from The Sly Fox
and Red Hen written by Sue Ullstein, illustrated by John Dyke © Ladybird
Books Ltd 1987.
● Group Marketing HSBC Holdings plc – HGHQ for the illustrations and
material included in Chapter 10 from stills from HSBC’s ‘Okey Doke’
motocycle television commercial © 2004.
● Toyota Motor Sales, U.S.A., Inc. for the Avalon image in the banner ad and
Toyota Finland for the other ads in Chapter 11.
● HaperCollins Publishers for permission to use 50 lines of Poem Five Bells
by Kenneth Slessor.
● Sydney Opera House Trust for courtesy to use images included in
Chapter 15.
Every effort has been made to acknowledge ownership of copyright. The
editors offer their apologies if any further copyright holders have been
infringed upon unknowingly. The publisher will be glad to make suitable
arrangements with any copyright holder whom it has not yet been possible
to contact.

xiv
Contributors
María Cristina Astorga is an Associate Professor in the Faculty of Humanities
at the National University of Río Cuarto, Argentina. She is also a Lecturer
in Theories of Second Language Acquisition in the MA Programs at the
National University of Río Cuarto and The National University of Córdoba.
The focus of her research is on development in academic foreign and second
language writing from systemic functional and cognitive perspectives.
Elisabeth El Refaie is a Lecturer in Communication at the Centre for
Language and Communication Research at Cardiff University, United
Kingdom. The focus of her research is on visual and multimodal forms of
narrative, rhetoric and humour, and she is currently working on a project
which uses the graphic novel to explore multimodal semiotics. Her work
has appeared in scholarly journals such as Visual Communication, Journal of
Pragmatics, Journal of Sociolinguistics and Journal of Contemporary European
Studies.
Charles Forceville, Associate Professor in the Media Studies Department
of the University of Amsterdam (Holland), studied English language and
literature. He published Pictorial Metaphor in Advertising (Routledge 1996),
and currently co-edits Multimodal Metaphor (Mouton de Gruyter 2009). His
research has appeared in journals including Metaphor and Symbol, Journal of
Pragmatics, Language and Literature, Poetics, Poetics Today, The New Review of
Film and Television Studies, and in edited books. He also serves on the advi-
sory boards of Metaphor and Symbol, Journal of Pragmatics, Public Journal of
Semiotics, Atlantis, and Digital Studies.
Anna Hopearuoho, a former student of the Department of English,
University of Helsinki, Finland whose MA thesis, titled Advertising to Women
through the Internet – a comparative multimodal study between Finnish and
English, focused on the multisemiotic aspects of Internet advertising and
the cultural factors influencing advertising through the Internet. She is
currently working for a Finnish company, Wärtsilä Corporation, where her
duties include preparing promotional material for both traditional and elec-
tronic media.
Victor Lim Fei is a PhD Research Scholar at the Multimodal Analysis Lab,
Interactive Digital Media Institute at the National University of Singapore,
Singapore. He has been awarded the Singapore Ministry of Education
Postgraduate Scholarship and has been a past recipient of the National
University of Singapore Research Scholarship and the Singapore Public
Service Commission Scholarship. His research interests are in multimodality,

Contributors xv
literacy and pedagogy and he has also published several papers and book
chapters on image–text relations and curriculum development.
Alison Love is an Associate Professor of English Language and Linguistics
at the National University of Lesotho, Lesotho, where she teaches discourse
analysis, sociolinguistics and pragmatics. She has also taught at universities
in Zambia, Zimbabwe and Botswana. Her research interests include the dis-
course of academic disciplines and political discourse, particularly that of
Southern Africa. She has published articles in English for Specific Purposes and
Discourse and Society and chapters in a number of collections.
Arianna Maiorani is a Lecturer in Linguistics at Loughborough University,
United Kingdom. Her main fields of research are discourse analysis applied
to the study of literary texts and multimodality discourse analysis applied
to the study of visual outputs, websites and online environments. Among
her most recent publications is ‘Movies “reloaded” into commercial reality:
representational structures in “The Matrix” trilogy promotional posters’, in
From Language to Multimodality, New Developments in the Study of Ideational
Meaning, Carys Jones and Eija Ventola, Equinox 2008.
Christian M. I. M. Matthiessen is a Chair Professor of Linguistics and Head
of the Department of English at the Hong Kong Polytechnic University.
He has been involved in the development of Systemic Functional theory,
description, modelling and application since his early work on text gener-
ation by computer, including exploration of multimodal generation in the
mid-1980s. His work on multimodality and multisemioticity has been in
the area of text analysis, computational modelling (e.g. the Multex system
in the second half of the 1990s), and theoretical underpinnings.
Arsenio Jesús Moya Guijarro, Professor of Language and Linguistics at
the Fray Luis de León Teacher’s College, University of Castilla-La Mancha,
Spain, does research in discourse and text analysis. He has published sev-
eral articles on information, thematicity and multimodal discourses in
international journals such as Word, Text, Functions of Language and Journal
of Pragmatics. His research interests are also in Children’s Literature and
Applied Linguistics. Within this framework he has co-edited The Teaching
and Learning of Foreign Languages within the European Framework.
Kay L. O’Halloran is Director of the Multimodal Analysis Lab, Interactive
and Digital Media Institute (IDMI) and Associate Professor in the Department
of English Language and Literature at the National University of Singapore.
Kay O’Halloran is an internationally recognized scholar in multimodal
analysis and she has given plenary addresses on multimodal approaches to
mathematics and science and the use of digital technology for multimodal
analysis at many international conferences. Kay O’Halloran is Principal
Investigator for several large projects in the Multimodal Analysis Lab. For
further information, please see http://multimodal-analysis-lab.org/.

xvi Contributors
Giulio Pagani, Department of Linguistics and English Language at Lancaster
University, United Kingdom, has been awarded the Arts and Humanities
Research Council grant to pursue doctoral research on representing the
state in the English and French semiotic and social systems. He also teaches
discourse analysis at the University of East Anglia. His research interests
include systemic functional linguistics, social semiotics and political dis-
course analysis, particularly in application to the discourse and activities of
public sector institutions.
María Jesús Pinar Sanz is a Lecturer in Linguistics and Discourse analysis
at the University of Castilla-La Mancha, Spain. Her research interests are in
multimodal discourse analysis and, more specifically, in aspects related to
the analysis of election campaigns and political advertising. She has pub-
lished several articles on the generic structure of political ads and the rela-
tionship between the verbal and visual elements not only in political texts,
but also in children’s narratives.
Maree Stenglin is a Lecturer in Literacy and Learning in the Faculty of
Economics and Business at the University of Sydney, Australia. Her research
interests include literacy and learning, discourse analysis, English for
Academic Purposes (EAP), multi-modality and the semiosis of 3D space. Her
most recent publications focus on interpersonal communication, spatial
semiotics and multimodal semiotics in the context of education.
Sabine Tan is a PhD Research Scholar in Language Studies at the National
University of Singapore, Singapore. Her research interests include social
semiotics, visual communication, and multimodal discourse analysis. She
is particularly interested in applications of systemic functional theory and
its derivatives to the analysis of business discourses, corporate television
advertisements, corporate web pages, televisual and Internet-based news
discourse, and other emergent multimodal discourse genres.
Martin Thomas works at the Centre for Translation Studies at the University
of Leeds, United Kingdom, where he has contributed to various projects
involving corpus linguistics and translator training. In 2005, he began his
doctoral research on multimodal variation across fast-moving consumer
goods packaging from English- and Chinese-speaking markets. Though
based in translation studies, this project also draws on multimodal dis-
course analysis, information design and computational linguistics.
Kathryn Tuckwell is a researcher at the Centre for Language in Social Life
at Macquarie University, Australia, and she has worked on discourse analy-
sis projects focusing on news reporting, professional discourse in medical
and legal settings and pharmaceutical advertising. She is also completing
a doctoral thesis on complexity, investigating in particular the linguistic
features of a popular science explanation of emergent complexity. Along

Contributors xvii
with multimodality and intersemiosis, her other interests include literary
stylistics and the semiotics of street art.
Eija Ventola, Professor at the Department of English at the University of
Helsinki, Finland, studied English language and literature in Finland and
linguistics in Australia. She has held professorial posts in Germany and in
Austria and visiting researcher and guest professorships in various coun-
tries. Her research areas include functional linguistics, analysis of various
kinds of discourses (e.g. casual, service, academic, business, media), applica-
tions of linguistics into teaching and learning, and issues of multisemiotic
aspects of communication. She has published and edited 14 books alto-
gether and written over 80 research articles. Together with her students, she
regularly organizes international MUST-research symposia, Multi-Semiotic
Talks (e-mail: must.finland@hotmail.com) which focuses on the challenges
that multisemiotic changes in global and media communication set on our
societies.

This page intentionally left blank

1
1
Introduction. The World Told and
the World Shown: Multisemiotic
Issues
Eija Ventola and Arsenio Jesús Moya Guijarro
This collection of papers represents the research of scholars working within
different contexts, sub-disciplines and languages in different parts of the
world, while sharing the frameworks of systemic functional linguistics and
visual semiotics. The volume is concerned with the development of multi-
modal, or rather multisemiotic, meaning-making theory, and it enhances
the ways of multisemiotic analysis of texts and visuals in today’s media-
oriented world, hence the title of the book, The World Told and the World
Shown: Multisemiotic Issues. It draws the attention of linguists and students
alike to the fact that language rarely stands alone in written and spoken
discourses, that is, mono-modally, and that we urgently need to sharpen
our tools in analysing discourses multisemiotically. We cannot continue
analysing language alone, but need an integrated multisemiotic approach,
and the volume shows various ways of analysis within such an approach,
which will be conducted on multisemiotically realized discourses.
The principal aim of the volume is to point out the ways in which spoken
and written discourses combine with other modes, simultaneously mak-
ing use of the multiple resources of different semiotic systems as they are
subsequently created and consumed. The chapters discuss the relationship
between the discourses that ‘tell’ and visuals (either still or moving, like
film) that ‘show’. The viewpoint that all the various modes specialize in the
transmission of particular meanings is shared by all of the writers of the vol-
ume, and their understanding of the way discourses work in today’s world is
a semiosis of such varied modes. Discourses in our modern societies always
make use of the various resources of semiotic systems, and the following
chapters show how we can interpret what people say and do by means of
words and images.
The innovative component of this book in comparison to those existing
in the field is the application of current multisemiotic theories to a great var-
iety of genres: picture books, billboards, cartoons, advertising, web games,

2 Eija Ventola and Arsenio Jesús Moya Guijarro
science documentaries, poetry, etc. The volume begins with chapters that
take the theorizing of the text/discourse-visualization a step beyond current
frameworks. The book, which is divided into five sections, highlights the
importance of cultural and social aspects in the configuration of language
and visualizations as well as their uses in the community.
The first Part, Multimodal Theories: Coding the Visual, contains five
chapters that represent multimodal views in systemic functional linguis-
tics, cognitive linguistics and social semiotics. They focus on some relevant
expansions of current multimodal theories from their own perspectives
which, for the reader, are complementary approaches. The concepts intro-
duced are, for example, the cline of integration of telling and showing,
multimodal cohesion, metonymy, multimodal issues in representations of
humour, semiotic metaphor and resemiotization. They challenge current
views and encourage the theoretical and analytical experimentation which
can break conventional boundaries of research on multisemiotics.
Part I begins with Chapter 2 by Christian Matthiessen, Multisemiosis
and Context-Based Register Typology: Registerial Variation in the
Complementarity of Semiotic Systems. Matthiessen discusses some essential
aspects of multisemiotic systems operating together in the same context. He
explores these systems in terms of a typology of systems of different orders –
physical, biological, social and semiotic systems, and he proposes ‘a cline of
integration’ for different semiotic systems. He argues that at one pole of this
cline, different semiotic systems are in fact integrated within one and the
same semiotic system and gives the integration of ‘melody’ into language
in the form of intonation as an example. However, as we move towards the
other pole of the cline of semiotic integration, he claims, we find semi-
otic systems that are increasingly distinct and separate from one another.
Thus, it is necessary to account for these distinct and separate systems that,
nevertheless, operate together to create meaning in a mutually supportive
way. This involves exploring the context in which they are coordinated. He
illustrates the operation of parameters set out to study context – in particular,
Mode and Field. He shows the value of investigating cooperation of multi-
semiotic systems, especially when some meanings are ‘at risk’ within a reg-
ister that operates in a particular kind of context.
Chapter 3, Developing Multimodal Texture, by Martin Thomas, first shows
how the theory of systemic functional linguistics has been adapted by semio-
ticians, and how the theoretical multisemiotic tools have been expanded to
cover such systems as information value, salience and framing. By looking
at designs of packages that come from three distinct locales, China, Taiwan
and the United Kingdom, he is able to point out the necessity of developing
the theory to begin to account for multimodal texture as well. In those cases
in which the systems of framing and salience proposed by the grammar of
visual design are not sufficient to account for the texture of multimodal

Introduction. The World Told and the World Shown 3
messages, the field of typography (modulation and segmentation) provides
us with further tools allowing the creation of multimodal cohesion.
In Chapter 4, Metonymy in Visual and Audiovisual Discourse, Charles
Forceville’s starting point is a cognitivist-oriented approach to an originally
literary concept, the metaphor, which, as he points out, has traditionally
been considered a matter of language. Now it is a common assumption
of cognitivist linguists that other tropes besides the metaphor are worthy
of their attention, particularly of metonymy, although research has still
strongly focused on verbal aspects of the manifestations of metonymy.
However, Forceville argues that, like the metaphor, metonymy is a concep-
tual phenomenon rather than a verbal one, and it should also appear in sign
systems other than language. In this chapter, he formulates parameters that
can help us guide further research into non-verbal and multimodal meton-
ymy. To support his claims, Forceville analyses a number of pictorial and
multimodal metonyms in advertisements and film to show that cultural
knowledge and narrative context turn out to be essential in the construc-
tion of the metonymy and its interpretation.
In Chapter 5, What Makes Us Laugh? Verbo-Visual Humour in Newspaper
Cartoons, Elisabeth El Refaie outlines the three main approaches to humour:
superiority theory, incongruity theory and release theory. She attempts to
formulate an integrated approach to what is told and what is shown in
cartoons in British newspapers. The chapter develops ways of understand-
ing humour and the creative mechanisms and social functions of laughter
and ridicule. Earlier approaches have focused on verbal humour only, jokes
in particular, and they have ignored the important role of visuals, music,
sound and voice in many cases of humour. The chapter develops ways of
theorizing and analysing multisemiotic humorous texts and emphasizes the
importance of perceived intentionality, cultural knowledge and the shared
common ground in understanding humour in cartoons.
In the last chapter of this section, Chapter 6, Citizenship and Semiotics:
Towards a Multimodal Analysis of Representations of the Relationships
between the State and the Citizen, Giulio Pagani examines the discursive
construction of states and citizens by considering the meanings of the mul-
tisemiotic texts made publicly available. He proposes a systemic function-
ally based model for analysing multisemiotic meaning-making resources.
His chapter focuses on the semiotic potential of discourses in public sector
service provision. Complementing the cognitive perspective discussed by
Forceville in the previous chapter, he demonstrates how a combined ana-
lysis of register, semiotic metaphor and ‘resemiotization’ can be used to track
meaning-making and interaction across a range of modes. He shows how
the critical investigation of multimodal discourse resources is a valuable and
worthwhile task for analysing how states and citizens shape their expecta-
tions of each other.

Part II, Children’s Narratives and Multisemiotics, includes two chapters
that cover the interaction between the verbal and the visual in children’s
narrative picture books. The meaning potential of such tales can only be
fully revealed by detailed multimodal analyses – they show how what is
told and what is shown complement and enhance one another. Using and
adapting earlier frameworks on visual design and functional linguistics, the
authors in this section highlight the ways in which the intersemiotic inter-
action of verbal and non-verbal modes contribute to the process of con-
structing meanings in picture books, written for both native and non-native
young second language readers.
In Chapter 7, On Interaction of Image and Verbal Text in a Picture Book.
A Multimodal and Systemic Functional Study, Arsenio Jesús Moya Guijarro
and María Jesús Pinar Sanz analyse the co-deployment and interaction
of verbal and visual elements in Guess How Much I Love You, a children’s
narrative for six-year-olds and under. The study reveals an essentially
symmetrical/complementary creation of meaning at both the visual and
verbal levels. As the narrative is intended for young children, no cases of
contradictory or counterpointing interactions have been identified; rather,
the visual and verbal components seem to reinforce each other and fulfil
complementary roles in the meaning-making process.
In Chapter 8, The Text-Image Matching: One Story, Two Textualizations,
María Cristina Astorga analyses the interaction of text and image within
the context of EFL/L2-learning. The author compares two different ver-
sions of the same story intended for young readers whose mother tongue
is English and for those learning English as a foreign or second language.
This comparative analysis focuses simultaneously on both modes – the told
and the shown – in order to determine to what extent these two versions
may or may not share a resemblance. Using the grammar of visual design,
the author captures the essential experiential meanings of the stories as
they are communicated by both language and images, and shows how links
between the processes, participants and circumstances are realized both lin-
guistically and visually. The findings from the study suggest that in order
to enhance the teaching of the visualized stories, EFL/L2-teachers need
to learn to reread picture books in new ways which involve the ability to
uncover relationships of meanings between language and image.
Part III, Text and Visual Interaction in Advertising and Marketing, brings
together three papers that discuss and theorize how texts and visuals interact
in advertising and marketing discourses. The advertising examples discussed
in the chapters vary in their dealing with traditional paper format, TV-film
and Internet modes. They share the common problem of sequencing in
advertising and how to deal with this in respective modes. Simultaneously,
perspectives are given on how luxury advertising is intermingled in media
print and how corporate and product advertising is realized in television
and pop-up advertisements on the Internet.

In Chapter 9, Sequential Visual Discourse Frames, Kay O’Halloran and
Victor Lim Fei explore questions such as: What are the systems that operate
in the visual mode? and How are meanings produced through sequential
visual discourses? Understanding the systemic operations of visual modal-
ity is empowering as it enables the design of advertising visuals that are
communicatively and ideologically effective. But at the same time, to bal-
ance this out, the consumers need to develop their critical reading abilities
of these advertising texts. The chapter focuses on developing new possibil-
ities for research on designing and reading visual discourses by consider-
ing the applications and limitations of the intersemiosis between language
and images; and thus demonstrating these in practice with the analysis of a
sequence of visual text in a themed Cartier paper advertisement.
In Chapter 10, A Systemic Functional Framework for the Analysis of
Corporate Television Advertisements, Sabine Tan shows how semiotic modes
and resources combine in complex ways in corporate television advertise-
ments. In order to enhance our understanding of these semiotic modes and
their resources, this chapter proposes an integrative systemic functional
multisemiotic framework for exploring the meaning potentials that are con-
veyed through the processes of intra- and intersemiosis in a dynamic multi-
modal text. It examines the multimodal meaning-making mechanisms that
operate in a corporate television advertisement for an international finan-
cial institution and discusses the methodological aspects of selection criteria
for the segmentation of dynamic text into appropriate constituent levels. It
concludes by evaluating the semiotic approach and industrial practices in
the analyses of corporate television advertisements.
In Chapter 11, Multisemiotic Marketing and Advertising: Globalization
versus Localization and the Media, Anna Hopearuoho and Eija Ventola dis-
cuss the need to localize global product marketing on the Internet and the
consequential multisemiotic realizational differences of global product ads
for local contexts. The chapter shows how a number of advertising agen-
cies in a local market see the ‘localization processes’ and then exemplifies
some of the multimodal strategies used for globalization and localization of
products in Internet marketing advertising. The analysis and results gener-
ated show that there is a growing need in this field to train interdisciplin-
ary experts able to design such advertisements while being linguistically
and semiotically sensitive to the localization needs of the global market.
Hopearuoho and Ventola highlight the fact that local contexts may demand
totally different linguistic and other semiotic realizations both in traditional
and current means of advertising through the use of the new medium, the
Internet (i.e. local languages are used for advertising, and certain cultural
semiotic realizations are also highlighted in the ads).
Part IV, Multisemiotics in Enacted Roles and Virtual Identities, discusses
the use of multisemiotic resources in an enactment of real and virtual iden-
tities. Here the focus is first on how verbal and visual modes complement

each other in a television documentary series and thus lead viewers to
interact with TV presenters and experts in the field of geology. The second
focus is on the interactions that are created in a virtual world. The chapters
together show how we construe our own world multimodally by the enact-
ment of our communicative roles through various semiotic modes.
In Chapter 12, Taking the Viewer into the Field: Interaction between Verbal
and Visual Representation in a Television Earth Sciences Documentary,
Alison Love discusses the verbal and visual strategies that are used in popu-
larizing science in a television documentary series, Earth Story, screened by
the BBC in 1998 (DVD 2006). The series sets out to answer questions about
the formation of the Earth, plus the forces that have changed it over time. The
chapter examines the ways in which the presenters use the verbal and visual
modes transporting viewers into the field of geology – literally – through
showing the places geologists go and the features they examine while, more
metaphorically, introducing viewers to the principles and methods of ‘doing
geology’. It shows how the two modes of representation, sometimes assisted
by the musical mode, complement each other to lead viewers to share and
enjoy an experience as a geologist.
In Chapter 13, Developing the Metafunctional Framework for Analysing
Multimodal Hypertextual Identity Construction, Arianna Maiorani focuses
on the fact that thousands of players all over the world, from a wide range
of ages and social backgrounds, are today attracted to the virtual world and
the adventures offered by online games. The chapter analyses the roles and
identity construction from one type of the Massively Multiuser Online Role
Player Games (MMORPG). Multimodal hyper-discourse is created as a result
of playing the game when one enters the discourse generated by the vir-
tual community of players. To do so, the player has to become a visually
active, interactive and creative participant. This process is social, which
therefore implies interaction and communication. The identity that a player/
participant takes on in order to participate in the hyper-discourse of the
game is a social construction that is created as a response to the hyper-social
context of the game and to his/her own social context. The chapter also
tests the ability of the Hallidayan metafunctional framework and its mean-
ing categories to capture these kinds of worlds, hyper-social multisemiotic
discourse activities and identities that the game generates through the use of
visual and verbal/audio resources.
The last section of the book, Part V, Integrating Text, Visual and Space
Multimodally, is concerned with the integration of text, visuals and space,
as well as the development and use of multimodal resources in meaning-
making contexts. The first chapter in this section gives us an interesting
view on how Western museums have developed over time and through
different stages into places of telling and showing, and even today into
multimodal places of entertainment. The second chapter in this section dis-
cusses how a piece of literature, such as a poem, is visualized in a culturally

significant way as a mural, thus completing the discourse of ‘the world told
and the world shown’ in this volume.
Chapter 14,FromMusingtoAmusing:SemogenesisandWesternMuseums,
by Maree Stenglin, applies social semiotic tools to illuminate the ideology
of Western museums in two seminal moments of their evolution as cul-
tural and multisemiotic institutions: the emergence of the public museum
in the eighteenth century, and the evolution of the hybrid museum of the
late twentieth and early twenty-first centuries. Stenglin uses the model of
social context, developed in systemic functional linguistics and the notion
of semogenesis, that is, the ways in which meanings unfold over time, to
show how the ideology of telling and showing in exhibition spaces has been
construed in concrete moments. In particular, semogenesis is conceived
as projecting both stratified planes of social context: context of situation
(register) and context of culture (genre). This relationship of projection is
an important one as it enables social semioticians to systematically explore
multisemiotic meanings from the perspective of social change.
Finally, in Chapter 15, Floods and Fidget Wheels: A Comparative Systemic
Functional Analysis of Slessor’s ‘Five Bells’ and Olsen’s ‘Salute to Five Bells’,
Kathryn Tuckwell discusses multisemiotic isomorphism between a poem
and a mural – two very distinct forms of art: the former ‘tells’ and the latter
‘shows’. The poem ‘Five Bells’ is very evocative of Sydney Harbour and is
written by the Australian modernist poet Kenneth Slessor who lived most
of his life near Sydney Harbour and drew his inspiration for the poem from
it. The mural is John Olsen’s ‘Salute to Five Bells’, and it pays homage to the
poet, the poem and the poem’s images of the Harbour. It was commissioned
from Olsen in 1971, the year Slessor died and the famous Sydney Opera
House was still being built on a small peninsula, surrounded on three sides
by the Harbour. The study of the comparison of the poem and the mural
demonstrates how multisemiotic systemic functional analyses can improve
our comprehension of how verbal and visual systems operate in meaning-
making. The isomorphism between different semiotic systems gives us evi-
dence that meaning-making has an inherent and universal structure, which
is used by everyone who makes meaning, regardless of their form of expres-
sion – not just artists, writers and musicians, but everyone who uses lan-
guage and other modes of meaning-making.
Human communication and experience has throughout ages been
recorded through writing systems of languages and images (sometimes as
parts of writing systems). Recording speech and action changed the descrip-
tion of human experience when gramophones, tape recorders, film/video
cameras, computers and Internet were developed in the previous century.
The challenge for this century is to develop tools to capture the complex-
ity of this new world of discourses as integrated multisemiotic realizations
of human communication and experience. The chapters in this volume
are a step towards developing ways of the intersemiosis of multimodal

meaning-making. The contributors have not only argued for the necessity
of studying contextual meanings in both verbal and non-verbal manifesta-
tions in different genres, but they have also sought ways and solutions for
analyzing such multimodal communicative artefacts as product packaging,
film and TV, cartoons, picture books, games, advertising in magazines, on
TV and in Internet, even public transport, museums and works of art in
order to see how the integration of modes works and what we can learn from
it. The authors in this volume are concerned with the effects and implica-
tions of multisemiotic integrations and call for further research in under-
standing our social realities in the integrated multisemiotic global world.
Their work points out that Multisemiotics seems to be an appropriate disci-
pline to deal with the complex communicative manifestations of the world
we live in now.
We, as editors, hope that the readers will find their immersion into ‘The
World Told and the Word Shown: Multisemiotic Issues’ a rewarding expe-
rience and we hope that the discussions in this volume will entice them
to participate and contribute to the exploration to this exciting area of
Multisemiotics.

Part I
Multimodal Theories:
Coding the Visual

11
2
Multisemiosis and Context-Based
Register Typology: Registerial
Variation in the Complementarity
of Semiotic Systems
Christian M. I. M. Matthiessen
2.1 Introduction
This chapter discusses some aspects of multisemiotic systems operating
together within one and the same context. Semiotic systems are systems
capable of carrying or even (in the case of higher-order semiotic systems
such as language) of creating meaning. Multisemiotic systems are semi-
otic systems that operate in parallel in the carrying or creation of meaning,
working together within one and the same context (for a recent overview
of systemic functional contributions to the study of such systems, see
Martinec, 2005, and for a recent foundational systemic functional account
of multimodal documents, see Bateman, 2008; for recent collections of
contributions, see, for example, O’Halloran, 2004; Ventola, Charles and
Kaltenbacher, 2004; Royce and Bowcher, 2006; and for a recent text book,
see Baldry and Thibault, 2006).
A prototypical example of multisemiotic systems would be people interact-
ing in face-to-face conversation engaging different parts of the body (vocal-
ization, facial expression, gesture, posture) to exchange meanings. From
the point of view of the interactants exchanging meaning, this semiotic
deployment of different bodily, or somatic, systems is a Gesamtkunstwerk, ‘a
unified work of art’.1
The question of how such systems operate together – of
how they are organized to create a unified, or at least a coordinated, flow of
meaning in their context, is one of the key concerns of this chapter.
Section 2.2 explores multisemiotic systems briefly in terms of a typology
of systems of different orders of complexity – physical (first-order systems),
biological (second-order systems), social (third-order systems) and semiotic
systems (fourth-order systems). This typology will make it possible to explore
multisemiotic systems in the environment of systems of lower orders, that
is, in the environment of social, biological and physical systems.

12 Christian M. I. M. Matthiessen
Then, after the presentation of this ordered typology of systems,
Section 2.3 proposes a sketch of a cline of integration of different semi-
otic systems. At one pole of this cline, different semiotic systems are in fact
integrated within one and the same semiotic system, as in the case of the
integration of ‘melody’ into language in the form of intonation. However,
as we move towards the other pole of the cline of semiotic integration, we
find semiotic systems that are increasingly distinct and separate from one
another, and we need to account for how they operate together to create
meaning in a mutually supportive way by exploring the context in which
they are coordinated.
The final section, Section 2.4, explores the contextual parameters – in
particular, Mode and Field, and illustrates the value of investigating mul-
tisemiotic systems by reference to the meanings that are ‘at risk’ within a
particular register (or ‘genre’) operating in a particular kind of context char-
acterized by some range of values of Field, Tenor and Mode.2
2.2 Multisemiotic systems and types of system
When different semiotic systems, such as language and ‘body language’, lan-
guage and image, or language and music, operate together in the creation of
meaning in a multisemiotic system, they operate within one and the same
context, and they are coordinated within this context (with context being
interpreted as a connotative kind of semiotic system, within which multiple
denotative semiotic systems operate; see Martin, 1992).3
So functionally
these different semiotic systems are integrated within the context they
operate in so that they can create meaning seamlessly and synergistically.
Context is the semiotic environment, the environment of meaning, in
which all semiotic systems operate. Since one key ‘architectural’ feature of
all semiotic systems is that they are stratified into two planes, the content
plane and the expression plane (each of which may be internally stratified
into further levels of organization), context can be interpreted as the high-
est stratum within this hierarchy of stratification (see Halliday, 1978; Martin,
1992; Ghadessy, 1999); it is the stratum above the content planes of all denota-
tive semiotic systems, see Figure 2.1.
At the other end of the hierarchy of stratification, the expression plane,
multisemiotic systems are, however, not integrated but they are instead
diversified: they are realized through different expression systems, such as
those of spoken language (vocalization), ‘body language’ and music. This is
of course the reason for recognizing the condition of multimodality in the
first place: ‘multimodality’ is a feature of the expression planes of semiotic
systems in the first instance.
In addition to being semiotic, this diversification of the expression plane
is also manifested materially. We can explore this material manifestation by
means of the ordered typology of systems originally proposed by Halliday

Multisemiosis and Context-Based Register Typology 13
(e.g. Halliday, 1996, 2005; Halliday and Matthiessen, 2006; Matthiessen,
2007, forthcoming). In this typology, systems operating in different phe-
nomenal realms are ordered in complexity from less complex systems to
more complex ones:
● First-order systems: Physical systems. These were the first systems to emerge
in the universe, with the ‘big bang’, and have the widest phenomenal cover-
age, extending throughout the universe.
● Second-order systems: Biological systems [+ life]. These are physical systems
with the added feature of ‘life’; they are living physical systems, which
means that they self-replicate, individuate and are subject to evolution. They
emerged under very special, constrained physical conditions – what James
Lovelock (1991) calls the narrow window of life – on our planet around
3.5 billion years ago.
● Third-order systems: Social systems [+ ‘value’, or social order]. These are
biological systems with the added feature of ‘value’, or social order; they
are biological populations organized socially into networks of social beings
(‘persons’) playing different roles in different networks and characterized by
division of labour.
● Fourth-order systems: Semiotic systems [+ ‘meaning’]. These are semiotic
systems with the added feature of ‘meaning’; they are social systems that can
also carry or even create meaning: persons operating in roles in social net-
works are also ‘meaners’ taking on speech roles and creating and sustaining
Context
Connotative Denotative
Content
Expression
Figure 2.1 The stratification of semiotic systems: connotative and denotative semi-
otic systems; context as connotative system, stratification of denotative systems into
content plane and expression plane

semiotic networks, or ‘communication networks’, through the ongoing
exchange of meanings.
Physical systems and biological systems can be grouped together as mater-
ial systems – systems of matter; and social systems and semiotic systems
can be grouped together as immaterial systems – socio-semiotic systems:
systems of value and meaning (or ‘meaning’ in a broad sense; see Halliday,
2005).
We can now interpret multimodality within the expression plane of semi-
otic systems as multimateriality within the lower-order systems of matter –
that is, within biological systems and physical systems as in Figure 2.2. This
multimateriality covers the ‘signifying body’ (cf. Thibault, 2004) operating
in its signifying environment.
The two higher-order systems in the ordered typology of systems – that
is, social systems and semiotic systems – coordinate and integrate patterns
within the two lower-order systems – that is, biological systems and phys-
ical systems. Socio-semiotic systems give ‘meaning’ to matter (cf. Halliday,
2005): social systems impose social order (‘value’) on the world of matter,
and semiotic systems impose semiotic order (meaning in its narrower sense
of valeur and signification; cf. Hasan, 1985) on this social world. For instance,
social constructs such as tools, artefacts and dwellings are manifested in
many materially diverse ways, but such materially divergent manifestations
may have the same ‘value’ in the social system.
Connotative: context
Denotative: content
Denotative: expression
Semiotic Social Biological Physical
{+ meaning}
“Meaning” “Matter”
{+ value} {+ life}
Socio-semiotic
integration
Material
diversiﬁcation
Figure 2.2 Multisemiotic integration and diversification as a cline represented in
terms of the ordered typology of systems

2.3 Cline of semiotic integration
Having discussed the typology of systems, let us now focus on integration
and diversification within semiotic systems. The hierarchy of stratification
within semiotic systems is as it were a replay of the ordered typology of
systems operating in different phenomenal realms – both within a primary
semiotic system such as protolanguage and within a higher-order semiotic
system such as language. Between the coordination and integration within
the context of semiotic systems and the diversification within the expression
plane of semiotic systems, we can recognize different degrees of integration
within the content plane of semiotic systems (see Figure 2.3). These different
degrees of integration form a continuum or cline of semiotic integration.
This cline of integration is defined by its two outer poles – the pole of
maximal integration and the pole of minimal integration.
(i) At the pole of maximal integration, there is one semiotic system,
and the different expressive systems involving different ‘modalities’ are
integrated within one and the same content stratum. This is how multimo-
dality within spoken language has been modelled in systemic functional
linguistics since Halliday’s groundbreaking work on intonation and gram-
mar in the early 1960s (e.g. Halliday, 1963, 1967; Halliday and Greaves,
2008). The approach is illustrated by the fragment of the lexicogrammatical
system of MOOD and the phonological system of TONE in Figure 2.4. Here
the integration is achieved within the stratum of lexicogrammar along the
systemic (paradigmatic) axis. That is, systemically it does not make a differ-
ence whether terms in systems are realized by the presence of elements in
the modal structure of the clause (e.g. ‘indicative’ realized by the presence
of the Mood element), by the relative sequence of elements (e.g. ‘declarative’
realized by the sequence of Subject ^ Finite), or by the direction of the pitch
movement in an intonation contour (e.g. ‘reserved’ realized by ‘tone 4’ –
phonetically a fall–rise pitch movement). What matters systemically is sim-
ply that systemic values such as ‘indicative’, ‘declarative’ and ‘insistent’ are
realized in such a way that these values are kept distinct in the expression.
(ii) At the pole of minimal integration, two or more semiotic systems are
completely separate as (denotative) semiotic systems – that is, separate in
terms of both their content systems and their expression systems, and these
systems are integrated and coordinated only at the highest level of semiotic
organization, that is, within the (connotative) semiotic system of context
(for the distinction between denotative and connotative semiotic systems,
based on Hjelmslev, 1943, see Martin, 1992). An example of this case relat-
ing to the earlier illustration of intonation being integrated within language
as linguistic ‘melody’ would be language and music in a folk ballad, as mod-
elled by Steiner (1988). He describes these two semiotic systems separately,
and, having done this, is then in a position to show how they interact.4

Cline of integration
Maximal
integration
Minimal
integration
Same system,
same stratum
Different
syntagm
Different
lower stratum
Different
semiotics
Same system,
same higher
stratum
Same system in
context
Degree of
diversiﬁcation
PRETONIC
Tone group
Clause
STATUS
Minor
Major
FREEDOM
Bound
Free
MOOD
Indicative
Interrogative
Yes/no interrogative
Finite ^ Subject
Wh- interrogative
Neutral
Protesting
Tentative
Reserved
Insistent
Marked
+Wh; Wh ^ Finite
Subject ^ Finite
Declarative
+Mood
(+Finite;
+Subject)
Imperative
TYPE
+Residue
(+Predicator)
+Tonic
TONIC
COMPOSITION
PhonologyLexicogrammar
Compound
+Tonic 2;
Tonic ^ Tonic 2
[Tone 1]
[Tone 3]
[Tone 2]
+Pretonic;
Pretonic ^Tonic
Without pretonic
With pretonic
Simple
Tone 1
Tone 2
Tone 3
Tone 4
Tone 5
1+ wide
1. medium
1- narrow
2. straight
2_ broken
4. high
4_ low
5. high
5_ low
.1 even
—1 bouncing
...1 listing
.2 high
—2 low
.3 mid
—3 low
Tone 13
Tone 53
Figure 2.3 Multisemiotic possibilities – cline of integration of different semiotic systems
10.1057/9780230245341 - The World Told and the World Shown: Multisemiotic Issues, Edited by Eija Ventola and Arsenio Jesús Moya Guijarro

PRETONIC
Tone group
Clause
STATUS
Minor
Major
FREEDOM
Bound
Free
MOOD
Indicative
Interrogative
Yes/no interrogative
Finite ^ Subject
Wh- interrogative
Neutral
Protesting
Tentative
Reserved
Insistent
Marked
+Wh; Wh ^ Finite
Subject ^ Finite
Declarative
+Mood
(+Finite;
+Subject)
Imperative
TYPE
+Residue
(+Predicator)
+Tonic
TONIC
COMPOSITION
PhonologyLexicogrammar
Compound
+Tonic 2;
Tonic ^ Tonic 2
[Tone 1]
[Tone 3]
[Tone 2]
+Pretonic;
Pretonic ^Tonic
Without pretonic
With pretonic
Simple
Tone 1
Tone 2
Tone 3
Tone 4
Tone 5
1+ wide
1. medium
1- narrow
2. straight
2_ broken
4. high
4_ low
5. high
5_ low
.1 even
—1 bouncing
...1 listing
.2 high
—2 low
.3 mid
—3 low
Tone 13
Tone 53
Figure 2.4 The system network of MOOD, with systems realized prosodically by tone integrated systemically (illustrated for ‘declarative’ mood)
with systems realized by the presence and relative sequence of elements of the modal structure of the clause
10.1057/9780230245341 - The World Told and the World Shown: Multisemiotic Issues, Edited by Eija Ventola and Arsenio Jesús Moya Guijarro

(iii) Intermediate between the two outer poles of the cline of integra-
tion are cases where two or more semiotic systems can be modelled as inte-
grated into one system at the stratum of semantics – that is, at the higher of
the two content strata. This possibility has in fact been explored since the
mid-1980s in computational systems capable to generating multisemiotic
presentations embodying coordinated semiotic strands, such as online text
accompanied by pointing gestures (e.g. Reithinger, 1987) and online text
accompanied by maps (e.g. Matthiessen et al., 1998). Even though they are
typically not referred to in the literature on ‘multimodal analysis’, systems
of this kind are quite important because they include fully explicit models
of multisemiotic systems.
In the modelling of face-to-face interaction in terms of the cline of inte-
gration, it is likely that gesturing and language can be integrated within
a single system of meaning at the level of semantics; the high degree of
interaction between them (including subtle synchronization of the onset
of gestures relative to points of the unfolding of clauses) suggests that this
may be both possible and necessary (cf. e.g. McNeill and Duncan, 2000;
Haviland, 2000). For example, in terms of experiential meaning, it seems
clear that gestures need to be related to the elements of the figure that is
realized by a clause in its experiential manifestation – the process, partici-
pants or circumstances (see Halliday and Matthiessen, 1999) – as points
of complementarity and synchronization; and language-gesture systems
differ with respect to the complementarity of the two in ways that indi-
cate some form of semantic integration, as shown by McNeill and Duncan
(2000, pp. 149–51) with respect to the construal of motion through space.
For instance, while Spanish tends to lexicalize the path of motion in verbs
of motion, English tends to lexicalize the manner of motion (verbs incorp-
orating path such as cross, exit, enter commonly being of Romance origin)
and use them with expressions of path (as in tango into the dining room,
float out of the harbour) (see Talmy, 1985, and the research building on his
foundational study); and speakers of Spanish tend to indicate the manner of
motion gesturally rather than lexically.5
In the modelling of the written mode in terms of the cline of integra-
tion, the same degree of integration within semantics may be possible
with language and images. For example, in our study of the World Health
Organization (WHO)’s Weekly Epidemiological Reports (WERs), we were able
to integrate the meanings of images (maps and graphs) and text in English
in an account of the semantics of the domain that the reports are concerned
with, the domain of communicable diseases, as shown in Figure 2.5 (for dis-
cussion, see Matthiessen, 2006).
The semantic system specific to the register of these reports can be
modelled based on the meanings realized by texts in English (or French),

yielding the network shown in the diagram. Within this overall semantic
system, it is then possible to locate the subset of meanings that are realized
by images – by graphs and maps (and also be tables). These meanings are
concerned with quantification and location of phenomena that are mea-
sured in the reports, like deaths and outbreaks of communicable diseases.
The images are integrated into the multimodal ‘text’ by means of clauses in
language that relate references to displays (Map 1, Table 1 and so on) to the
linguistic text, as in: In 2003, Afghanistan reported 8 polio cases (5 P1 and 3
P3). As at the end of May 2004, 2 P1 and one P3 cases had been reported (Map 1);
this relationship between text and image is often construed explicitly
Figure 2.5 Integration of (ideational) meanings realized by images into linguistic
semantics in WHO weekly reports

by ‘relational’ clauses of the ‘identifying’ type, as in Table 1 summarizes
the scope of the SIAs and their impact on reported cases of neonatal tetanus
(NT) (from WER 8113). In this later case, the clause construes an identity
between a display, a table and an interpretation of this display; the table is
construed as Token (Table 1) and it is related to a Value giving a ‘gloss’ in
English indicating how it is to be interpreted; Figure 2.6. represents this
relationship. Similarly: Table 3 shows the results in more detail; Fig. 1 shows
the proportionate coverage of at-risk populations in implementation units by type
of drugs used in MDA; Fig. 3 shows the involvement of various research institu-
tions in the priority areas of research; Table 1 reports the distribution of the 9585
cases of dracunculiasis reported in 2007 by month and compares this with 2006.
The Process of such relational clauses construes the relationship between
the two semiotics; it is realized by a verbal group with a ‘symbolizing’ verb:
show, report, summarize and the like.
The cline of integration can also be explored in terms of semohis-
tory – in terms of phylogenesis (the evolution of semiotic systems in the
species), ontogenesis (the development of semiotic systems in the individ-
ual) and logogenesis (the unfolding of semiotic systems as texts; for these
three timeframes, see Halliday and Matthiessen, 1999, p.18). It seems likely
that all three semohistories may involve movements in either direction
along the cline of integration: over time, semiotic systems may become
more highly integrated, moving towards the pole of maximal integration;
or alternatively, integrated semiotic systems may gradually split into more
Value:
the scope of the SIAs and their
impact on reported cases of
neonatal tetanus (NT)
Process:
summarizes
Token:
Table 1
Figure 2.6 Integration of display (table) in multisemiotic text by means of identify-
ing relational clause, with display construed as Token and interpretation of display
as Value

independent semiotic systems. Let me illustrate these movements in ref-
erence to phylogenesis and ontogenesis, and then comment briefly on
logogenesis.
Within the phylogenetic timeframe, there have been significant shifts
in the degree of integration of language and image on the page. Before the
twelfth century, illuminations ‘were a critical part of the text’, but in the
eleventh and twelfth centuries ‘images became subordinated to the text
which came, increasingly, to be seen as the primary conveyor of meaning’
(Olson, 1994, p. 112). After the development of printing technology in the
fifteenth century, books would often be assembled from different sources:
the author of the text had nothing to do with the illustrations, which would
be added later by the printer. Later, text and image gradually became more
highly integrated. However, today both printed matter and electronically
delivered web material often take the form of a collage where images domin-
ate, and it can be difficult for readers to know how to integrate them with
the text (this being one of the challenges for educational institutions in
dealing with multisemiotic literacy).
Within the ontogenetic timeframe, we can explore how young chil-
dren learn how to mean, and the study of this process of development
can shed interesting light on movements along the cline of integration:
somewhere around the age of five to eight months, human infants begin
to develop a protolanguage in interaction with their intermediate care-
givers in four contexts that are critical in early development – regulatory
(a kind of ‘enabling’ context from an adult point of view; cf. Table 2.2 in
Section 2.4.4), instrumental (a kind of ‘doing’ context), interactional (a kind
of ‘sharing’ context) and personal (also a kind of ‘sharing’ context) (see e.g.
Halliday, 2004).
This protolanguage is inherently multimodal (as shown by Halliday’s
description; for discussion, cf. Matthiessen, 2006), and from this multi-
modal protolanguage, both language and ‘paralanguages’ will develop,
as young children make the gradual transition from their protolanguages
to the mother tongues spoken around them, thus expanding their over-
all meaning potentials, as shown schematically in Figure 2.7. Here there
would thus seem to be a move from one integrated protolinguistic system
to coordinated but less integrated post-infancy semiotic systems – language
and paralanguages. (Thus a (post-infancy) language, a mother tongue, can
be fully instantiated in a phone conversation, where the channel can only
convey vocalizations; but a protolanguage cannot, since it is not confined
to vocalization.)
The expression plane of protolanguage is somatic; it is based on the bodily
resources of infants. The same is true of the expression planes of the lan-
guage and paralanguages that emerge during the transitional period, but at
some point children often begin to experiment with an exosomatic expres-
sion plane – drawing (going through a number of stages of different drawing

systems; see e.g. Willats, 1997); and when they enter their first institution of
formal education, they will gradually learn to write.6
Within the logogenetic timeframe, we can explore how speakers and
writers create meaning by instantiating semantic regions within the overall
meaning potential – typically, meanings that are ‘at risk’ within some par-
ticular register. As we investigate how meanings are created in the course of
the unfolding of text, we can identify emergent patterns involving two or
more semiotic systems. Such emergent patterns can be interpreted in terms
of the notion of local, instantial systems (cf. Matthiessen, 1993a, 1995,
2002) – systems that are formed out of the patterns of the instantiation of
a more general meaning potential further up the cline of instantiation (a
registerial sub-potential or by a further step the overall meaning potential
of a language). It seems plausible that when different semiotic systems that
are part of a multisemiotic system are instantiated alongside one another in
a multisemiotic text, they become more highly integrated in the instantial
systems of the unfolding text, and patterns of correlations across the sys-
tems emerge.
2.4 Context: Mode, Tenor and Field
Regardless of the degree of integration of semiotic systems deploying dif-
ferent media of expression, the key question in a multisemiotic system is
Somatic expression
Protolanguage
{multimodal}
Language
{monomodal:
phonology—
phonetics:
vocalization}
Paralanguages
{vocalization,
gesture, facial
expression,
posture & CC}
Phase I
Ontogenesis
Phase II Phase III
+ Exosomatic expression
{+Writing}
{+Drawing}
Figure 2.7 Ontogenesis – from multimodal protolanguage (Halliday’s, 2004, phase I)
via a transitional period to language and ‘paralanguages’ (phases II and III)

how the different resources for creating meaning complement one another
and how the semiotic labour (the work of creating meaning in context) is
divided among them. We can certainly approach this question ‘from below’,
considering it in terms of the affordances of the different media of expres-
sion – including, for example, the temporal characteristics and durability of
the medium in material terms.7
This is clearly significant, but the possibil-
ities will ultimately have to be accounted for within our description of the
context in which the different semiotic systems operate and within which
the semiotic labour is divided among them.
2.4.1 Mode: Medium and Channel
Within context, the contextual parameter of Mode relates to the ‘modality’
of the expression planes of different semiotic systems since it is Mode that is
concerned with the semiotic role of the semiotic systems operating within
a given context and the role of a particular semiotic system depends on the
affordances of its medium or media of expression (cf. Matthiessen, 2006, in
relation to multimodality).
In terms of Mode, both MEDIUM (spoken/written) and CHANNEL (aural/
visual/tactile/olfactory/gustatory) are important factors determining the
potential for different combinations of semiotic systems. With respect to
language, the MEDIUM is either spoken or written – or signed, in the case
of sign languages of deaf communities such as Auslan (see e.g. Johnston,
1992); and these different modes can combine with different ranges of other
semiotic systems, but these ranges will also depend on the nature of the
CHANNEL.8
One complex but important aspect of the interaction between
medium and channel has to do with implications in terms of time and
space in semiotic terms – whether ‘speaker’ and ‘addressee’ operate in the
same spatio-temporal realm or not, and whether they process instances in
real-time or not. Here sign languages are particularly interesting from a
multimodal point of view because they are like spoken languages in being
processed in real time and like written languages in being processed visu-
ally – one can compare the cline from paralinguistic gestures to signs in sign
languages (cf. McNeill, 2000a, 2000b) to the cline from drawing to writing
(cf. Matthiessen, 2006, and references therein).
The CHANNEL of communication is, as Martin (1992, p. 510) puts it, ‘the
semiotic construction of communication technology’, thus representing the
semioticization of the affordances of the material channel; it is concerned
with the bandwidth of semiosis between ‘speaker’ and ‘addressee’ as far as
the expression plane is concerned, ranging from minimal bandwidth when
they are not in any direct sensory contact to maximal bandwidth when they
are in full sensory contact. Naturally, the greater the bandwidth is, the greater
the opportunities will be for multiple semiotic systems to operate together in
the creation of meaning simply because a greater range of expression systems
will be available.

Exploring the possibilities of channel, Martin (1992, p. 511) presents a
matrix where the different possibilities of AURAL and VISUAL CONTACT are inter-
sected. He gives examples of the all intersections represented by the cells of
the matrix, and we can now present a version of his matrix with indications
of the potential for combinations of semiotic systems (Table 2.1); these inter-
sections define settings for different registers – for meanings at risk from the
point of view of aural and visual types of contact.
If we add other sensory channels of contact, we can make further differ-
entiation in terms of bandwidth (e.g. tactile contact determining the poten-
tial for semiotics of touch and olfactory contact determining the potential
Table 2.1 Intersection of visual and aural contact values (adapted from Martin,
1992: Table 7.3), with examples of multisemiotic combinations; cells representing
prototypical spoken language and prototypical written language are shaded
Visual contact
none one-way two-way
Aural
contact
none print media,
electronic media
silent film,
surveillance
signing (sign
language); mime
writingϩimages
[diagrams,
drawings,
photos etc.]
imagesϩwriting
[subtitles,
captions]
signing
ϩgesturing
one-way radio, audio
recording
television, film,
video; electronic
media (with audio)
lip-reading
speaking
ϩparalanguage
imagesϩ
speakingϩ
paralanguageϩ
writing [subtitles,
captions]
...ϩ gesturing
ϩfacial expression
ϩgazeϩposture
(ϩ proxemics)
two-way telephone
(including
mobile),
intercom,
internet chat
video intercom face-to-face
conversation,
video mobile,
video internet
chat
speaking
ϩparalanguage
speaking
ϩparalanguage
speaking
ϩparalanguage
ϩgesturing
ϩfacial expression
ϩgazeϩposture
(ϩproxemics)
ϩtouchϩsmell
(ϩ taste)

for the semiotics of fragrances and odours, as in the use of perfume). For
instance, while face-to-face conversation prototypically involves maximal
bandwidth (although there are principled exceptions, like service encoun-
ters at a ticket counter with a very limited view of the server), face-to-face
conversations based on the technologies of videophones and video Internet
chat are more constrained (not only in terms of audio [visual contact] and
video [aural contact], but also obviously in terms of smell, touch and taste
since these cannot yet be handled digitally in the same way as audio and
video).
2.4.2 Mode: Division of semiotic labour
Within Mode, the CHANNEL of communication thus determines the poten-
tial for different combinations of semiotic systems as far as the expression
plane is concerned, but Mode is also concerned with the DIVISION OF SEMIOTIC
LABOUR among the semiotic systems that enter such combinations. We can
characterize this crudely in reference to language as a cline between two
outer poles – one where all semiotic labour is done linguistically, and one
Connotative semiotic system:
context (mode)
Division of socio-semiotic labour
Division of (semiotic) labour
Denotative semiotic system
Language
Social systemParalanguage
Figure 2.8 Mode, calibration of division of labor among systems operating in con-
text – division of semiotic labour between denotative semiotic systems (language
and paralanguage); division of socio-semiotic labour between denotative semiotic
systems and social systems

where all semiotic labour is done non-linguistically, by some semiotic sys-
tem or systems other than language.9
For instance, in the WHO’s Weekly Epidemiological Reports (WERs), a
good deal of the semiotic labour is done linguistically (in English or French
text), and images (maps and graphs, in particular) make quite restricted,
specific contributions, as indicated above. Here it is possible to integrate
the account of the meaning of images into the semantic system of language
(the linguistic ‘domain model’; see Matthiessen et al., 1998; Matthiessen,
2006) – the mid-region of the cline of integration in Figure 2.3; and it seems
plausible that this would generally be the case when one semiotic system is
‘nuclear’ in the contribution to the semiotic labour in a given context and
the other system or systems make more of a supporting, peripheral contri-
bution. In cases such as the WHO’s WERs, language carries the main semi-
otic burden, but in other cases it may be another kind of semiotic system
that carries the main burden – as when language is used to label or annotate
images for purposes of presentation, sorting or retrieval from some kind of
archive.
One interesting – and critical – aspect of the DIVISION OF SEMIOTIC LABOUR
among denotative semiotic systems is the extent to which they operate in
semiotic harmony with one another: the interpretation of music in terms
of the harmony of chords, consonance, dissonance, counterpoint and so on
may be a useful model for exploring how different simultaneous semiotic
systems work together. For instance, a good deal of multisemiotic humour is
based on the ‘dissonance’ among different semiotic systems, as in Norman
Thelwell’s classic cartoons involving jocular advice about buying a house,
gardening or caring for a pony; here the interpretation of the apparently
straight-laced text is undercut by the meaning conveyed by the drawing.
Taking account of the division of semiotic labour along these lines of semi-
otic harmony is a considerable challenge in terms of theory, modelling and
description: we must take account of the meanings that are engendered
under different conditions of semiotic harmony, including the tensions
associated with irony and humour. In investigations of language, such phe-
nomena are of course also familiar, as in the work on the ‘polyphony’ of the
different metafunctional strands of meaning and the work on additional
layers of meaning created through the strategy of metaphor.10
Baldry and
Thibault (2006), for example, take account of the way that different semiotic
systems work together under the heading of ‘resource integration’.
In an account of multisemiotic systems and ‘texts’, Mode is thus respon-
sible for the ‘semiotic construction’ of the affordances within the expression
plane and for the division of semiotic labour. However, Field and Tenor are,
of course, also part of the account ‘from above’ – the contextual account – of
multisemiotic conditions. The focus will be on Field, but first the obvious
point that Tenor is equally important is made briefly.

2.4.3 Tenor (in relation to Mode)
Tenor interacts in interesting ways with visual and aural contact (see
Table 2.1 in Section 2.4.1). As part of his investigation that he called ‘prox-
emics’, Hall (e.g. 1966) has shown that ‘material distance’ realizes ‘semiotic
distance’ – more specifically, interpersonal distance within Tenor (to put
this in our terms): see Figure 2.9.
The more ‘intimate’ the Tenor of the relationships is, the wider the band-
width of the channel of communication is – the widest bandwidth being
associated with intimate face-to-face conversation, where there is two-way
contact in terms of all senses (cf. Table 2.1). The wider the bandwidth is,
the greater the range of interpersonal meanings that can be expressed
will be, since the face is a key resource for the expression of interpersonal
meanings.
Conversely, the more ‘public’ the Tenor of the relationship is, the nar-
rower the bandwidth of the channel of communication is – the narrowest
being associated with public addresses such as public speeches, where there
is at best one-way contact in terms of vision and hearing (members of the
public can see and hear the speaker as an individual but the speaker can
only see and hear members of the public as a collective).11
This interpersonal distance obviously relates directly to the design of pub-
lic spaces and buildings (cf. O’Toole, 1994) and also to the design of furni-
ture (illuminated by the investigation of the semiotics of IKEA tables by
Public
Social-consultative
Casual-personal
Intimate
Wholefacein
fovealvision
Arm’slength
Novisual
distortion
Offaction+
heatsense
Handreach
Faceoutof
focus
Fullbody
contact
8"
18"
30"
48"
7'
12'
15'
Figure 2.9 Material distance and semiotic distance (tenor) – Edward T. Hall’s ‘dis-
tance sets’

Anders Björkvall, whose research shows that tables differ in terms of their
potential for interaction).
After these brief comments on Tenor, let us now turn to Field.
2.4.4 Field
Field has been characterized in terms of two basic parameters (see e.g.
Halliday, 1978, pp. 142–3) – what I will call the SOCIO-SEMIOTIC PROCESS (‘that
which is going on’) and the PHENOMENAL DOMAIN (‘subject matter’).12
Both are
relevant to the ideational aspect of semiotic systems, as can be seen in the
important distinction Kress and van Leeuwen (1996) draw between ‘narra-
tive representation’ (Chapter 2) and ‘conceptual representation’ (Chapter 3).
The SOCIO-SEMIOTIC PROCESS is ‘what’s going on’ in the Field – the nature of
the activity. Drawing on Ure’s (unpublished) work on a context-based regis-
ter typology, we can recognize eight distinct primary socio-semiotic process
types (see Matthiessen, Teruya and Wu, forthcoming), grouping them into
first-order and second-order processes.
(i) One of these is a first-order process in Halliday’s (1978, pp. 142–3)
sense – processes that we can now interpret as processes within social sys-
tems in the ordered typology of systems in Figure 2.2; this first-order pro-
cess is the social process of ‘doing’, such as teamwork in a fishing expedition
or in surgery in an operating theatre, and here language and other denota-
tive semiotic systems come in merely to facilitate the execution of this first-
order social process.
(ii) The other seven processes are second-order ones (again in Halliday’s
sense) – processes that we can now interpret as processes within semiotic
systems in the ordered typology of systems in Figure 2.2; they are inher-
ently semiotic processes (and so also social, of course) and operate in con-
texts that are constituted not only socially but also semiotically.
We can summarize the socio-semiotic processes of Field as follows:
● semiotic processes (semiotic processes constitutive of context):
❍ processes of expounding (general knowledge) – explaining/classifying;
❍ reporting (on sequences of particular events, or regions of places) –
recording (events)/surveying (places);
❍ recreating various aspects of socio-semiotic life (typically particular,
personal and imagined experiences) – narrative and/or dramatizing;
❍ sharing (typically particular, personal experiences and values);
❍ recommending (courses of action) – advising/exhorting (promoting);
❍ enabling (courses of action) – empowering/regulating;
❍ exploring (positions and values) – arguing/evaluating;
● social processes (social processes constitutive of context, semiotic pro-
cesses facilitating):
❍ doing (social action, with semiotic processes facilitating).

These different socio-semiotic process types put different ideational aspects
of denotative semiotic systems ‘at risk’ and also have different implications
for multisemiotic combinations. For instance, ‘expounding’ processes are
likely to mobilize ‘conceptual representations’ in Kress and van Leeuwen’s
(1996) account, and they may also mobilize ‘narrative representations’; but
‘reporting’ and ‘recreating’ processes are more likely to mobilize ‘narrative
representations’.
If we intersect the different SOCIO-SEMIOTIC PROCESS types with values along
the other contextual parameter within Field – the PHENOMENAL DOMAIN, we
can give examples of likely multisemiotic combinations (just as we did
for combinations of two types of channel within Mode, in Table 2.1), as
illustrated in Table 2.2 for written document (i.e. the upper left region of
Table 2.1) with language as the primary semiotic.
The examples given in Table 2.2 can, of course, be multiplied, but they
illustrate certain favoured combinations of values within socio-semiotic
process and values within phenomenal domain. For example
● expounding and −temporal sequence: taxonomic diagrams;
● reporting and +temporal sequence: timeline diagrams;
● recreating and +temporal sequence: story book illustrations;
● enabling and +temporal sequence: flowcharts.
An investigation of the conditions of multimodality and multisemiotic
systems raises various interesting questions – one key question being to
what extent different semiotic systems are general-purpose ones and to
what extent they are register-specific (in the sense, for example, of a register-
specific semantic system; see e.g. Halliday, 1973; Patten, 1988). It does seem
that visual semiotic systems tend to be much more register-specific than
language is – that is, they tend to have evolved within particular types of
situation as register-specific semiotic systems for operating within a certain
range of Field, Tenor and Mode settings. For example, the flowcharts con-
strue temporal sequences of events within some phenomenal domain, and
they are typically used in either enabling contexts (designing or enabling
a procedure, including algorithms) or in expounding ones (documenting
a flow of events), having originally been designed in the 1920s for use in
mechanical engineering. If we compare them with language, we can locate
the systemic paths used for construing temporal sequences of events by
means of clause complexes (cf. Halliday and Matthiessen, 2006, pp. 119–22,
364–5).
2.4.5 Semiotic systems and registerial range
Different settings of values within the Field, Tenor and Mode potential
define different cultural domains – different sub-systems of the context of

Table 2.2 The different SOCIO-SEMIOTIC PROCESS types and PHENOMENAL DOMAIN values for
written mode with examples of multisemiotic documents
Phenomenal domain
example: written
text with image
temporal
sequence
spatial
extension generality concreteness
expounding ϩ ϩ ϩ Ϫ chemistry textbook:
explanation with
diagram of sequences
of chemical reactions
Ϫ Ϫ ϩ Ϫ chemistry textbook:
classification of
elements with
display of periodic
table
reporting ϩ ϩ Ϫ ϩ newspaper: news
report of maritime
disaster with
sequence of photos
of rescue of
passengers
ϩ ϩ Ϫ ϩ history book:
historical recount
with time line
diagram
Ϫ ϩ Ϫ ϩ guide book: survey
of a region with
map showing major
natural features and
places of interest
recreating ϩ ϩ Ϫ ϩ story for adolescents:
narrative with
drawings of scenes
and characters
sharing (Ϯ) (Ϯ) Ϫ ϩ (personal) email
message with photos
doing Ϫ ϩ Ϫ ϩ shopping list with
drawing of product
to be purchased
recommending Ϫ Ϫ ϩ Ϫ newspaper: agony
aunt column
(without image, or
with image of ‘agony
aunt’ as authority
projecting advice)
enabling ϩ ϩ ϩ ϩ chemistry textbook:
procedure for lab
experiment with
flow chart
Continued

culture within which (denotative) semiotic systems operate13
(see Halliday,
2007 [1991], p. 275). These different cultural domains constitute the eco-
logical niches in which registers – different sub-systems of a semiotic sys-
tem – operate. Any semiotic system can thus be characterized in terms of
the range of registers that it embodies; in fact, it can be modelled as an
aggregate of those registers (cf. Matthiessen, 1993b). Some semiotic systems
embody a wide registerial range, whereas other semiotic systems embody a
narrow range. We can thus identify a cline extending from semiotic systems
that are special-purpose systems embodying a single register to semiotic
systems that are general-purpose ones embodying a wide – and open – range
of registers (see Figure 2.10).
As far as language is concerned, the outer poles of the cline ordering semi-
otic systems from ‘special-purpose’ to ‘general-purpose’ ones are defined
by protolanguages and standard languages. Protolanguages are very con-
strained in terms of their contexts of use – these contexts being instrumen-
tal, regulatory, interactional and personal, as mentioned in Section 2.4.3;
each context of use has its own little specialized meaning potential. In con-
trast, standard languages have huge registerial ranges, encompassing not
only the spoken registers of the home and the neighbourhood, but also
the extended range of written registers associated with the modern nation
states – registers of science and technology, on the one hand, and registers
of administration and control, on the other.
In a similar way, we can locate semiotic systems other than language along
the cline represented in Figure 2.10. The extent of the registerial range of
any semiotic system will be subject to considerations of what combinations
of Field, Tenor and Mode values they can operate across. For example, while
language can operate across the modal combinations set out in Table 2.1,
most other semiotic systems are more constrained in terms of ‘channel’: one
of the reasons why language has such a wide registerial range is precisely that
it has evolved a written mode alongside the earlier spoken mode (which can
in turn be transferred to contact by touch through Braille). It is possible to
Table 2.2 Continued
Phenomenal domain
example: written
text with image
temporal
sequence
spatial
extension generality concreteness
ϩ ϩ Ϫ ϩ guide book: walking
tour (topographic
procedure) with map
showing route
exploring Ϫ Ϫ Ϫ Ϫ newspaper: book
review with image of
cover and/or author

think of other examples such as notation systems for movement, including
dance (e.g. Laban Movement Analysis) and for music; but these do not seem
to extend the registerial ranges of dance and music beyond transcription –
beyond the fact that they can be transcribed in a notation system, just as
spoken language can be transcribed using writing. Here it is significant that
(unlike notation schemes for music and dance) writing did not evolve as a
notation system for transcribing a different mode of itself – spoken language;
rather it evolved out of drawing in contexts of use other than those associ-
ated with the existing registers of spoken language. Writing encroached on
the ‘registers’ of drawing – registers of book-keeping and trade.
2.5 Conclusion
This chapter has touched on some central issues in the modelling of multi-
semiotic systems operating synergistically in a unified context, approach-
ing these issues in terms of systemic functional theory – a theory that has
informed some of the most central contributions to our understanding of
multimodality and of multisemiotic systems.
The chapter began by locating the phenomena under consideration
within a holistic conception of an ordered typology of systems of operating
within phenomenal realms of increasing complexity – physical, biological,
Kinds of semiotic system
Special-purpose
semiotic systems
Protolanguage
Language:
vernacular
dialect Language:
standard
Range of registers
{variation in content
plane according to
context of use}
General-purpose
semiotic systems
Figure 2.10 Kinds of semiotic system in relation to registerial range – from special-
purpose systems to general-purpose ones

social and semiotic systems. While multimodality and multisemiosis are
semiotic phenomena in the first instance, they also have social, biological
and physical implications precisely because the typology of systems is an
ordered one – ordered in increasing complexity: semiotic systems are also
social, and social systems are also biological, and biological systems are also
physical. If we confine ourselves to multimodality and multisemiosis in the
human species (but see e.g. Benson et al., 2002; Benson and Greaves, 2005;
and cf. Matthiessen, 2004, for an evolutionary perspective), we can explore
semiotic systems in terms of the conditions – both enabling and constrain-
ing ones – inherent in human societies (see e.g. Johnson and Earle, 2000)
and human bodies (see e.g. Thibault, 2004). For example, we can investi-
gate the affordances inherent in the human body in terms of the expres-
sive resources of different semiotic systems; and we can also investigate
the affordances of extending the human body as a resource for expression
inherent in the (socially constructed) biological and physical environment
of the human body – our ‘habitat’.
After locating the phenomena under investigation within a holistic con-
ception of systems of different kinds, a cline of semiotic integration was
sketched, extending it from maximal integration to minimal integration.
In the case of maximal integration, different ‘modalities’ operate on the
expression plane within one and the same semiotic system, being integrated
within the content plane of that system. An example of this case is the
account of intonation within systemic functional linguistics, pioneered by
Halliday in the 1960s and discussed above. Here the integration is possible
even within the lower of the two content strata – the stratum of lexico-
grammar; the integration takes place within the systemic (paradigmatic)
axis of organization and the ‘modality’ of intonation is handled by realiza-
tion statements associated with terms in systems. There may of course be
tensions between intonation and (segmental) structure, as in the case of an
example such as // -2 ^ you / have a /photograph of / this girl // (from Halliday,
1970), where the modal structure is that of a ‘declarative’ clause – Subject:
you ^ Finite: have, but the tone is that prototypically associated with a ‘yes/
no interrogative’ clause – tone 2, phonetically a rising tone. However, such
tensions are handled without difficulty in the description in terms of deli-
cacy: while the ‘unmarked’ declarative key is realized by tone 1 (phonet-
ically a falling tone), there are also four marked declarative keys, including a
querying one realized by tone 2 (as shown in Figure 2.4 in Section 2.3).
In the case of minimal semiotic integration, independent (denotative)
semiotic systems operate in parallel within one and the same context (con-
notative semiotic system), and the coordination of semiotic processes within
these parallel semiotic systems is purely a matter of context. An example of
this case is the account of language and music in a folk ballad offered by
Steiner (1988). Here one challenge is precisely to account for how the dif-
ferent semiotic systems complement one another in creating meaning and

Ventola, guijjaro 2009 the world told and the world shown multisemiotic issues

Ventola, guijjaro 2009 the world told and the world shown multisemiotic issues

Recommended

Recommended

More Related Content

Similar to Ventola, guijjaro 2009 the world told and the world shown multisemiotic issues

Similar to Ventola, guijjaro 2009 the world told and the world shown multisemiotic issues (20)

Recently uploaded

Recently uploaded (20)

Ventola, guijjaro 2009 the world told and the world shown multisemiotic issues