1. Multimodal Semiotics.
The Importance of Visual Communication and
Interaction.
Dienstag, 10. August 2010

2. Outline
1. Context
2. Thesis
3. Reading Multimodal Texts
1. Modalities
2. Approaches of Analysis
4. Case studies
1. BooneOakley
2. The Guardian: War Logs
3. Driving Game (NYT)
4. Alice in wonderland
5. Sources
Dienstag, 10. August 2010

3. Current tendencies influencing digital content
production
Rethinking Browser and
Data Storage
Desktop-Application-Flow
of Web Services (AJAX, Application Web,
HTML5) Multimodal Content
Supply, Policies and and Interactions, APIs
Control (App-Store)
Advertising (iAds) and
monetarizing (Flattr)
Dienstag, 10. August 2010

4. Thesis
The production of content and information
architecture is not exhausted in structuring
information.
The meaning of texts is determined by more
than verbal, visual and interactive elements.
Visuality ≠ Decoration
Digital Content > Visuality + Verbality
Dienstag, 10. August 2010

5. Modalities of (current) Online and Mobile Media
VISION: Fonts, Images, Colors, Layout, Brightness...
AUDITION: Sound, Speech, Music ...
TACTITION: Vibrating, Pressure, Gestures
EQUILIBRIOCEPTION: Balance
Dienstag, 10. August 2010

6. Meaning of visual texts
tied to the context of the recipient
production of knowledge
grammar is defined by culture and social life
heavily uncodified and often subtile
analyzed in a bunch disciplines
Dienstag, 10. August 2010

7. Concepts
Discouse
“a set of statements that articulate a particular way of thinking, feeling and being in
the world” (Cranny-Francis, 2005, P. 41)
Genre
“it enables us to understand the kinds of meanings users do generate from them
[the texts]” (Cranny-Francis, 2005, P. 41)
Intertexuality
“enables us to understand how the web site relates to other visual genres - not
simply formal, but also in terms of (potential) meanings.” (Cranny-Francis, 2005, P.
43)
Dienstag, 10. August 2010

8. CASE STUDIES
Dienstag, 10. August 2010

9. BooneOakley
http://www.booneoakley.com
Dienstag, 10. August 2010

10. BooneOakley
multiple
drawing style + childish storytelling
message: min. efforts, max. effect
navigating through timeline
embedding and other youtube-features
Dienstag, 10. August 2010

11. Afganistan War Logs
http://www.guardian.co.uk/world/the-war-logs
Dienstag, 10. August 2010

12. War Logs
Access to Source-Data / Documentation
Interactive Mapping
Timeline
Datajournalism: Presentation of Statistics
Meaning of Statistics: Non-emotional, rational
approach to war events. Subtile Opinionmakings.
Dienstag, 10. August 2010

13. Ludic Journalism/Storytelling
(1) Driven to Distruction (NJT)
http://www.nytimes.com/interactive/2009/07/19/technology/20090719-driving-game.html
(2) Alice in Wonderland (iPad-App)
http://www.youtube.com/watch?v=gew68Qj5kxw
Dienstag, 10. August 2010

14. Sources
Cranny-Francis, A. (2005): Multimedia: texts and contexts. London: SAGE.
Foucault, M. (1969) The Archaeology of Knowledge. London: Routledge.
Halliday, M. A. K. (1978) Language as social semiotics: The social interpretation of language and
meaning. Maryland: University Park Press.
Kress, G., and T. van Leeuwen (1996) Reading Images: The Grammar of Visual Design, London:
Routledge.
Kress, G., and T. van Leeuwen (2001) Mulitmodal Discourse: The Modes and Media of Contemporary
Communication. London: Arnold.
Mitchell, W. J. T. (2005) What do Pictures Want? The Lives and Loves of Images. Chicago: University of
Chicago Press.
Morville, P. / Rosenfeld, L. (2006) Information Architechture for the World Wide Web 3rd Ed., Sebastopol:
O’Reilly.
Dienstag, 10. August 2010