1. A Distributed Staged Architecture for
Multimodal Applications
Alessandro Costa Pereira, Falk Hartmann,
Kay Kadner
SAP Research/TU Dresden

2. Agenda
 The Project SNOW
 Prerequisites
 Enabling Multimodal Access (MMI-F)
 The SNOW Architecture
 The Implementation
 Lessons Learned/Research Questions
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications
2/13

3. Project SNOW
3/13
 Services for Nomadic Workers
 EU FP6 Project, 2004-2006, 6 Partners
 www.snow-project.org
A large-scale industrial
diffusion of multimodal mobile documentation
for maintenance operations.
 How to author multimodal mobile maintenance documentation?
 How to exploit this documentation through robust interaction
modalities?
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

4. 4/13
Project SNOW
From paper …
a
ob C
J
rd
Maintenance
procedures
and related
information
…to wearable electronic documentation
 Airplane maintenance worker uses paper-based documentation
 Unanticipated problems cause high expenses
 Multimodality essential
 Increased productivity for future maintenance
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

5. Prerequisites
5/13
 Multimodality is the use of multiple types of interaction between a
user and a computer.
 W3C Activity "Multimodal Interaction"
 "Multimodal Interaction Framework" (MMI-F)
 "Extensible MultiModal Annotation markup language" (EMMA)
 A role model is "the description of a (possibly) infinite set of object
collaborations using role types".
 Riehle/Gross "Role model based framework design and integration"
 A staged architecture is "a sequence of n subarchitectures, in
which stage produces the (data for the) next stage"
 Aßmann "Invasive Software Composition"
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

6. Enabling Multimodal Access
6/13
W3C Multimodal Interaction Framework (MMI-F)
The multimodal interaction framework is not an
architecture .
The multimodal interaction framework is a level of
abstraction above an architecture. […]
High-Level View
OutputComponents
Input Components
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

7. Enabling Multimodal Access
MMI-F is a role model
 Short illustration of the core of the MMI-F
 Applicable to the software architecture of your choice
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications
7/13

8. The SNOW Architecture
Architectural Requirements
 Client-/Server architecture
 Domain independence
 Device independence (⇐ multimodality)
 (Smallest) End-user device: PDA
 No speech-/gesture-recognition, no speech synthesis
 Available modalities depend on network connection
 Reuse standard software on PDA (Browser)
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications
8/13

9. 9/13
The SNOW Architecture
Staged Architecture
Browser
Client
Server
Device-specific
Adaptation
Manager
Dialog
Manager
Device-independent,
Domain-independent
Domain-independent
Documentation
Application
Domain-specific
Procedures
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

10. 10/13
The SNOW Architecture
Keyboard
Control
Integration
Manager
Speaker
Control
Camera
Control
Microphone
Control
Text to Speech
Gesture
Recognition
Speech
Recognition
Browser
Client
Server
Adaptation
Manager
Dialog
Manager
Procedures
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications
Rendering
Interpretation
Styling
Integration
Documentation
Application
Recognition
Generation
Interaction Manager

11. The Implementation
11/13
Staged Architecture → XML transformation pipeline
 Problem 1: Implementation effort
 More components
 Use of a common base technology in the components
 Special template engine with bypassing (XTL Engine)
 Problem 2: Performance penalty
 Several parse/serialize processes
 Mitigated using StAX for XML processing
 Compensated by deployment options (esp. within development)
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

12. Lessons Learned/Research Questions
12/13
Lessons Learned
 Mapping the MMI-F to a concrete architecture can be difficult
 Not very suitable for client/server architectures
 Improvement of the MMI-F (fission role)
 Staged Architecture can enable domain independence
 Evaluated the architecture in a medical scenario (WearIT@Work)
 Only the Documentation Application needs to be substituted
 But: there is no best-suited architecture for multimodality
Research Questions
 How to generically apply the MMI-F role model?
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

13. 13/13
Questions
WLAN Access
Point
“Server”
(Text-to-speech,
Speech Recognition,
Gesture recognition)
Mobile Client
(PDA w/
camera and
headset)
…Tablet PC based demonstrator now also available
© SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications