Nowadays the web is an ubiquitously available source of information that can be accessed through a broad range of devices, such as smart phones, tablets and notebooks. Although web applications can be used through several devices, they are controlled and designed for a one-to-one connection type of interaction, which prevents device-spanning multi-modal interactions. We propose a model-based run-time framework to design and execute multi-modal interfaces for the web. Different to a model-based design that implements reification, a process to derive concrete models from abstract ones by transformation, we design interactors that keep all design models alive at run-time. Interactors are based on finite state machines that can be inspected and manipulated at run-time and are synchronized over different devices and modalities using mappings. We show the expressiveness of state charts for modeling interactions, interaction resources, and interaction paradigms. We proof our approach by checking its conformance against common requirements for multimodal frameworks, classify it based on characteristics identified by others, and present initial results of a performance analysis.

1. Engineering Device-spanning, Multimodal
Web Applications using a Model-based
Design Approach
Sebastian Feuerstack and Ednaldo Pizzolato
Dr.-Ing. Sebastian Feuerstack
Universidade Federal de São Carlos
Departamento de Computacão
16. Oktober 2012 Sebastian Feuerstack I Sebastian@Feuerstack.org

2. Outline
• Introduction / Example
• Related Work
• Models (Interactors, Mappings)
• Evaluation
16. Oktober 2012 2

3. Basic Question
How to model and run
multimodal web applications ?
16. Oktober 2012 3

4. Multimodal Interaction Design
Augmented Drag-and-Drop
• Furniture shop
– Shopping cart
– Drag and Drop
– Into my environment
16. Oktober 2012 4

5. Drag and Drop using gestures
16. Oktober 2012 5

6. Why modeling?
• Visual language that
– is declarative and precise to discuss &
execute
– Is located between a tool and source code
– Supports different forms of interaction and
flexible prototyping of multimodal interfaces
16. Oktober 2012 6

7. What is State of the Art ?
1. Task AUI CUI FUI
CUI FUI
2. Widget
W1 W2
Alg.
A1 D1
Driver
16. Oktober 2012 7

8. How to model?
AUI
Widget Mapping CUI
CUI
W W
Modality
D1
• Models
– Interactors (Abstract & Concrete Media, Mode)
• Static: class diagram, Behavior: state-chart
– Mappings
• Custom Notation
16. Oktober 2012 8

9. Abstract UI Model
Abstract Interactor Model (AIM) Interactor
- name, states
previous
next AIContext
- text
context
refers AIO
AIIN 1
AIOUT
0..n
children
AIINDiscrete AIINContinuous parent AIOUTDiscrete AIOUTContinuous
- data, min, max - data, min, max
AIReference AIINRange AIContainer
- text - min_data, max_data
AISinglePresence AISingleChoice
AICommand AIChoiceElement - active_child
AISingleChoiceElement AIMultiChoiceElement AIMultiChoice
16. Oktober 2012 9

10. Abstract Behavior Model
AUI:AIO:AIChoiceElement:
AISingleChoiceElement
Presenting
drop listed
initialized next||prev||parent
focus
dragging defocus /aio=find(act);
organize aio.focus
drag focused
organized
present Selection
H
organize unchosen
[in(focused)] choose
suspend unchoose / aios=find(parent.childs.chosen);
aios.all.unchoose
chosen
suspended
16. Oktober 2012 10

11. Mode Model
Gesture Interactor
IR:IN:HandGesture:PointingSelect
BothHands
NoLeftHand
NoRightHand
left_hand_appeared right_hand_appeared
left_hand_disappeared right_hand_disappeared
Left Hand
Right Hand
Pointing Command
stopped
initialized
stop select
move
confirm
moving confirmed selected
move
select
left_hand_appeared right_hand_appeared
NoHands
right_hand_disappeared left_hand_disappeared
16. Oktober 2012 11

12. Mappings
Application-level Mapping
h = Gesture.LeftHand.Pointing.stopped cio.highlight
S R
cio = CUIControl.find_from_coordinates(h.x,h.y) Sound.click
Synchronisation Mapping
cio=Interactor.CIO.highlighted
S aio.focus
NOT aio = cio.name:Interactor.AIO.focused
Paradigm-level Mapping
RightHand.Command.confirmed S RightHand.Command.selected C dst.drop(item)
item=AIChoiceElement.focused dst=AISingleChoice.focused
src=item.parent item.drag
src.drop(item) fail
16. Oktober 2012 12

13. Evaluation
1. W3C Multimodal Framework Requirements
– Structuration Mechanism
– Explicit Control Structures
– Extensible Event Definition
– Data Modeling
– Reusable Components
2. Framework Classification Survey (Dumas 2009)
– 8 frameworks, two available, one under active
development
– criteria:
• finite-state machine processing
• no frame-based or symbolical-statistical fusion
• high-level programming features
16. Oktober 2012 13

14. Evaluation 2
3. Processing Performance Evaluation
– Most critical part:
server-sided processing of continuous user actions
– Pointing Movements (by Hand or Mouse)
• Automated test, 24 interactors, Core2Duo P8600,
250 repetitions
16. Oktober 2012 14

15. Conclusions & Future Work
What’s the advantage?
• Detailed Modeling of
Multimodal
Interactions
including behavior
• Existing tools
• No gap between
design- and runtime
16. Oktober 2012 15

16. Whats next?
MINT-Composer - A Toolchain for
the Model-based Specification of
Post-WIMP Interactors.
Focus on:
– Fusion
– Paradigm Design
– Formalization
– Tools
– W3C Standardization
http://www.multi-access.de
All papers, Videos, MINT Framework as Open Source
16. Oktober 2012 Design of Multimodal Interaction 16

17. 16. Oktober 2012 17