Model-driven engineering of multimodal user interfaces

Model-Driven Engineering of User Interfaces for Multimodal Web Applications Jean Vanderdonckt Université catholique de Louvain (UCL) Louvain School of Management (LSM) Information Systems Unit (ISYS) Belgian Laboratory of Computer-Human Interaction (BCHI) http://www.isys.ucl.ac.be/bchi

What is a multimodal user interface? ,[object Object],[object Object],[object Object]

What is a multimodal user interface? ,[object Object],[Tison,2003]

What is a multimodal user interface? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[Bodnar,2004]

What is a multimodal user interface? ,[object Object],[object Object],[object Object],[object Object],[Bellik,2002]

Why are multimodal user interfaces important? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[IntuiLab,2006]

Why are multimodal user interfaces challenging? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Concepts: The CARE Properties ,[object Object],[object Object],[object Object],[object Object],[object Object],Put that there [Bolt,1979]

What are the appropriate abstractions? ,[object Object],[object Object],[object Object],[object Object]

Why a design space? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Multimodal Web Applications ,[object Object],[object Object],[object Object],[object Object],[W3C,2006]

Solution space ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[Stanciulescu et al.,2005]

Design options for graphical web UIs Design options for graphical UIs Sub-task presentation Sub-task navigation Concretization of navigation & control Navigation concretization Control concretization

Design options for graphical web UIs ,[object Object],Sub-task presentation separated combined one by one combined box tabbed dialog box float window many at once float window all in one areas with separators group boxes bulleted list numbered list Abstract level Concrete level

Sub-task presentation – example ,[object Object],[object Object],T11 T12 T13 T1 AC11 AC12 AC13 AC1 T11 T12 T13 T1 AC11 AC12 AC13 NAC LHS RHS NAC LHS RHS

Design options for graphical web UIs ,[object Object],Navigation concretization [Stanciulescu,2006] placement cardinality local global simple multiple

Sub-task navigation - example ,[object Object],[object Object],AC1 T11 T12 T13 T1 AC11 AC12 AC13 AIC111 AIC121 AIC131 AC1 T11 T12 T13 T1 AC11 AC12 AC13 AIC11 NAC LHS RHS NAC LHS RHS

Design options for graphical web UIs ,[object Object],Sub-task navigation asynchronous sequential (synchronous)

Design options for graphical web UIs ,[object Object],Concretization of navigation & control separated combined

Design options for graphical web UIs ,[object Object],Control concretizations placement cardinality local global multiple simple

Design options for multimodal web UIs ,[object Object],[object Object],[object Object],Design options for multimodal UIs [Scapin & Bastien,1997] [Vanderdonckt,1995] Prompt Grouping / Distinction of items Immediate feedback Guidance

Design options for multimodal web UIs ,[object Object],[object Object],[object Object],[object Object]

Design options for multimodal web UIs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Design options for multimodal web UIs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Multimodal example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[Stanciulescu,2006] Prompt: multimodal Please say your name (vocal + graphical) 1 Input: multimodal Juan Gonzalez 2 (vocal + graphical) Juan Gonzalez Guidance for input: iconic Guidance for feedback: iconic Feedback:multimodal Your answer was: Juan Gonzalez 3 (graphical + vocal)

Vocal Concrete Interaction Objects ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Vocal Concrete Interaction Objects ,[object Object]

Multimodal Concrete Interaction Objects ,[object Object],outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance groupBox + radioButtons vocalMenu + vocalItems + vocalInput groupBox + radioButtons + vocalMenu + vocalItems + vocalInput + guidance Graphical Vocal Multimodal

Concrete UI Relationships ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Tool support ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Multimodal Concrete Interaction Objects ,[object Object],[object Object]

Specification of transformation ,[object Object],[object Object],[object Object]

How transformation rules work… ,[object Object],[object Object],[object Object],[Limbourg,2004] Transformation rule Initial Model Resulting Model

UsiXML syntax of transformation rules NAC NAC RHS mappings

Transformation rule NAC: LHS: RHS: Multimodal prompt Guidance for input Multimodal input Guidance for output Vocal feedback

Conclusion ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Conclusion ,[object Object],[object Object],[object Object],[object Object],[object Object],[Stanciulescu & Vanderdonckt,2006] [Beaudouin-Lafon,2004]

Future work ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

References ,[object Object],[object Object],[object Object]

Thank you very much for your attention For more information and downloading, http://www.isys.ucl.ac.be/bchi http://www.usixml.org User Interface eXtensible Markup Language http://www.similar.cc European network on Multimodal UIs Special thanks to all members of the team!

Model-driven engineering of multimodal user interfaces

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (19)

Ähnlich wie Model-driven engineering of multimodal user interfaces

Ähnlich wie Model-driven engineering of multimodal user interfaces (20)

Mehr von Jean Vanderdonckt

Mehr von Jean Vanderdonckt (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Model-driven engineering of multimodal user interfaces