Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Engineering Device-spanning, Multimodal Web Applications using a Model-based Design Approach

499 Aufrufe

Veröffentlicht am

Nowadays the web is an ubiquitously available source of information that can be accessed through a broad range of devices, such as smart phones, tablets and notebooks. Although web applications can be used through several devices, they are controlled and designed for a one-to-one connection type of interaction, which prevents device-spanning multi-modal interactions.
We propose a model-based run-time framework to design and execute multi-modal interfaces for the web. Different to a model-based design that implements reification, a process to derive concrete models from abstract ones by transformation, we design interactors that keep all design models alive at run-time. Interactors are based on finite state machines that can be inspected and manipulated at run-time and are synchronized over different devices and modalities using mappings. We show the expressiveness of state charts for modeling interactions, interaction resources, and interaction paradigms.
We proof our approach by checking its conformance against common requirements for multimodal frameworks, classify it based on characteristics identified by others, and present initial results of a performance analysis.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Engineering Device-spanning, Multimodal Web Applications using a Model-based Design Approach

  1. 1. Engineering Device-spanning, Multimodal Web Applications using a Model-based Design Approach Sebastian Feuerstack and Ednaldo PizzolatoDr.-Ing. Sebastian FeuerstackUniversidade Federal de São CarlosDepartamento de Computacão 16. Oktober 2012 Sebastian Feuerstack I Sebastian@Feuerstack.org
  2. 2. Outline• Introduction / Example• Related Work• Models (Interactors, Mappings)• Evaluation16. Oktober 2012 2
  3. 3. Basic Question How to model and run multimodal web applications ?16. Oktober 2012 3
  4. 4. Multimodal Interaction Design Augmented Drag-and-Drop • Furniture shop – Shopping cart – Drag and Drop – Into my environment16. Oktober 2012 4
  5. 5. Drag and Drop using gestures 16. Oktober 2012 5
  6. 6. Why modeling? • Visual language that – is declarative and precise to discuss & execute – Is located between a tool and source code – Supports different forms of interaction and flexible prototyping of multimodal interfaces16. Oktober 2012 6
  7. 7. What is State of the Art ?1. Task AUI CUI FUI CUI FUI2. Widget W1 W2 Alg. A1 D1 Driver 16. Oktober 2012 7
  8. 8. How to model? AUI Widget Mapping CUI CUI W W Modality D1• Models – Interactors (Abstract & Concrete Media, Mode) • Static: class diagram, Behavior: state-chart – Mappings • Custom Notation 16. Oktober 2012 8
  9. 9. Abstract UI Model Abstract Interactor Model (AIM) Interactor - name, states previous next AIContext - text context refers AIO AIIN 1 AIOUT 0..n children AIINDiscrete AIINContinuous parent AIOUTDiscrete AIOUTContinuous - data, min, max - data, min, max AIReference AIINRange AIContainer - text - min_data, max_data AISinglePresence AISingleChoice AICommand AIChoiceElement - active_child AISingleChoiceElement AIMultiChoiceElement AIMultiChoice16. Oktober 2012 9
  10. 10. Abstract Behavior Model AUI:AIO:AIChoiceElement: AISingleChoiceElement Presenting drop listed initialized next||prev||parent focus dragging defocus /aio=find(act); organize aio.focus drag focused organized present Selection H organize unchosen [in(focused)] choose suspend unchoose / aios=find(parent.childs.chosen); aios.all.unchoose chosen suspended16. Oktober 2012 10
  11. 11. Mode Model Gesture Interactor IR:IN:HandGesture:PointingSelect BothHands NoLeftHand NoRightHand left_hand_appeared right_hand_appeared left_hand_disappeared right_hand_disappeared Left Hand Right Hand Pointing Command stopped initialized stop select move confirm moving confirmed selected move select left_hand_appeared right_hand_appeared NoHands right_hand_disappeared left_hand_disappeared16. Oktober 2012 11
  12. 12. Mappings Application-level Mapping h = Gesture.LeftHand.Pointing.stopped cio.highlight S R cio = CUIControl.find_from_coordinates(h.x,h.y) Sound.click Synchronisation Mapping cio=Interactor.CIO.highlighted S aio.focus NOT aio = cio.name:Interactor.AIO.focused Paradigm-level MappingRightHand.Command.confirmed S RightHand.Command.selected C dst.drop(item)item=AIChoiceElement.focused dst=AISingleChoice.focused src=item.parent item.drag src.drop(item) fail 16. Oktober 2012 12
  13. 13. Evaluation1. W3C Multimodal Framework Requirements – Structuration Mechanism – Explicit Control Structures – Extensible Event Definition – Data Modeling – Reusable Components2. Framework Classification Survey (Dumas 2009) – 8 frameworks, two available, one under active development – criteria: • finite-state machine processing • no frame-based or symbolical-statistical fusion • high-level programming features 16. Oktober 2012 13
  14. 14. Evaluation 23. Processing Performance Evaluation – Most critical part: server-sided processing of continuous user actions – Pointing Movements (by Hand or Mouse) • Automated test, 24 interactors, Core2Duo P8600, 250 repetitions16. Oktober 2012 14
  15. 15. Conclusions & Future Work What’s the advantage?• Detailed Modeling of Multimodal Interactions including behavior• Existing tools• No gap between design- and runtime 16. Oktober 2012 15
  16. 16. Whats next? MINT-Composer - A Toolchain for the Model-based Specification of Post-WIMP Interactors.Focus on: – Fusion – Paradigm Design – Formalization – Tools – W3C Standardization http://www.multi-access.de All papers, Videos, MINT Framework as Open Source 16. Oktober 2012 Design of Multimodal Interaction 16
  17. 17. 16. Oktober 2012 17