MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP Interactors

Sebastian Feuerstack
Sebastian FeuerstackPostDoc um OFFIS - Institute for Information Technology
MINT-Composer – A Toolchain for the Model-
     based Specification of Post-WIMP
               Interactors
                             Sebastian Feuerstack


Dr.-Ing. Sebastian Feuerstack
Universidade Federal de São Carlos
Departamento de Computacão

  18. Oktober 2012                           Sebastian Feuerstack I Sebastian@Feuerstack.org
Basic Question


                   How to model and run
                   multimodal Post-WIMP
                       interactions?

18. Oktober 2012                          2
Post-WIMP Interacton

• WIMP interaction style:
    Window, Icon, Menu, Pointing device

• Post-WIMP Interfaces ”…attempt to appear
  more natural and less obtrusive. Ideally this
  kind of post-WIMP interaction feels as
  familiar and fluid as using simple nondigital
  artifacts” [Jetter 2012]

 18. Oktober 2012                            3
Interactive music sheet




18. Oktober 2012          4
Why modeling?
   • Visual language that
         – is declarative and precise to discuss &
           execute
         – Is located between a tool and source code
         – Supports different forms of interaction and
           flexible prototyping of multimodal interfaces

18. Oktober 2012                                      5
How to model?
  AUI
              Widget              Mapping             CUI
                                                     CUI
                                                           W   W

                                  Modality
                                          D1
• Models
    – Interactors (Abstract & Concrete Media, Mode)
          • Static: class diagram, Behavior: state-chart
    – Mappings
          • Custom Notation

 18. Oktober 2012                                              6
Development Process




18. Oktober 2012      7
Interactor Modeling




 18. Oktober 2012     8
Mode Modeling
          IR:Head

                      NoHead
       discover
                                      lost
                  Head Captured

      right_tilted                 center
                           right
                                        centered
                    left
      left_tilted             center




18. Oktober 2012                                   9
Demonstration




                                                 Home Network
                                       MINT-                  Head
                                      Platform             Recognition
                             App
                           Download


                   WebServer          Home Gateway        End-Device
                                       WebServer          Webbrowser


18. Oktober 2012                                        The Augmented “Drag-and-Drop   10
Conclusions & Future Work


                What’s the advantage?
• Detailed Modeling of
  Multimodal
  Interactions
  including behavior
• Existing tools
• No gap between
  design- and runtime
 18. Oktober 2012                       11
Whats next?



Focus on:
      –   Fusion
      –   Paradigm Design
      –   Formalization
      –   Tools
      –   W3C Standardization


                    http://www.multi-access.de
      All papers, Videos, MINT Framework as Open Source

 18. Oktober 2012           Design of Multimodal Interaction   12
18. Oktober 2012   13
1 von 13

Recomendados

hcid2011 - Gesture Based Interfaces: Jacques chueke (HCID, City University L... von
hcid2011 -  Gesture Based Interfaces: Jacques chueke (HCID, City University L...hcid2011 -  Gesture Based Interfaces: Jacques chueke (HCID, City University L...
hcid2011 - Gesture Based Interfaces: Jacques chueke (HCID, City University L...City University London
3.8K views17 Folien
Mobileappdev fah von
Mobileappdev fahMobileappdev fah
Mobileappdev fahAlexandra Instituttet
515 views19 Folien
Creating a corporate intranet portal and enable the evolution of the digital ... von
Creating a corporate intranet portal and enable the evolution of the digital ...Creating a corporate intranet portal and enable the evolution of the digital ...
Creating a corporate intranet portal and enable the evolution of the digital ...Thomas Maeder
4.9K views21 Folien
From Idea to Web - Creating Linked Data Apps von
From Idea to Web - Creating Linked Data AppsFrom Idea to Web - Creating Linked Data Apps
From Idea to Web - Creating Linked Data AppsBenjamin Nowack
1.6K views26 Folien
Application-oriented IT Research af Frank Allan Hansen, Alexandra Instituttet von
Application-oriented IT Research af Frank Allan Hansen, Alexandra InstituttetApplication-oriented IT Research af Frank Allan Hansen, Alexandra Instituttet
Application-oriented IT Research af Frank Allan Hansen, Alexandra InstituttetInfinIT - Innovationsnetværket for it
594 views19 Folien
Towards application development for the internet of things updated von
Towards application development for the internet of things  updatedTowards application development for the internet of things  updated
Towards application development for the internet of things updatedPankesh Patel
1.2K views35 Folien

Más contenido relacionado

Similar a MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP Interactors

Publicis Mojito von
Publicis MojitoPublicis Mojito
Publicis Mojitoisaoyagi
350 views19 Folien
Adobe Key Trends in Technical Communication at MEGAComm von
Adobe Key Trends in Technical Communication at MEGACommAdobe Key Trends in Technical Communication at MEGAComm
Adobe Key Trends in Technical Communication at MEGACommPaula Stern
1.2K views21 Folien
Drupal + HTML5 + CSS3 + JS = Rich Internet Application von
Drupal + HTML5 + CSS3 + JS = Rich Internet ApplicationDrupal + HTML5 + CSS3 + JS = Rich Internet Application
Drupal + HTML5 + CSS3 + JS = Rich Internet ApplicationDrupalcampAtlanta2012
733 views35 Folien
Drupal + HTML5 + CSS3 + JS = Rich Internet Application von
Drupal + HTML5 + CSS3 + JS = Rich Internet ApplicationDrupal + HTML5 + CSS3 + JS = Rich Internet Application
Drupal + HTML5 + CSS3 + JS = Rich Internet ApplicationAppnovation Technologies
3.1K views35 Folien
Yocto Project - OSCON 7-17-2012 von
Yocto Project - OSCON 7-17-2012Yocto Project - OSCON 7-17-2012
Yocto Project - OSCON 7-17-2012Jeffrey Osier-Mixon
4.2K views28 Folien
DIY Robotics aOG MIT 26-10-2012 von
DIY Robotics aOG MIT 26-10-2012DIY Robotics aOG MIT 26-10-2012
DIY Robotics aOG MIT 26-10-2012Cesar Vandevelde
650 views23 Folien

Similar a MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP Interactors(20)

Publicis Mojito von isaoyagi
Publicis MojitoPublicis Mojito
Publicis Mojito
isaoyagi350 views
Adobe Key Trends in Technical Communication at MEGAComm von Paula Stern
Adobe Key Trends in Technical Communication at MEGACommAdobe Key Trends in Technical Communication at MEGAComm
Adobe Key Trends in Technical Communication at MEGAComm
Paula Stern1.2K views
iRail Summer of Code von Yeri Tiete
iRail Summer of CodeiRail Summer of Code
iRail Summer of Code
Yeri Tiete356 views
Userforum Process Visualization von duncanseidler
Userforum Process VisualizationUserforum Process Visualization
Userforum Process Visualization
duncanseidler406 views
Google Developer Group(GDG) DevFest Event 2012 Android talk von Imam Raza
Google Developer Group(GDG) DevFest Event 2012 Android talkGoogle Developer Group(GDG) DevFest Event 2012 Android talk
Google Developer Group(GDG) DevFest Event 2012 Android talk
Imam Raza1.4K views
Nagios Conference 2012 - Nathan Vonnahme - Monitoring the User Experience von Nagios
Nagios Conference 2012 - Nathan Vonnahme - Monitoring the User ExperienceNagios Conference 2012 - Nathan Vonnahme - Monitoring the User Experience
Nagios Conference 2012 - Nathan Vonnahme - Monitoring the User Experience
Nagios1.8K views
DITA Proof-of-Concept Publishing System von Matso Limtiaco
DITA Proof-of-Concept Publishing SystemDITA Proof-of-Concept Publishing System
DITA Proof-of-Concept Publishing System
Matso Limtiaco767 views
Enterprise Social Networking with Alfresco 4 von rivetlogic
Enterprise Social Networking with Alfresco 4Enterprise Social Networking with Alfresco 4
Enterprise Social Networking with Alfresco 4
rivetlogic1.7K views
A Site for All Eyes: Considerations for Responsive Design von Dori Kelner
A Site for All Eyes: Considerations for Responsive DesignA Site for All Eyes: Considerations for Responsive Design
A Site for All Eyes: Considerations for Responsive Design
Dori Kelner795 views
SEMLIB Final Conference | IN2 presentation von SemLib Project
SEMLIB Final Conference | IN2 presentationSEMLIB Final Conference | IN2 presentation
SEMLIB Final Conference | IN2 presentation
SemLib Project373 views
IPTC Rights Working Group Autumn 2012 von Stuart Myles
IPTC Rights Working Group Autumn 2012IPTC Rights Working Group Autumn 2012
IPTC Rights Working Group Autumn 2012
Stuart Myles777 views
Multichannel User Interfaces von Icinetic
Multichannel User InterfacesMultichannel User Interfaces
Multichannel User Interfaces
Icinetic7.4K views

Más de Sebastian Feuerstack

Revealing Differences in Designer‘s and Users‘Perspectives von
Revealing Differences in Designer‘s and Users‘PerspectivesRevealing Differences in Designer‘s and Users‘Perspectives
Revealing Differences in Designer‘s and Users‘PerspectivesSebastian Feuerstack
364 views14 Folien
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiT von
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiTComparing the Multimodal Interaction Technique Design of MINT with NiMMiT
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiTSebastian Feuerstack
801 views18 Folien
Engineering Device-spanning, Multimodal Web Applications using a Model-based ... von
Engineering Device-spanning, Multimodal Web Applications using a Model-based ...Engineering Device-spanning, Multimodal Web Applications using a Model-based ...
Engineering Device-spanning, Multimodal Web Applications using a Model-based ...Sebastian Feuerstack
578 views17 Folien
Designing and Executing Multimodal Interfaces for the Web based on State Char... von
Designing and Executing Multimodal Interfaces for the Web based on State Char...Designing and Executing Multimodal Interfaces for the Web based on State Char...
Designing and Executing Multimodal Interfaces for the Web based on State Char...Sebastian Feuerstack
493 views18 Folien
Model-based Design and Generation of a Gesture-based User Interface Navigatio... von
Model-based Design and Generation of a Gesture-based User Interface Navigatio...Model-based Design and Generation of a Gesture-based User Interface Navigatio...
Model-based Design and Generation of a Gesture-based User Interface Navigatio...Sebastian Feuerstack
564 views14 Folien
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval... von
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...Sebastian Feuerstack
529 views15 Folien

Más de Sebastian Feuerstack(8)

Revealing Differences in Designer‘s and Users‘Perspectives von Sebastian Feuerstack
Revealing Differences in Designer‘s and Users‘PerspectivesRevealing Differences in Designer‘s and Users‘Perspectives
Revealing Differences in Designer‘s and Users‘Perspectives
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiT von Sebastian Feuerstack
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiTComparing the Multimodal Interaction Technique Design of MINT with NiMMiT
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiT
Engineering Device-spanning, Multimodal Web Applications using a Model-based ... von Sebastian Feuerstack
Engineering Device-spanning, Multimodal Web Applications using a Model-based ...Engineering Device-spanning, Multimodal Web Applications using a Model-based ...
Engineering Device-spanning, Multimodal Web Applications using a Model-based ...
Designing and Executing Multimodal Interfaces for the Web based on State Char... von Sebastian Feuerstack
Designing and Executing Multimodal Interfaces for the Web based on State Char...Designing and Executing Multimodal Interfaces for the Web based on State Char...
Designing and Executing Multimodal Interfaces for the Web based on State Char...
Model-based Design and Generation of a Gesture-based User Interface Navigatio... von Sebastian Feuerstack
Model-based Design and Generation of a Gesture-based User Interface Navigatio...Model-based Design and Generation of a Gesture-based User Interface Navigatio...
Model-based Design and Generation of a Gesture-based User Interface Navigatio...
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval... von Sebastian Feuerstack
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
Model-based Design of Interactions that can bridge Realities – The Augmented... von Sebastian Feuerstack
 Model-based Design of Interactions that can bridge Realities – The Augmented... Model-based Design of Interactions that can bridge Realities – The Augmented...
Model-based Design of Interactions that can bridge Realities – The Augmented...

Último

ICANN von
ICANNICANN
ICANNRajaulKarim20
64 views13 Folien
CWP_23995_2013_17_11_2023_FINAL_ORDER.pdf von
CWP_23995_2013_17_11_2023_FINAL_ORDER.pdfCWP_23995_2013_17_11_2023_FINAL_ORDER.pdf
CWP_23995_2013_17_11_2023_FINAL_ORDER.pdfSukhwinderSingh895865
507 views6 Folien
The Accursed House by Émile Gaboriau von
The Accursed House  by Émile GaboriauThe Accursed House  by Émile Gaboriau
The Accursed House by Émile GaboriauDivyaSheta
158 views15 Folien
Community-led Open Access Publishing webinar.pptx von
Community-led Open Access Publishing webinar.pptxCommunity-led Open Access Publishing webinar.pptx
Community-led Open Access Publishing webinar.pptxJisc
74 views9 Folien
Google solution challenge..pptx von
Google solution challenge..pptxGoogle solution challenge..pptx
Google solution challenge..pptxChitreshGyanani1
98 views18 Folien
STERILITY TEST.pptx von
STERILITY TEST.pptxSTERILITY TEST.pptx
STERILITY TEST.pptxAnupkumar Sharma
125 views9 Folien

Último(20)

The Accursed House by Émile Gaboriau von DivyaSheta
The Accursed House  by Émile GaboriauThe Accursed House  by Émile Gaboriau
The Accursed House by Émile Gaboriau
DivyaSheta158 views
Community-led Open Access Publishing webinar.pptx von Jisc
Community-led Open Access Publishing webinar.pptxCommunity-led Open Access Publishing webinar.pptx
Community-led Open Access Publishing webinar.pptx
Jisc74 views
Lecture: Open Innovation von Michal Hron
Lecture: Open InnovationLecture: Open Innovation
Lecture: Open Innovation
Michal Hron96 views
Compare the flora and fauna of Kerala and Chhattisgarh ( Charttabulation) von AnshulDewangan3
 Compare the flora and fauna of Kerala and Chhattisgarh ( Charttabulation) Compare the flora and fauna of Kerala and Chhattisgarh ( Charttabulation)
Compare the flora and fauna of Kerala and Chhattisgarh ( Charttabulation)
AnshulDewangan3316 views
Drama KS5 Breakdown von WestHatch
Drama KS5 BreakdownDrama KS5 Breakdown
Drama KS5 Breakdown
WestHatch71 views
OEB 2023 Co-learning To Speed Up AI Implementation in Courses.pptx von Inge de Waard
OEB 2023 Co-learning To Speed Up AI Implementation in Courses.pptxOEB 2023 Co-learning To Speed Up AI Implementation in Courses.pptx
OEB 2023 Co-learning To Speed Up AI Implementation in Courses.pptx
Inge de Waard167 views
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively von PECB
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks EffectivelyISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively
ISO/IEC 27001 and ISO/IEC 27005: Managing AI Risks Effectively
PECB 545 views
Ch. 7 Political Participation and Elections.pptx von Rommel Regala
Ch. 7 Political Participation and Elections.pptxCh. 7 Political Participation and Elections.pptx
Ch. 7 Political Participation and Elections.pptx
Rommel Regala72 views
American Psychological Association 7th Edition.pptx von SamiullahAfridi4
American Psychological Association  7th Edition.pptxAmerican Psychological Association  7th Edition.pptx
American Psychological Association 7th Edition.pptx
SamiullahAfridi482 views
Scope of Biochemistry.pptx von shoba shoba
Scope of Biochemistry.pptxScope of Biochemistry.pptx
Scope of Biochemistry.pptx
shoba shoba124 views
The basics - information, data, technology and systems.pdf von JonathanCovena1
The basics - information, data, technology and systems.pdfThe basics - information, data, technology and systems.pdf
The basics - information, data, technology and systems.pdf
JonathanCovena188 views
11.28.23 Social Capital and Social Exclusion.pptx von mary850239
11.28.23 Social Capital and Social Exclusion.pptx11.28.23 Social Capital and Social Exclusion.pptx
11.28.23 Social Capital and Social Exclusion.pptx
mary850239281 views
Psychology KS4 von WestHatch
Psychology KS4Psychology KS4
Psychology KS4
WestHatch68 views
Narration lesson plan.docx von TARIQ KHAN
Narration lesson plan.docxNarration lesson plan.docx
Narration lesson plan.docx
TARIQ KHAN104 views

MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP Interactors

  • 1. MINT-Composer – A Toolchain for the Model- based Specification of Post-WIMP Interactors Sebastian Feuerstack Dr.-Ing. Sebastian Feuerstack Universidade Federal de São Carlos Departamento de Computacão 18. Oktober 2012 Sebastian Feuerstack I Sebastian@Feuerstack.org
  • 2. Basic Question How to model and run multimodal Post-WIMP interactions? 18. Oktober 2012 2
  • 3. Post-WIMP Interacton • WIMP interaction style: Window, Icon, Menu, Pointing device • Post-WIMP Interfaces ”…attempt to appear more natural and less obtrusive. Ideally this kind of post-WIMP interaction feels as familiar and fluid as using simple nondigital artifacts” [Jetter 2012] 18. Oktober 2012 3
  • 5. Why modeling? • Visual language that – is declarative and precise to discuss & execute – Is located between a tool and source code – Supports different forms of interaction and flexible prototyping of multimodal interfaces 18. Oktober 2012 5
  • 6. How to model? AUI Widget Mapping CUI CUI W W Modality D1 • Models – Interactors (Abstract & Concrete Media, Mode) • Static: class diagram, Behavior: state-chart – Mappings • Custom Notation 18. Oktober 2012 6
  • 8. Interactor Modeling 18. Oktober 2012 8
  • 9. Mode Modeling IR:Head NoHead discover lost Head Captured right_tilted center right centered left left_tilted center 18. Oktober 2012 9
  • 10. Demonstration Home Network MINT- Head Platform Recognition App Download WebServer Home Gateway End-Device WebServer Webbrowser 18. Oktober 2012 The Augmented “Drag-and-Drop 10
  • 11. Conclusions & Future Work What’s the advantage? • Detailed Modeling of Multimodal Interactions including behavior • Existing tools • No gap between design- and runtime 18. Oktober 2012 11
  • 12. Whats next? Focus on: – Fusion – Paradigm Design – Formalization – Tools – W3C Standardization http://www.multi-access.de All papers, Videos, MINT Framework as Open Source 18. Oktober 2012 Design of Multimodal Interaction 12