Masters Thesis Presentation Master Thesis document: http://wise.vub.ac.be/content/multimodal-interaction-investigation-and-implementation-context-dependent-adaptation

1. 2 December 2005
Mobile Multimodal Interaction:
An Investigation and Implementation of
Context-dependent Adaptation
Promoter: Prof. Dr. Beat Signer
Advisor: Dr. Bruno Dumas
María Solorzano

2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2February 17, 2011
Outline
 Introduction
 Background studies
 Systematic study and analysis
 Multimodal Adaptive Agenda
 Conclusions

3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3February 17, 2011
Introduction
 Mobile Interaction deals with multiple limitations
 Device hardware and size
 Context
 Multimodal Interaction is one way to address
these limitations
 Adaptation is paramount in mobile and multimodal
systems

4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4February 17, 2011
 A systematic study
 Context-aware adaptive multimodal interfaces
 A proof-of-concept application based on the study
findings
Objectives
Problem Definition
 Current studies do not provide an unified overview of
the field.
 No systematic description of the design stages
Introduction

5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5February 17, 2011
Background Studies
M-G1 MU-G1M-G1 MU-G1
MG-2.6
MG-2.7
MU-G2
MG-2.5
MG-2.9
MU-G6
M-G2.2 MU-G5 CA-G4
CA-G5CA-G5
CA-G1 A-G1
A-G2
A-G3
 Mobile Interaction
 Multimodal Interaction
 Context-awareness
 Adaptive Interfaces

6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6February 17, 2011
Systematic Study
“Analysis of multimodal mobile input channels with a
special focus on adaptation triggered or influenced by
environmental factors”

7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7February 17, 2011
Systematic Study
Selection Criteria
System Induced
Adaptation
User Induced
Adaptation
(6 articles)
(8 articles)

8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8February 17, 2011
Systematic Study
Multimodal Interaction
 Modalities
 Interaction techniques
 Interaction sensors
 Output Influence
 CARE properties
Mobile Interaction and Context-awareness
 Device (Taxonomy for mobile Terminals - Schiefer and Decker’s )
 Environmental properties (Cameleon Reference Framework)
Adaptive Interfaces
 Monitoring entity
 Adaptation policy
 Adaptation mechanism
Classification Schema

9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9February 17, 2011
Systematic Study
Analysis
Combination of Modalities
Context Influence
System-Induced Adaptation
*
*

10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10February 17, 2011
Systematic Study
Combination of Modalities
E =equivalent and C= complementary
* The numbers refer to the bibliographic references

11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11February 17, 2011
Systematic Study
System-induced Adaptation
* The numbers refer to the bibliographic references

12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12February 17, 2011
Systematic Study
Guidelines for Automatic Adaptation of
Input Channels
Context and Modality Suitability Analysis
1. Conduct a context analysis
2. Define suitable modalities for each semantic location
Multimodal Tasks Definition
1. Select tasks that will support a multimodal behaviour
2. Define equivalent modalities
Adaptation Design
1. Define adaptation triggers and monitoring entities
2. Define adaptation policy
3. Define adaption mechanism
4. Provide feedback

13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13February 17, 2011
Multimodal Adaptive Agenda
Objectives
Mobile Calendar Application
+ multimodal capabilities
+ adaptive capabilities
 Test proposed design guidelines
 Explore less frequently used modalities
 Explore fusion engine and adaptation
mechanism integration

14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14February 17, 2011
Easiness
Of
Interaction
Multimodal Adaptive Agenda
Guidelines Category 1:
Context and Modality Suitability Analysis
Context
Analysis

15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15February 17, 2011
Multimodal Adaptive Agenda
Guidelines Category 2:
Multimodal Tasks Definition
Equivalent modalities
Multimodal
Tasks

16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16February 17, 2011
Multimodal Adaptive Agenda
Guidelines Category 3:
Adaptation Design
 Adaptation triggers and Monitoring entities
 semantic location (indoors, outdoors) GPS
 noise level (low, medium, high) Microphone
 user activity (walking, still) Accelerometer

17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17February 17, 2011
Multimodal Adaptive Agenda
Architecture

18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18February 17, 2011
Multimodal Adaptive Agenda
User Interface
Context and available
modalities status

19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19February 17, 2011
Multimodal Adaptive Agenda
Demo

20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20February 17, 2011
Conclusions
Future Work
 Extend the systematic study
 User context
 Device context
 Develop a rapid prototyping tool
 Conduct a field study with users
 But, the application’s functionality should be
enhanced before

21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21February 17, 2011
Conclusions
Summary
 Review of the state-of-the-art
 Selection, classification and analysis of relevant
research projects
 Proposal of a set of design guidelines
 Design and development of a proof of concept
based on the guidelines