The document outlines the research of the EHCI group which focuses on engineering human-computer interaction, interaction plasticity, post-WIMP interaction, modality, and multimodality. It discusses the group's research axes including models for designing HCI, adaptation to interaction context, vision-based tracking, and mixed reality. The EHCI group has conducted experiments on multimodality using a mobile game called RAZZLE to study subjects' usage of speech, gesture, and tactile modalities.
1. Laurence Nigay - laurence.nigay@imag.fr
Outline
Mobile Human-Computer Interaction
• Focus on Human-Computer Interaction
Introductory Course
• EHCI Group
Laurence Nigay • Mobility: a broad concept
University of Grenoble
• Interaction with handheld devices
Grenoble Informatics Laboratory - LIG
Engineering Human-Computer Interaction Group - EHCI
CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France
EHCI group & Mobile HCI
• EHCI Group • “EHCI” for
“Engineering for Human-Computer
Interaction”
– Created in 1990
– 7 professors and assistant-professors
– 15 PhD students
– 4 research engineers and 1 post-doc
EHCI group EHCI group: Research thematic
• Software Engineering for HCI
• EHCI group belongs to LIG
(Computer Science Laboratory of Grenoble)
Interaction
• 165 academic researchers Computer science Supported by
System Human
• 240 doctoral and post-doctoral students contribution Human sciences
• 65 technical and administrative staff
– 24 autonomous research teams • Development of conceptual and technical tools
based on HCI principles : Utility, usability, context
– 4 principal themes • Establishing links between psychology-
• Infrastructure (networks and data) ergonomics and software engineering
• Software (foundations and design models)
• Interaction (perception, action and dialog)
• Knowledge (learning, agent models and web-ontologies)
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
2. Laurence Nigay - laurence.nigay@imag.fr
EHCI group: Research axes EHCI group: Research axes
• 4 complementary research axes • Models and processes for designing
– Models and processes for designing HCI, HCI, formal testing
formal testing
– Interaction plasticity • Task Hierarchy: a model at runtime
• Adaptation of HCI to interaction context
– Post-WIMP interaction
• Vision based finger tracking and two-handed
interaction …
• Mixed reality
– Modality and multimodality
EHCI group: Research axes EHCI group: Research axes
• Interaction plasticity • Interaction plasticity
• Adaptation of HCI to interaction context • Adaptation of HCI to interaction context
• Context: interaction resources / user preferences /
social context / location etc.
EHCI group: Research axes EHCI group: Research axes
• Interaction plasticity • Interaction plasticity
Control panel
Slides
2. PDA: Partial
migration (only the
1. PC only
1. PC control panel, not the
speaker video)
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
3. Laurence Nigay - laurence.nigay@imag.fr
EHCI group: Research axes EHCI group: Research axes
• Interaction plasticity • 4 complementary research axes
» camnote_shortRepetWeaving.mov – Models and processes for designing HCI,
formal testing
– Interaction plasticity
• Adaptation of HCI to interaction context
– Post-WIMP interaction
Control panel • Vision based finger tracking and two-handed
interaction
• Mixed reality
– Modality and multimodality
EHCI group: Research axes EHCI group: Research axes
• WIMP • Vision-based tracking (finger, token)
– Windows: scroll, resize, move
– Icons: representations, drag/drop
– Menus: pop-up, pull-down
– Pointers: mouse, trackball
• Post-WIMP
EHCI group: Research axes EHCI group: Research axes
• Mixed Reality • New technique of menu: multi-stroke
» Video Geonote_26.avi marking menu •Submenus
» Video MenuWave.wmv Previsualization
» Video WaveMenuIphone.mov
– Rapid scan of possibilities
get
•Parent menus:
1) Menu appears
– Always visible
2) Menu is enlarged – Can be directly selected
3) Submenu appears
4) Final item is selected •Visual stability of
the focus of attention
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
4. Laurence Nigay - laurence.nigay@imag.fr
EHCI group: Research axes EHCI group: Research axes
• 4 complementary research axes • Modality and multimodality
– Models and processes for designing HCI, • RAZZLE: Mobile game
– The goal of the player is to collect the pieces of a digital
formal testing puzzle
– Interaction plasticity – The digital puzzle pieces are scattered all over a
• Adaptation of HCI to interaction context modeled playground
– Post-WIMP interaction
• Vision based finger tracking and two-handed
interaction
• Mixed reality
– Modality and multimodality
EHCI group: Research axes EHCI group: Research axes
• Modality and multimodality • Modality and multimodality
• Multimodality for the task of collecting a selected • RAZZLE experimental tests (10 subjects / 3 sessions)
puzzle piece – Use of the modalities
– Speech modality Get – All sessions / All subjects
Video RAZZLE
RAZZLEExpe2.dv
– Gestural modality
24% Tactile
45% Gestural
Speech
Video RAZZLE
– Tactile modality
RAZZLEExpe1.dv 31%
EHCI group: Research axes EHCI group: Research axes
• Modality and multimodality
• Modality and multimodality
• RAZZLE experimental tests (10 subjects / 3 sessions)
• RAZZLE experimental tests (10 subjects / 3 sessions) – Subject 4: Use of the modalities per action
– Usage of the modalities per subject (for the three sessions)
100%
90% 100%
80% 90%
70% 80%
70%
60%
Tactile 60% Tactile
50%
Gestural 50% Gestural
40%
Speech 40% Speech
30%
30%
20%
20%
10%
10%
0% 0%
Rotate 1 Collect 1 Rotate 2 Collect 2 Rotate 3 Collect 3
1 2 3 4 5 6 7 8 9 10
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
5. Laurence Nigay - laurence.nigay@imag.fr
EHCI group: Research axes EHCI group: Research axes
• Modality and multimodality • Modality and multimodality
• RAZZLE experimental tests (10 subjects / 3 sessions) • RAZZLE experimental tests (10 subjects / 3 sessions)
– Subject 9: Use of the modalities per action – The subjects used all of the modalities
(for the three sessions)
100%
=> Usefulness of multimodality
90%
80%
– Individual preferences leading in some cases to
70%
60%
specialization
Tactile • Subject 4: Tactile modality for rotating a puzzle piece
50%
Gestural
40%
• Subject 9: Speech for rotating and collecting a puzzle piece
Speech
30% => Usefulness of multimodality
20%
10%
0%
Rotate 1 Collect 1 Rotate 2 Collect 2 Rotate 3 Collect 3
EHCI group: Research axes
• Engineering for HCI
– Models and processes establishing links between
• EHCI group
psychology-ergonomics and software engineering • Mobility
– Interaction plasticity
– Post-WIMP interaction
– Modality and multimodality
– http://iihm.imag.fr/en/
Mobility: a broad concept Mobility: a broad concept
(1) moving between devices - hot-desking (1) moving between devices - hot-desking
(2) moving within an instrumented (2) moving within an instrumented
environment - intelligent buildings and environment - intelligent buildings and
intelligent appliances intelligent appliances
(3) devices within moving vehicles- (3) devices within moving vehicles-
computers in cars computers in cars
(4) small devices that move with you - (4) small devices that move with you -
PDAs, laptop computers etc PDAs, laptop computers etc
Alan Dix (Lancaster University) Alan Dix (Lancaster University)
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
6. Laurence Nigay - laurence.nigay@imag.fr
Mobility: Mobility:
moving between devices moving between devices
• Nomadic computing
Mobility:
moving between devices Mobility: a broad concept
• In the 1990s, the Internet (1) moving between devices - hot-desking
– a global transport for e-mail (2) moving within an instrumented
and data exchange, environment - intelligent buildings and
– but mostly for people intelligent appliances
at fixed locations (3) devices within moving vehicles-
computers in cars
• More recently (4) small devices that move with you -
– the office extended to employees on-the-go
PDAs, laptop computers etc
("nomadic workers")
=> similar to the case 4 « small devices that move Alan Dix (Lancaster University)
with you »
Moving within an instrumented Moving within an instrumented
environment environment
Smart spaces
How many
people are in
What is the the gym? Is
nearest available the treadmill
conference room? free?
• Examples: Motion Sensor:
– Smart spaces Room 3201 is
Empty What is the
– Augmented classroom / Augmented museum average
temperature
on the second
– Wireless supermarket floor?
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
7. Laurence Nigay - laurence.nigay@imag.fr
Moving within an instrumented Moving within an instrumented
environment environment
• Active Badge
• Olivetti Research Laboratory in
– Experimented with 32 staffs in ORL company in 2 weeks
Cambridge, UK, in 1990-92 – All staff wearing badge that emits IR signal every 15 seconds
• Active Badge: location-aware systems – Intended to aid telephone receptionist
– FIND(name)
» Provides current location of the named badge
wireless
IR network – WITH(name)
Active Badge Badge Sensor Server/Application » Locates a named badge and other badges around the badge
– LOOK(location)
Active Badge » Provides badges near the specified location
Badge Sensor – HISTORY(name)
... » Generates report of the location history for the named badge
Active Badge
Moving within an instrumented Moving within an instrumented
environment environment
• Server • ATT Lab Cambridge: Active Bat
• Network Control
– Polling all sensors on the network – Sentient Computing: a form of ubiquitous
• Representation computing which uses sensors to perceive its
– Builds linked list of (ID, location, time) in time order
• Data Processing environment
– Process large amount of data from the active badge network - A "follow-me phone" which would cause the
• Display Interface telephone nearest the recipient to ring.
– Showing textual information
- Teleporting desktops via VNC just by clicking their
Active Bat near the computer.
Video Sentient.mpg
Moving within an instrumented Moving within an instrumented
environment environment
• ParcTab (Rank Xerox)1995
– http://sandbox.parc.xerox.com/parctab/ • Projet EasyLiving Microsoft
• Video
EasyLiving.mpeg
7 ,8
cm
cm
2 ,4
5 cm
10,
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
8. Laurence Nigay - laurence.nigay@imag.fr
Moving within an instrumented Moving within an instrumented
environment environment
• Location-aware system (within a buidling)
• Context-triggered actions are simple IF-
THEN rules used to specify how context-
aware systems should adapt
Like living in a rule-based expert system
• Examples:
– Smart spaces
– Augmented classroom / Augmented museum
– Wireless supermarket
Moving within an instrumented Moving within an instrumented
environment environment
• Augmented classroom: AR (video)
• Examples:
– Smart spaces
– Augmented classroom / Augmented museum
geometry and mathematics education – Wireless supermarket
Moving within an instrumented
Wireless supermarket concepts
environment
• Augmented surpermarket • The Shopping Buddy
– Based on RFID for products wireless touch-screen device attached to a
– WLan for communication inside the shopping cart
supermarket – scans in items placed in the cart by shoppers
– Intelligent chariots (with PC tablet) – delivers personalized services and incentives
when activated with a frequent-shopper card
– RFID which triggers certain offers and can
help shoppers find anything in the store and
draw a path to find it.
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
9. Laurence Nigay - laurence.nigay@imag.fr
Wireless supermarket Operation Wireless supermarket Operation
• As you shop, you can scan in each item
– keeps a running total of how much you are spending
– eliminates the need to wait in line at the check-out
– you can also check the price of each item before you buy
• As you walk down the aisles
– promotions and paperless coupons “pop-up” on the screen
– Reminders of articles you search
– Directions to articles
• You can place an order at the deli (ex. Half a kilo of ham)
from anywhere
– notification to pick it up (No more taking a number and
standing in line)
Wireless supermarket Operation Wireless supermarket Operation
• The Everywhere Display (IBM) transforms any surface
• The Everywhere Display (IBM) into an interactive computer
• Beamed • A recent MIT project: Sixthsense
from the supermarket ceiling – wearable gestural interface that augments the
• transforms any surface physical world around us with digital infor
into an interactive computer
• On the floor
• On the items
Moving within an instrumented
Mobility: a broad concept
environment
• Technological approach (1) moving between devices - hot-desking
– 1. Augment the user (2) moving within an instrumented
• The user wears or carries a device to obtain environment - intelligent buildings and
information about physical objects. intelligent appliances
• => similar to the case 4 « small devices that move (3) devices within moving vehicles-
with you »
computers in cars
– 2. Augment the physical object
(4) small devices that move with you -
• The physical object is changed by embedding
input, output or computational devices on or within PDAs, laptop computers etc
it.
– 3. Augment the environment surrounding the Alan Dix (Lancaster University)
user
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
10. Laurence Nigay - laurence.nigay@imag.fr
Devices within moving vehicles Devices within moving vehicles
micro
Tactile Surface
Keyboard
Devices within moving vehicles Devices within moving vehicles
• FutureLab of Ars Electronica (Autria)
• INTUITION DGA project
• Augmented Reality
• Augmented cockpit (Rafale)
• Instar.mov video
Devices within moving vehicles Mobility: a broad concept
• INRETS-Lille
(1) moving between devices - hot-desking
(2) moving within an instrumented
environment - intelligent buildings and
intelligent appliances
(3) devices within moving vehicles-
computers in cars
(4) small devices that move with you -
PDAs, laptop computers etc
Alan Dix (Lancaster University)
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
11. Laurence Nigay - laurence.nigay@imag.fr
Evolution of personal computers:
Mobility: small devices
a continuum
Desktops Laptops Palmtops Handhelds Wearables
Eye-pod, projection on contact lenses (univ. Washington)
Evolution of personal computers: Evolution of personal computers:
a continuum a continuum
External support + hand(s) 1 hand (at least) Hands-free
Desktops Laptops Palmtops Handhelds Wearables Desktops Laptops Palmtops Handhelds Wearables
Weight
portability/mobility/maniability
Wearable Computer Aim of Wearable Computing
• A small portable computer that is designed to be worn on •To develop new interfaces that
the body during use. mediate (augment, deliberately
diminish, or otherwise modify) non-
• In this, wearable computers differ from PDAs, which are computer activities
designed for hand-held use.
• Wearable computers are usually either integrated into
the user's clothing or can be attached to the body • Without interfering with the user's
through some other means, like a wristband. everyday tasks
• They may also be integrated into everyday objects that
are constantly worn on the body, like a wrist watch or a • The design of wearable computers is
hands-free cell phone. still a topic of research, and a variety
of user interfaces are being proposed
University of Waterloo CA University of Waterloo CA
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
12. Laurence Nigay - laurence.nigay@imag.fr
Evolution of
Application domains
Wearable Computers
• Medical domain
– Medical
monitoring
device
– (blood pressure,
temperature)
• Military domain Amon: ICT project
GlucoWatch
Application domains Wearable Computing
• Mobile workers
• Key characteristics
– Hands free
• Constant: Always ready. It is always on and always
running.
• Personal: Human and computer are inextricably
intertwined.
Mobility: Huge variability of
devices Mobility: small devices
• Usage spaces
of wireless devices
Desktops Laptops Palmtops Handhelds Wearables
Marcus A., Chen E “Designing the PDA of the Future” Interactions 2002
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
13. Laurence Nigay - laurence.nigay@imag.fr
Mobility: small devices Mobility: small devices
• Information • M-Commerce
– Personal Information Management – Electronic money, e-coupons
– Business information: specific professionals, such as a drug
interaction reference for doctors
• Self-Enhancement
– Active assistance
• Ways in which the device extends one’s normal capabilities.
(Finland: cell-phones "Kanny » extension of the hand »)
– Health and Safety
• Contact with emergency response teams or family
– Education
Mobility: small devices Mobility: small devices
• M-Commerce • Usage spaces
– Electronic money, e-coupons of wireless devices
• Relationships
– Core competence of phones to connect people • Huge variety
• Entertainment of applications
– Music, games
– The mobile game market tends to follow the more
mature console/PC game market
– Game performance must match or exceed the users’
expectations
Marcus A., Chen E “Designing the PDA of the Future” Interactions 2002
Variety of applications Variety of applications
• Device: designed for a certain user task
• Device: designed for a certain user task
– impractical to have several different information
Sony – Reader-ebook PRS-505
Screen has no backlight or flicker
appliances for different purposes
to give the experience of reading a paper page
• Device: « Swiss-army-knife » useful in many
different situations
• + Different devices are appropriate for – HCI complicated / non intuituive
different tasks and this should also guide the
application design • the question of information appliance or
• - Impractical to have several different information Swiss-army knife still remains unanswered
appliances for different purposes
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
14. Laurence Nigay - laurence.nigay@imag.fr
Mobility: Huge variability of
Mobility: a broad concept devices
(1) moving between devices - hot-desking
(2) moving within an instrumented Desktops Laptops Palmtops Handhelds Wearables
environment - intelligent buildings and
intelligent appliances
(3) devices within moving vehicles-
computers in cars
(4) small devices that move with you -
PDAs, laptop computers etc
Alan Dix (Lancaster University)
Objectives
• EHCI group
• Mobility
• Interaction with handheld devices
From: MobileHCI 07 N. Savio J. Braiterman
Design Sketch: The context of Mobile Interaction
Challenges Variability of usage context
• Key issue: Transparency/Usability of
mobile devices
• Challenges for HCI
– Variability of platforms
– Variability of usage context
– Limited interactionnal resources
From E. Kaasinen: User acceptance of mobile services - value, ease of use, trust
and ease of adoption
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
15. Laurence Nigay - laurence.nigay@imag.fr
Challenges Challenges
• In mobile computing: the problem space
• In conventional HCI: the problem space is is unbounded
bounded
– The target user (e.g., an office clerk)
– accomplishes a well-defined set of tasks (e.g., writing a report)
– using a fixed class of devices (e.g., a workstation)
– in a predefined set of environments (e.g., in the office) Task User
Env. Platform
Task User
Env. Platform
Introduction: how to address this Handhelds: diversity, but in
problem? common …
• Approach1: to develop the systems on a case per case basis (ad- • Small screens
hoc manner)
• Limited input capabilities
– Development cost and maintenance: very high!
– Consistency problem between the various versions
• No standards yet
• Approach2: to improve the methods and tools in order to support • Recommandation
– Portability • Apply the foundations of HCI - e.g., the method
– Reusability
• Create without re-inventing the wheel: draw your
– Modifiability
design on existing solutions (in order to support
– Scalability in terms of
consistency across applications/brands) and invent
• Computing resources new design solutions to overcome current limitations
• Services
• Interaction techniques (multimodality, UI for small and
large devices)
=> Plastic UI
Objectives Focus on HCI
Gulf of Evaluation
Mobile Device Goals
Gulf of Execution
From: MobileHCI 07 N. Savio J. Braiterman
Design Sketch: The context of Mobile Interaction
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
16. Laurence Nigay - laurence.nigay@imag.fr
The design principle of
The Design of Everyday Things
transparency
• How to design things that real people can
a design principle is to try to make systems transparent so
that people can understand them better and know what to
use: successfully, easily, pleasantly
do • 7 stages of performing an action
• NOT to be understood as literal
• Conceptual models – the device should
• useful feedback explain itself
• easy to understand • Visibility and Affordances
• intuitive to use
• clear & easy to follow instructions • Mappings
• context sensitive guidance of • Feedback
how to proceed when stuck
From Prof. Steve Whittaker From Prof. Steve Whittaker
Seven Stages of Action Gulfs of Execution & Evaluation
Goals
Goals
Evaluation of
Evaluation of Intention to act the interpretations
Intention to act the interpretations
GULF OF EXECUTION
Interpreting
Interpreting the perception
Sequence of actions
the perception
Sequence of actions GULF OF EVALUATION
Execution of the Perceiving the state Execution of the Perceiving the state
action sequence of the world action sequence of the world
The World The World
From Prof. Steve Whittaker From Prof. Steve Whittaker
Focus on HCI The Gulf of Execution
• Does the system provide actions that
correspond to the user’s intentions?
Gulf of Evaluation
• The difference between intentions and
Mobile Device Goals allowable actions is the Gulf of Execution
Gulf of Execution
From Prof. Steve Whittaker
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
17. Laurence Nigay - laurence.nigay@imag.fr
The Gulf of Evaluation The Seven Stages as Design aids
• Does the system provide a physical How easily can the user:
representation that can be readily Determine the function
perceived and interpreted in terms of the of the system?
user’s intentions and expectations? Tell what actions Tell if the system is
in the desired state?
• The Gulf of Evaluation reflects the amount are possible?
of effort that the person must exert to Determine a mapping from
Determine a mapping from
interpret the physical state of the system intention to physical action? system state to interpretation
and determine how well the intentions
have been met. Perform the action? Tell what state the
system is in?
From Prof. Steve Whittaker From Prof. Steve Whittaker
… principles for good design Mobile HCI: inputs/outputs
• Visibility – by looking, users can tell the state of the device and
possible actions
• Conceptual model – operations are presented consistently; users
can easily understand “how the system works”
• Mappings – it is possible (easy) to determine the relationship
between actions and results, between controls and their effects, and
between the system state, and visible indicators
• Feedback – users get full and continuous feedback about the
results of actions
From Prof. Steve Whittaker
Mobile HCI Mobile HCI
• A very active domain
• A very active domain
• Approach1: to develop the systems on a case per case basis
• Approach2: to improve the methods and tools in order to support
– Portability • Plasticity (approach 2)
– Reusability
– Modifiability • Innovative interaction techniques
– Scalability in terms of (approach 1)
• Computing resources
• Services – Innovative interaction techniques
• Interaction techniques (multimodality, UI for small and – Multimodality
large devices)
=> Plastic UI
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
18. Laurence Nigay - laurence.nigay@imag.fr
Plasticity of an interactive
Plasticity
system
• Capacity of the system to adapt to the
• Materials: capability to change their own context of use while preserving their own
shape while preserving usage, they do not utility and usability
break (within some limits, of course)
• Animals, plants: they adapt to the • Context of use …
environment to survive • Adaptation …
• Utility and Usability …
• By analogy in HCI …
Context of use Plasticity of an interactive
system
• Context: it’s complex! Generally speaking, it is an information space
– that is shared between actors, dynamic, structured, and
– that serves interpretation (by the actors)
• There is “no context without context” -> context of use
• Capacity of the system to adapt to the context
• Context of use: an information space of use while preserving their own utility and
– that is shared (between the software components of the interactive system and
between the system and the users), dynamic, and structured into 3 sub-spaces: usability
• The users
• The platform (from an elementary platform such as a mobile phone to a
cluster of elementary devices)
• The physical environment (location, light-sound-heat and social conditions)
– that supports the adaptation process to preserve the system utility and usability • Context of use …
• Adaptation …
• Utility and Usability …
Two means for UI adaptation UI remolding (reshaping)
• Example: Plastic Clock
• Remolding (reshaping)
• Redistribution
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
19. Laurence Nigay - laurence.nigay@imag.fr
UI remolding (reshaping)
UI remolding (reshaping)
• Physical Presentation level (PP)
NSButton :
Contrôleur de
Nabaztag dialogue
Button :
ANF Présentation
Interaction
JButton :
NF Interaction
Interaction
The classes of widgets are maintained but their
behaviour (possibly, their spatial and temporal
layout) has changed
PC
PDA
UI remolding (reshaping): may UI remolding (reshaping)
cover multiple levels of abstraction
• Logical presentation level (LP) • Dialogue Control level (DC)
Label + ComboBox : Contrôleur de
dialogue
Tasks are maintained but their ordering has changed
DC Modification -> LP modification -> PP
Contrôleur de
dialogue
Contrôleur de
Contrôleur de
modification
dialogue ANF dialogue Présentation
Présentation
ANF
Label + T extField :
Présentation NF Interaction
Interaction
NF Interaction
Interaction
Substitution of widgets
LP Modification -> PP Modification
UI remolding (reshaping) UI remolding (reshaping)
• Functional Core Adaptator level (FCA) • Functional Core level (FC)
Contrôleur de
Contrôleur de
Book (workstation) Contrôleur de
Contrôleur de
ANF
dialogue
dialogue
- title ANF
dialogue
dialogue
The nature of the services of the functional core has
changed
ANF ANF
ANF
Présentation
- authors ANF
Présentation
- summary
FC modification ->FCA Modification -> DC modification
NF Interaction NF Interaction
- keywords
… -> LP modification > PP modification
Book(Handheld)
- title
- first author
-keyword
Tasks and domain-dependent concepts may be added/suppressed
FCA Modification -> DC modification -> LP modification > PP modification
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
20. Laurence Nigay - laurence.nigay@imag.fr
UI Redistribution
UI Redistribution
• Rekimoto’s “pick
and drop”
• The UI uses interaction resources that are distributed over multiple elementary
platform (distributed UI) – Distribution is static
• UI redistribution may be static or dynamic
– Distribution at the
workspace level
• Dynamic redistribution => on-the-fly migration of the UI
– No remolding
– Migration may be total: the UI releases (frees) all of the interaction resources
currently used and migrates entirely to other interaction resources
– Migration may be partial: only portions of the UI migrate -> what is the
• i-LAND
granularity for UI migration? – Distribution is dynamic
– Distribution at the pixel
level
– No remolding
http://www.ipsi.fraunhofer.de/ambiente/english/projekte/projekte/i_land.html
UI Redistribution Plasticity of an interactive
2. Arrival of the PDA: UI partial migration
system
• CamNote
• Capacity of the system to adapt to the context
of use while preserving their own utility and
usability
Remolding of the
control panel • Context of use …
at the FCA level
1. PC only: centralized UI
• Adaptation …
3. Distributed UI (PC and PDA) +
• Utility and Usability …
remolding
Usability of an interactive
Utility of an interactive system
system: many faces!
• Capacity of the system to provide the target users with
the appropriate functions/services (not less, not more) • To make it simple: Capacity of the system to allow the
target users to access and manipulate the services
• Appropriate = in conformity with users’need (not with provided by the interactive system in conformity with
designers’ and developers’ needs!) their cognitive, motor and perceptual capabilities
– Cognitive is the hard part ! Knowledge, pleasure,
• How to elicit the appropriate services? emotion, mood come into play!
– Talking to users is not a luxury, it is mandatory!
– Questionnaires, focus group, evaluation, rapid • How to develop a usable system?
prototyping (low-fidelity), observation of users in – Use theory-based knowledge (e.g., Theory of action,
their daily activities, etc. Fitts law)
– In short: user-centered design – Apply ergonomics rules
– Design, evaluate with the end-user and iterate
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
21. Laurence Nigay - laurence.nigay@imag.fr
Utility and Usability are not intrinsic to a product!
It depends on the context of use Mobile HCI
• Utility and usability depend on the “values”
that the user associates to the system • A very active domain
• The “values” depend on the context of use • Plasticity
• Innovative interaction techniques
– Innovative interaction techniques
• Example: possible values for a heating control
system that may depend on the context of use – Multimodality
– Money saving
– Comfort
Interaction techniques:
Interaction techniques
small screen
• Small screens • Fisheye technique
• Limited input capabilities – Allows the user to see
the details
without loosing
the context of
the details
(Furnas:
seminal work)
– Video
– datelens-video-web-server.wmv
Interaction techniques: Interaction techniques:
small screen small screen
• A peep hole on a large information space [Ka-Ping Yee, CHI2003]
• Problem: loss of the context - • Microsoft Research Lab
• Halo tends to alleviate this problem
• Projet Halo
Halo to make observable the information that
is of interest to the user
• Problem +
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
22. Laurence Nigay - laurence.nigay@imag.fr
Interaction techniques: Interaction techniques:
small screen small screen
• Hybrid technique: thumbnail+editing
Video Halo.avi • Collapse-to-Zoom
– Pages that are to big to fit the screen are presented as a “summary
thumbnail”
– Thumbnails are editable (can be suppressed/enlarged, etc.)
Solution expand
collapse
[Baudisch CHI 2004]
Interaction techniques: Interaction techniques:
small screen small screen
• Hybrid technique: thumbnail+editing • Menu techniques
• Collapse-to-Zoom
– Space on screen
– Pages that are to big to fit the screen are presented as a “summary
thumbnail” – No keyboard for shortcuts (novice mode)
– Thumbnails are editable (can be suppressed/enlarged, etc.) – The best way to interact is to use only one-hand
Grey lines indicate the
existence of collapsed areas Circular menus
(advertisement)
video collapsetozoom
Interaction techniques: Interaction techniques:
small screen small screen
• Marking menus • Marking menus
• Multi-stroke marking menus
• Advantages • Instead of considering a spatial compound stroke,
– Circular design • Multi-Stroke menus introduce a serie of simple strokes
– Fluid transition
– Scale independance
• Require less physical input space
in novice & expert modes
– A submenu is displayed on top of
• Limitations its parent menu
– Screen space requirement
– Overlapped marks
– Number of commands
• Error rate in expert mode • Increase accuracy in expert mode
• Ambiguous marks in expert mode
• Increase the number of items
– No ambiguous gestures in expert mode
video `Multistroke
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
23. Laurence Nigay - laurence.nigay@imag.fr
Interaction techniques: Interaction techniques:
small screen small screen
Wave menus Wave menus
• Few physical input space
• A submenu is displayed at the
center of the menu system. In
order to remain visible, its parent
menu is then enlarged to surround Novice Mode
this submenu
• In expert mode, Wave Menus (1)
work exactly the same way as
Multi-Stroke Menus (2)
Expert Mode
video WaveMenu
Interaction techniques: Interaction techniques:
small screen small screen
Wavelet menus: extension on Iphone
Wavelet menus: extension on Iphone
•The Wavelet menu appears centered around the
contact point.
•By drawing a stroke towards the desired item, the
first level is enlarged permitting progressive
appearance of the submenu.
•A second stroke selects an item in the submenu.
Interaction techniques: Interaction techniques:
small screen small screen
• Menu techniques
Wavelet menus: extension on Iphone – Space on screen
– The best way to interact is to use only one-hand
Long lists management: the linear list appears in
the center of the Wavelet menu and is surrounded video Leafmenunovice Leafmenuexpert
by its parent menus. ( video Wavelet.mov)
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
24. Laurence Nigay - laurence.nigay@imag.fr
Interaction techniques: Interaction techniques:
small screen small screen
• Menu techniques
• Non graphical techniques => Eye-Free interaction
– Space on screen
– Tactile & Sound feedback
– The best way to interact is to use only one-hand
– ArchMenu, ThumbMenu • Spatialized sound
Meeting at 10
Soundbeam
Neckset
Interaction techniques: Interaction techniques:
small screen small screen
Vibro-Tactile feedback
Shoogle senses data (and makes sense of them!) [ Univ. Vibro-tactile feedback
Glasgow 07] Shape-change feedback: the dynamic knob [Deutsche Telecom
Sensors: the Shake includes an accelerometer, gyroscope, et Univ. Postdam]
magnetometer, vibro-tactile display communicating via
bluetooth
Expresses the existence of messages, remaining resources
(e.g., the battery level) using a dynamic model inspired from
mechanics (balls anchored with springs bouncing against walls)
Interaction techniques Text entry
• Small screens
• Limited input capabilities
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
25. Laurence Nigay - laurence.nigay@imag.fr
Text entry Keyboard: Text entry
Text entry Text entry
• LucidTouch: • LucidTouch:
a see-through mobile device, pseudo-tranparency a see-through mobile device, pseudo-tranparency
Keyboard: entry Text entry
• Oulu University (Finland) • Hand writting + Movements
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
26. Laurence Nigay - laurence.nigay@imag.fr
Pointing Techniques Pointing Techniques
• The best way to interact is to use only • “Offset Cursor” [Potter, Shneiderman 1988)]: how to replace the stylus
one-hand •
with the finger without hiding the target
Selection technique: finger touches the screen, display the cursor at 1/2
• Using the thumb inch over the finger, move the finger to bring the cursor over the target,
remove finger
Target Finger hides the target Finger moves to bring the cursor
Gestures of the thumb Cursor is offset over the target
MicroRolls from the target
Pointing Techniques Pointing Techniques
• Shift [P. Baudisch 07, MicroSoft]: improves the “offset cursor” technique
Occlusion
Occlusion of targets Occlusion of the interface
Border accessibility
Thickness of screen
Areas more difficult to reach [Parhi 06] edges
Accuracy
Video: shiftMini.m4v Larger than the stylus Prediction hard
Pointing Techniques New input Techniques
• Tap-Tap Anne Roudaut (AVI 2008) Video: taptapMini.m4v
• Temporal multiplexing strategy
• Camera phone based motion sensing
• 1ST tap: Specify the focus area
Uni. of California, Berkeley
• 2nd tap: Select the target at a larger scale & IBM
Video: Phone3DMvt-UIST06.mov
Target to select 1st tap Magnified popup 2nd tap
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
27. Laurence Nigay - laurence.nigay@imag.fr
New input Techniques New input Techniques
• Movement-based UI – 3D gesture
• Tilting using accelerometers [Rekimoto 1996, Harrison&Fishkin 1998] • Movement-based UI – 3D gesture
• Embodied UI: Rank Xerox
Photos album
Video: RocknScroll.mov
New input Techniques New input Techniques
• Embodied UI: Rank Xerox
• Embodied UI
• Prototyping tool
for embodied UI
– Stanford HCI group
–Video: MapPDADesignplatform-UIST06.mov
New input Techniques New input Techniques
• RodDirect: the stylus as a peripheral device (2 degrees of
freedom) + the second hand • Distorsion: bendable display
• Two types of stylus movement in the stylus holder are • The Gummi interface: just bend it! (Sony, Schwesig CHI 2004)
associated with parameters
– (1) Twist (Rotation) (2) Push/Pull (Sliding)
Map Browser Application Switching Scheduler
JAIST: Japan Advanced Institute of Science and Technology, JAPAN
Video: RodDirectDemo.mpg
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France
28. Laurence Nigay - laurence.nigay@imag.fr
New input Techniques Mobile HCI
• Gummi: zoom-in a subway station
• A very active domain
• Plasticity
• Innovative interaction techniques
• Alpha-blending of 2 graphics layers – Innovative interaction techniques
– Multimodality
Which actor performs modalities
Multimodal Interaction
selection ?
• Multimodal systems
– Multi-Sensori-Motor Systems
– extend the sensori-motor capabilities of
computer systems = mobile devices
Designer User System
• Multimodal interaction makes use of several
input and/or output (feedback) modalities in Information Selection of Multimodal
to be one or several modalities
interacting with a computer system. conveyed
Expression
– Examples of modalities: manual gestures, gaze,
touch, speech, head & body movements Actor of the selection
– Modality: human sensory channel, different
representation modality, or different input method
Multimodality selection Multimodality Adaptability
Go to the middle
of the message
No adaptation
Gestural modality Speech
Adaptability
Adaptivity Selection by Selection by Selection by
the designer the user the system
Embodied modality Direct manipulation
CNRS Summer School on Mobile Learning
4-9 July 2009, Brest, France