1. Improving access to mobile
technologies using tactile feedback
Huimin Qian
Information Systems Department
UMBC
October 3rd, 2012

2. Motivations
Proposed solution
Projects
Interacting with mobile devices
Summary
Desktop computers Interacting with desktop computers
Mobile devices In mobile contexts
2

3. Motivations
Proposed solution
Projects
Interacting with mobile devices
Summary
Desktop computers Interacting with desktop computers
Mobile devices In mobile contexts
3

4. [Shaun, 2011]
Motivations
Proposed solution Inaccessible for
Projects
blind users
Summary
4

5. [Sears et al., 2003; Brown et al., 2005; Wobbrock, 2006]
Motivations
Proposed solution Inaccessible for
Projects
sighted users in a number of situations
Summary
Walking Driving
In a class In social situations
5

6. [Sears et al., 2003; Brown et al., 2005; Wobbrock, 2006]
Motivations
Proposed solution Inaccessible for
Projects
sighted users in a number of situations
Summary
“Situationally-induced impairments and disabilities”
Walking Driving
SIID [Sears & Young, 2003]
In a class In social situations
6

7. Motivations
Proposed solution
Projects
5 communication channels
Summary
• Sight Restricted
• Smell Impractical
• Taste
• Hearing
• Touch
7

8. Motivations
Proposed solution
Projects
5 communication channels
Summary
• Sight Restricted
• Smell Impractical
• Taste
In loud & noisy environment • Hearing
In situations like those,
users’ hearing capability
is restricted.
• Touch
In social situations
8

9. Motivations
Proposed solution
Projects
Proposed solution
Summary
• Develop touch-based interaction techniques,
using vibrations/tactile icons (tactons) to
convey peripheral messages via mobile
devices in the absence of vision and audio.
• Note:
• Vibrations, Tactile icons, Tactons, Haptic
cues
9

10. Motivations
Proposed solution
Projects
Advantages of proposed solution
Summary
 Reduces visual demands associated with
using mobile applications
 Avoids disruption on users’ main tasks
 Enables eyes-free and hands-free interactions
 Provides a private communication channel
10

11. Motivations
Proposed solution
Projects
Design, evaluate & use tactile icons
Summary
• [Design]
Phase 1: Develop tactile icons for use with mobile devices
• [Evaluation]
Phase 2: Examine the effectiveness of tactile icons with
auditory distracters
• [Use]
Phase 3: A haptic personal training application
11

12. Phase 1: Develop tactile icons for use with mobile devices
Motivations
Proposed solution
Projects
Phase 1: Design
 Phase 1
 Phase 2
 Phase 3 • Goals:
Summary
– To build a low-cost & configurable tactile
display system
– To broaden design space of tactile icons by
exploring new parameters of touch
– To understand human tactile perceptual
limitations
12

13. Phase 1: Develop tactile icons for use with mobile devices
Motivations
Proposed solution
Projects
Phase 1: Tactile display system
 Phase 1
 Phase 2
 Phase 3
Summary
Nokia N95
Vibration actuators
Arduino BT Board
13

14. Phase 1: Develop tactile icons for use with mobile devices
Motivations
Proposed solution
Projects
Phase 1: Parameters of touch
 Phase 1
 Phase 2
 Phase 3
Parameters Value One Value Two
Summary Duration Long (0.8Sec.) Short (0.2Sec.)
Interval Long (1Sec.) Short (0.2Sec.)
Intensity Strong (255Hz) Weak (153Hz)
Location Volar Wrist Dorsal Wrist
16 types of tactile icons
14

15. Phase 1: Develop tactile icons for use with mobile devices
Motivations
Proposed solution Phase 1: Examine parameters of
Projects
touch
 Phase 1
 Phase 2
 Phase 3 • 20 participants ,within-subject study
Summary – Training (10 minutes)
– Trials (40 minutes)
• Each type of tactile icons repeated 3 times
• Participants can replay each icon as many times as
they feel need to
• Participants submit answer in an online form after
recognizing a tactile icon
– Post-study interview (10 minutes)
15

16. Phase 1: Develop tactile icons for use with mobile devices
Motivations
Proposed solution
Projects
Phase 1: Results
 Phase 1
 Phase 2
 Phase 3 • The tactile icons are easy to identify:
Summary – Recognition rate - > 87%
– Recognition time - <14 seconds
– Cognitive workload - below 3 in 1-5 Likert scale (1:
extremely low workload; 5: extremely high workload)
 Users can recognize 4 tactile parameters with
moderately high recognition rate and medium level
of cognitive workload
16

17. Phase 2: Examine tactons in lab with auditory distracters
[Qian, Kuber & Sears, IJHCS ’11]
Motivations
Proposed solution
Projects
Phase 2: Evaluation
 Phase 1
 Phase 2
 Phase 3 • Goal:
Summary
– To examine the effects of auditory distracters
on the use of tactile icons in mobile devices
17

18. Phase 2: Examine tactons in lab with auditory distracters
[Qian, Kuber & Sears, IJHCS ’11]
Motivations
Proposed solution
Projects
Phase 2: Experiment design
 Phase 1
 Phase 2
 Phase 3 • Three types of auditory distracters:
Summary
1. White noise
2. Street noise
3. Rock music
• Presented via headphones and speakers
• Two types of tacton pairs:
– A distinct pair
– A non-distinct pair
• 18 participants (8F, 10M, Age 19-27)
18

19. Phase 2: Examine tactons in lab with auditory distracters
[Qian, Kuber & Sears, IJHCS ’11]
Motivations
Proposed solution
Projects
Phase 2: Results & implications
 Phase 1
 Phase 2 • Different audio affects tactile perception
 Phase 3
differently
Summary
– Higher recognition accuracy and less recognition time
in quiet lab with no external sound presented
 Tactile perceptual tests should be performed under
conditions which include background audio.
– Longer recognition time and higher cognitive workload
in music, compared with street noise
– White noise resulted in increased cognitive workload,
but did not impact the tactile recognition
 White noise is not an acceptable substitute for more
realistic sound.
19

20. Phase 2: Examine tactons in lab with auditory distracters
[Qian, Kuber & Sears, IJHCS ’11]
Motivations
Proposed solution
Projects
Phase 2: Results & implications
 Phase 1
 Phase 2
 Phase 3 • The more distinct tacton pair can better resist
Summary auditory distracters
– Less recognition time to identify distinct
tacton pair
– Higher recognition accuracy to identify
distinct tacton pair
 Careful design to maximize distinctiveness of
tactons is even more important for mobile
devices.
20

21. Phase 3: A haptic personal training application
[Qian, Kuber & Sears, CHI ’10, INTERACT ‘11; Qian, Kuber, Sears & Murphy, INTCOM ’11]
Motivations
Proposed solution
Projects
Phase 3: Use
 Phase 1
 Phase 2
 Phase 3 • A mobile exertion application integrated with
Summary tactile feedback to help senior adults monitor and
adjust walking speed
• Step rate detected every 4 seconds using built-in
accelerometers
• Two tactile/auditory icons to notify users to walk
faster/slower
A pedometer & a phone holder
21

22. Phase 3: A haptic personal training application
[Qian, Kuber & Sears, CHI ’10, INTERACT ‘11; Qian, Kuber, Sears & Murphy, INTCOM ’11]
Motivations
Proposed solution
Projects
Phase 3: Use
 Phase 1
 Phase 2
 Phase 3 • Goals:
Summary
– To assess the effectiveness of tactile icons in
conveying commands in mobile applications
– To examine the impact of motion on
perception of tactile icons
– To study the change of users’ behaviors along
with the use of the mobile application
22

23. Phase 3: A haptic personal training application
[Qian, Kuber & Sears, CHI ’10, INTERACT ‘11; Qian, Kuber, Sears & Murphy, INTCOM ’11]
Motivations
Proposed solution
Projects
Phase 3: A field study
 Phase 1
 Phase 2
 Phase 3 • 3 healthy 65+ senior adults over 2-week
Summary
period
• Training in lab
– 3 types of feedback with the same rhythm
(Audio icons, Tactile icons, Audio+Tactile icons)
• Data collection methods
– Automated logging on the phone
– Participant-maintained diaries
– Interviews
23

24. Phase 3: A haptic personal training application
[Qian, Kuber & Sears, CHI ’10, INTERACT ‘11; Qian, Kuber, Sears & Murphy, INTCOM ’11]
Motivations
Proposed solution
Projects
Phase 3: Results & implications
 Phase 1
 Phase 2
 Phase 3 • Multimodal icons work better than unimodal
Summary icons
 In compromising environment or demanding
tasks, multimodal icons should be used
• Tactile icons work better than auditory icons
• Users’ preferences change depending on the
situational context
 The design of interaction system for senior
adults needs to consider the age-related
declines in hearing and touch
24

25. Motivations
Proposed solution
Projects
Summary
Summary
• [Design]
A set of identifiable tactile parameters are identified
A low-cost tactile display system is developed
• [Evaluation]
The impact of various auditory distracters is examined
Implications for mobile interface designers are provided
• [Use]
A haptic personal training application is developed
The use of tactile icons are evaluated in field
Guidance to developers and researchers is provided
25

26. Motivation
Related work
Proposed solution Questions
Research Qs
Finished studies
Upcoming studies
Timeline Huimin Qian
Ph.D. Candidate
Information Systems Department
UMBC
Email: huimin1@umbc.edu
26

27. Motivations
Proposed solution
Projects
Publications
Summary
• Towards Identifying distinguishable tactons for use with mobile
devices (ASSETS, 2009)
• Towards Developing Perceivable Tactile Feedback for Mobile
Devices. (International Journal of Human Computer Studies,
2011)
• Maintaining levels of activity using a haptic personal training
application. (CHI ,2010)
• Maintaining and Modifying Pace Through Tactile and Multimodal
Feedback (INTCOM, 2011)
• A Longitudinal Pilot Study to Evaluate Non-Visual Icons in a
Mobile Exertion Application. (Proceedings of INTERACT, 2011)
27