Gesture design document overview and touch surface technologies

Gesture-Based Interaction Design
Lane@LaneKuhlman.com


Overview
 Overview
State of the Art
State of the Industry

Overview gesture sensing technologies currently in the market
Gesture
Open Source

Possibilities and diIemmas within the industry
Usability


Elements of gestural communication
Demo
Great Literature

Free applications and advice for building TUIs and touch tables

Usability study comparing single-user to multi-user interaction

Demonstration a DIY TUI Using Flash, FLOSC & Reactivition

Some great sources for more in depth information


Touch Surfaces -Video Based
Overview
 State of the Art

Microsoft Surface
Gesture
• Runs on Vista OS
Open Source
• Develop using .Net
Usability
• Bluetooth Capabilities
Demo
• $15,000 for table
Great Literature
• $15,000 for development kit


Overview

Ion Touchlight
Gesture
• Tranlucent projection surface
Open Source
• Uses 3 Cameras & includes calibration
Usability
• Can copy an image held on the surface
Demo
• Vertical Display
Great Literature
• Forerunner to Surface


Overview

Perceptive Pixel
Gesture
• FTIR - Internal Reflection
Open Source
• Jeff Han - perceptivepixel.com
Usability
• Pressure sensing
Demo
• Patents Pending
Great Literature
• Not disclosing price


Touch Surfaces -Video Based?
Overview

Multitouch, Finland - Cellular
Gesture
• Expandable
Open Source
• Not disclosing price
Usability
• 60 FPS
Demo
• Differentiates Multiple
Great Literature
Fingers & Hands


Touch Surfaces -Capacitive
Overview

I-Phone
Gesture
• Use the capacitance of skin - won't work with gloves
Open Source
• An accelerometer enables flipping from portrait to landscape view
Usability
• An ambient light sensor adjust screen brightness automatically
Demo
• Infrared sensor disables touch screen when near the head
Great Literature


IR Camera & Accelerometers
Overview

Wii
Gesture
• Low Cost Accelerometer
Open Source
• Less expensive processor than competitors
Usability
• Small Infrared Camera
Demo
• Encouraging new groups to start gaming
Great Literature
• Uses pantomime gestures with baton for things
like sword fighting, archery, and other gestures


Accelerometers
Overview

Think/Thing OUiP!
Gesture
• Baby friendly motion sensing
Open Source
• Visualize your daily jog
Usability
Demo
Great Literature


High Res. Mocap
Overview

Polhemus FaskTrack
Gesture
• Electromagnetic
Open Source
• 6 Degrees of Freedom
Usability
• Small senors provide high resolution tracking
Demo
• No Occlusion
Great Literature

Vicon Motion Systems - Cameras
• Can expand to include tons of cameras
• High Resolution
• Occlusion can occur, but is less likey with more cameras


Data Gloves
Overview

4D Dataglove
Gesture
Gestural Character Control
Open Source
This work is investigating how gestures can
Usability
be used to build and conduct virtual stories.
Demo
It is comprised of two simulations. In the first,
Great Literature
you can direct a character using programmed
gesture signals, some of which are loosely based
on modified American Sign Language. Using
hand gestures you can tell Eva,
a 3-D Virtual Character, when to walk, run,
sit, stand, jump and open doors.The second
simulation involves an interface that allows you
to use the glove to navigate through the joint
hierarchy of a virtual mannequin and manipulate
its posture by selecting and reorienting it’s
joints.
The project was displayed on
Good Day Columbus in May


RFID Technology
Overview
Digital Seeds
Gesture
Open Source
Usability
Demo
Great Literature


TUI Table
Overview
Geometry Drawing Table
Gesture
Open Source
Usability
Demo
Great Literature


Overview
State of the Art
 State of the Industry
Difficulties
Gesture
Open Source
Over pricing limits the audience and
Usability
the amount of people who can
Demo
develop for new platforms
Great Literature

Companies are quot;over protectivequot;,
which slows development and
innovation

The quot;WOWquot; factors gets in the way
of the quot;WHYquot;

While most of these technologies
are facinating, sometimes the tech-
nologies fail to demonstrate the
ability to fulfill a need that cheaper
technologies cannot


Overview
State of the Art
Patent Frenzy
Gesture
Open Source
quot;Multitouch technology itself
Usability
is far from being an Apple
Demo
invention. Jeff Han wowed
Great Literature
the crowd at TED in 2006
with a demo of his multitouch
research, and Bill Gates demo'd
Microsoft's Surface Table in
early 2007, months before the
iPhone was released. Even Apple
CEO Steve Jobs reportedly used
Microsoft's work with tablet
PCs as the jumping-off point for
iPhone development.quot; - Wired
Magazine

NOTE: Myron Krueger was using
pinching as a gesture by 1982


Overview
State of the Art
Open Source
Gesture
Open Source
• Lots of Sharing is going on
Usability
Demo
• People seeking new quot;gesture
Great Literature
librariesquot; and openly sharing
about new standards arising
through practice
www.interactivegestures.com

• Artist and Developers have the
opportunity to take advantage
of computer vision frameworks
developed by others, so they can
focus on visual design, UI, UX
and usability issues


Gesture Classifications
Overview
State of the Art
 Gesture
Open Source
Usability
Demo
Great Literature

Diectic & Pictographic Kinetographic Beat Gestures
Spatial Gestures Gestures
depict the shape depict bodily action
Movement used to beat time
of objects, people, that may be
Gestures and create rhythm
depict spatial boundaries or performed with or in speech
position and regions. without reference
relationships to an object at denote divisions in
between objects includes drawing in hand timing and pace
& people (real or the air or on a emphasis on
imagined) surface speech

Path & Location Shape Motion Pace


How it works
Overview
State of the Art
State of the Industry Scientific American
 Gesture How it Works:
Open Source Multi-touch Surfaces
Usability Explained
Demo June 18, 2008
Great Literature
Hands-On Computing:
How Multi-Touch
Screens Could Change
the Way We Interact
with Computers and
Each Other:
The iPhone and even
wilder interfaces could
improve collaboration
without a mouse or
keyboard

By Stuart F. Brown


Open Source - Finger Tracking
Overview
State of the Art

Touchlib
Gesture
 Open Source
• Touchlib is a library for creating multi-touch
Usability
interaction surfaces.
Demo
Great Literature
• Track IR Blobs and has built recognition of events
like FLOSC - quot;a standalone
'finger down', 'finger moved', and 'finger released' application written in
Java that sends and
• Sends messages to several programs that recieve receives OSC packets
OSC protocol ( sent via the UDP port ) via UDP, translates
bidirectionally between
NOTE: You can use Touchlib only on WIndows and binary OSC packets and
it requires you to download a application called an XML encoding of OSC
FLOSC before you can use it with FLASH packets, and sends and
receives XML entities
Visit Ben Chun's website for a free download: via TCP in a way that's
http://www.benchun.net/flosc/ compatible with Flash's
http://www.transmote.com/flosc/ XMLSocket feature.quot;


Open Source - Symbol Tracking
Overview
State of the Art

Reactivison
Gesture
 Open Source
• A computer vision engine that tracks the position
Usability
and rotation of specialized markers on a table top
Demo
surface
Great Literature

• Software, markers, and a calibration template are
free to download. Calibration is simple and built
into the software

• Now also includes finger tracking

• Calibration is simple and built into the software

NOTE: This also requires you to download FLOSC

Visit http://mtg.upf.edu/reactable/?software


Open Source - Symbol Tracking
Overview
State of the Art
Gesture
 Open Source
Usability
Demo
Great Literature


Open Source - Computer Vision
Overview
State of the Art

Processing
Gesture
 Open Source
• Includes Code Libraries for computer
Usability
vision techniques such as shape tracking,
Demo
blob tracking, & face detection
Great Literature

• Includes libraries for Reactivision and
the JMyron
( formerly known as WebcamXtra in
Director )

Visit http://processing.org/reference/
libraries/
scroll down to the computer vision section


Usability Research
Overview
State of the Art

Purpose
Gesture
Open Source
• The study was conducted using
 Usability
the DNA Workbench, and interface
Demo
designed to teach about DNA base
Great Literature
pairing

• QUESTION: Do people learn more
when multiple people are interacting
with the table simultaneously than they
do when they are interacting alone?

• The study was conducted to
determine the benefits single-user vs
The interface was created by:
multi-user interaction and to explore
http://accad.osu.edu/~aprice
unique design considerations related to
and was temporarily on display
multi-user design and the multi-user
at COSI
experience.


Usability Research
Overview
State of the Art

Methodology
Gesture
Open Source
• 6 test groups total - 3 multi-user, 3 single-user
 Usability
• Each group was told to interact until they felt like they understood the
Demo
rules of the system and were able to explain it, groups had to come to
Great Literature
consensus

• Afterwards each person was individual asked to explain the rules of the
system and they were measured on the detail and accuracy of their answers
Did they understand:
• Nucleotide pairing
• Mutation vs non-mutation
• The role of the editase

• Each participant took a survey upon completing the study in which they
compared their experience learning with the DNA to previous experiences
where they might have learned about DNA


Usability Research
Overview
State of the Art

Results
Gesture
• When measured against five learning objectives, individuals within
Open Source
groups performed above average while individuals acting alone performed
 Usability
either at or below the average.
Demo
Great Literature
• The time each individual required to feel they had adequate
understanding of the system increased by an average of 3 minutes for
individuals acting alone

• The DNA Workbench has sounds that occur when nucleotides and editase
randomly collide. At times it was diffifult to tell whether the sounds they
heard were related to their actions or action on the other sides of the table.

• After several errors the interface went into quot;Auto-Completequot; Mode.
Individual all went into auto at some point, while member of groups did not

• Members of groups gave for thourough and accurate answers than did
participants who acted alone. They also gave descriptions with more
confidence.


Usability Research
Overview
State of the Art

Findings
Gesture

 People who learn with multi-user interfaces are likely to learn more
Open Source
 Usability information at a faster rate than individual interacting alone
Demo
Great Literature
 People are more aware of issues, problems and questions and
when these matters arise there is more opportunity to debate and share
information related to the topic

 Members of a group are more confident in the information they learn
and are likely to take action with the knowledge they have more quickly.

 Multi-User interfaces cannot use simple sound errors unless they
associate a visual cue with the source of the error. Within gaming scenarios
it is difficult to tell who or what may have caused the error.


DIY TUI
Overview
State of the Art

Step 1 - Download & Install
Gesture
Open Source
• Visit: http://mtg.upf.edu/reactable/?software
Usability
• print Fiducial Markers and calibration grid
 Demo
Great Literature


DIY TUI
Overview
State of the Art

Step 1 - Download & Install
Gesture
Open Source
• Visit: http://www.benchun.net/flosc/
Usability
• Also look at this tutorial: http://www.transmote.com/flosc/
 Demo
Great Literature


DIY TUI
Overview
State of the Art

Step 2 - Get a camera
Gesture
Open Source
• Camera need to have a firewire connection
Usability
• It helps to have wide angle so you can get
 Demo
close to the table
Great Literature
• Some camera have detachable lenses
• Some nice board cameras with detachable
lenses can be found at:

http://www.unibrain.com

• To create an infrared camera, cover the lens
2-3 pieces of exposed film
• It is better to have a black and white camera
• Once you camera is positioned, you can calibrate it using the grid.


DIY TUI
Overview
State of the Art

Step 3 - Build your table
Gesture
• Before going full tilt, stack some plexi on books and use clamps to
Open Source
positions lights. Test out several scenarios until you get the best lighting
Usability
and vision from your camera.
 Demo
Great Literature
• Wait until you have everything working
well before you build anything to spec.
It's good to have some wiggle room.
Consider the lighting scenario of the
location where the table will eventually
live. Is it going to have a lot of natural
light? You may need to invest in some
shades.

• For projection underneath the table try
using a mirror tilted at a 45 degree angle to increase the throw. Note that
tilting projectors can cause the bulbs to burn out much faster. Some nice
projectors exist that allow you to throw from short distances. These are
ideal for table top interfaces where there isn't a lot of room for a projector.


DIY TUI
Overview
State of the Art

Step 4 - Lighting
Gesture
Open Source
• Lighting is the most difficult element of creating a table. In order for the
Usability
camera to see finger tips or markers while not recieving interference from
 Demo
a projector beam, the camera needs to look at a different spectrum of light,
Great Literature
thus the need for IR.

• It is important to have very smooth
even lighting in order to get quality
sensitivity and optimal performance.
Hot spots will create glitches for most
open source computer vision engines.
Try to use ambient or diffused lighting.
Getting the lighting right is not a simple
task.

• Sanding the tips of LED lights can make
light more diffused


Great Gesture Books
Overview
State of the Art
Gesture
Open Source
Usability
Demo
 Great Literature

Saffer, Dan Buxton, Bill.
Designing Gestural Inter- Sketching User Ex-
faces. perince: Getting the
COMING SOON! design right and the
First Chapter and Intro- right design. Boston,
duction available at: MA: Elsevier, 2007.
http://www.designingges-
turalinterfaces.com/


Great Gesture Books
Overview
State of the Art
Gesture
Open Source
Usability
Demo
 Great Literature

Kendon, Adam. Goldin-Meadow, Gesture-Based
Gesture:Visible Susan , Hearing Communication in
Action as Utterance. Gesture: How our Human-Computer
(Boston: MIT Press, Hands help us Interaction. 5th In-
2007) 2006. Think, Cambridge, ternational Gesture
MA: Belknap Press Workshop, GW
of Harvard Univer- 2003, Genova, Italy,
sity Press, 2003; April 15-17, 2003

Thanks You!
Lane@LaneKuhlman.com

Gesture design document overview and touch surface technologies

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (13)

Similar to Gesture design document overview and touch surface technologies

Similar to Gesture design document overview and touch surface technologies (20)

Gesture design document overview and touch surface technologies