Join Tim Huckaby in a discussion of the lessons learned in 7+ years of design, development and delivery of interactive software driven by NUI (touch, gesture, and voice). All the bad; all the good; and the tools, tips and tricks learned along the way that will help you in designing great natural interactive usability into your own software.
Way back in 2007, Microsoft shipped Windows Vista which was touch capable, but not touch usable. With Windows 7, Microsoft introduced NUI capabilities like Multi-touch native to the OS. Windows 8 is designed for touch. Hardware vendors are now shipping large form factor multi-touch capable HD screens with native Windows 7 & 8 drivers at a consumer price point. Couple that with other form factors including mobile form factors where multi-touch is the norm and the power of WPF and WinRT then you have compelling low cost high fidelity hardware and software solutions that are revolutionizing numerous vertical industries.
This session will also update you on the types of multi-touch capable devices available right now and those coming in the immediate future. And what is now capable with .NET and WinRT in the multi-touch kiosk and interactive digital signage space.
But, frequently touch capability is not possible or just doesn’t make sense. In many airports, for instance, you do not want to touch anything. It’s just not safe. Bacteria (and other pathogens) live on touch screens in public places and are a real threat. In sterile environments like a hospital touch is legally not possible either. That is where gesture and voice controlled software comes into play nicely. With Microsoft Kinect (and other 3D cameras) innovation solutions are not only possible, but ultra-cool…and come with their own myriad of usability issues.
This demo-heavy session was designed to show you a number of real NUI solutions and how they were built and all the usability problems they evolved through.
3. Delivering the Art of Software
interknowlogy.com
1525 Faraday Ave, Suite 250, Carlsbad CA 92008
curious@interknowlogy.com
Natural User Interface (NUI). IK has been focused in
NUI for the past 7 years. We ideate, design, build, and
deploy NUI apps on all shapes of hardware, including
phone, tablet, wall, and/or table. IK knows how to build
apps where touch and gesture are the primary ways
users interact with the experience.
Enterprise Mobile Apps (EMA). IK has two focus
areas in EMA – expanding your brand and service
through massive consumer consumption, and increasing
the productivity of your employees through mobile
enablement of your enterprise software.
User Interaction Design. IK’s expertise in
User Interaction design, (IxD), as implemented
in the Natural User Interface (NUI), is heavily
focused on satisfying the needs and desires of
the majority of people who will use our
software products. Our main focus is on
natural behaviors (touch, gesture and voice
recognition) when using the software we build.
Other disciplines, like software engineering,
have a heavy focus on designing for technical
stakeholders of a project. We believe in
designing software for the actual users.
Big DATA + Rich Data Visualization. IK
visualizes data in context to the user, breaking
the barriers of standard maps and charts for
information viewing. By intelligently displaying
data, rather than using broad and generic
strokes, we enable users to glean insights not
before possible.
App Renovation. IK transforms existing apps with NUI
technology. We can re-factor any app into a NUI
experience. But our app renovation isn’t skin deep – we
go beyond transforming your app for touch and bringing
the well-travelled app pathways to the forefront. IK
rebuilds the app with a solid app architecture foundation
so that new features can be easily added, and the app
can be more thoroughly tested in less time.
InterKnowlogy (IK) is an end-to-end service delivery shop focused on bringing tech to the enterprise that is intuitive and
easy to use. IK services include ideation to transform business challenges and vague concepts into amazing solutions,
interaction design to create beautiful user experiences, to complex app development and deployment with support
services. We build our natural user interface (NUI) solutions on XAML, WPF, WinRT, HTML5 and/or Windows Phone.
5. Interactive Kiosk & Active Digital Signage Software
Actus Interactive Software
www.Actus-Software.com
The Content Defines the Experience
•Agnostic to Industry and Vertical Market
•Large Tiles / Shallow Navigation / Flexible Content
The Natural User Interface (NUI): Touch,
Gesture & Voice Controlled
Attract Mode Engages
Powerful CMS / Dynamic Synchronization
of Content
JayH@Actus-Software.com
www.Actus-Software.com
6. The Engaging User Experience & the Natural User InterfaceDelivering the Art of Software
Objectives And Agenda
The Good, the Bad & the Ugly (The Success Stories and the Failures) from 8 years
of RealWorld Experience in NUI which includes Lessons Learned the hard way
Technology Roadmap, Speculations & Predictions
WhereWe have Been andWhere we are Going
Demos,Videos, Demos,Videos, Demos….
This is a Design Session; It is not a “Stare at C#” session. It is targeted at
Technologists Generically
Guidance in NUI:Touch, Gesture &Voice Driven User Experiences-
User Engagement & User Interaction Design
7. Delivering the Art of Software
Converse World-Wide Shoe
Launch
Tim Huckaby
Chairman / Founder, InterKnowlogy & Actus Software
8. The Engaging User Experience & the Natural User InterfaceDelivering the Art of Software
Multi-Touch Capable
•Common Gestures of Touch
•Physical Object Interaction
•Virtual Object Interaction
NUI:
Gesture Capable
•Motion Based Interface
•Tracking of full-body movement, facial expression, and voice with precision
Voice Capable
•Multi-array Microphone
•Uniquely Distinguish Audio & Voices
Neural Capable
•Conscious and non-conscience interfaces
•Non Invasive
The Natural User Interface
11. Breakout Session
Delivering the Art of Software
The “Cancer App” & The “AutoCAD App”
Tim Huckaby
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
14. Delivering the Art of Software
The 7 Tenets of an Engaging UX
1. Use the Power of Faces
People love faces; We see faces where they don’t exist
We have DNA with the sole purpose of recognizing faces
Take advantage of this by putting faces on your App
Have the faces to look at important part of your App
People instinctively follow the eyes of the face.
2. Use Food, Sex and Danger
When someone sees something new, they place it into one of four categories:
1. Can I Eat it?
2. Can it Eat Me?
3. Can I Mate with it?
4. …and everything else.
If you want your App to be engaged and interesting, put it in one of the first three
categories.
15. Delivering the Art of Software
The 7 Tenets of an Engaging UX
3. Tell a story
We learn through stories. It’s how we teach our children. It’s the Bible; It’s what we see
in movies.
Stories are a big part of how information is conveyed to us.
If you have information to give to the user, put it in story form.
It doesn’t matter what medium you use (words, pictures, music), but using a story
narrative will help your user understand and retain your information.
4. Build commitment over time
Let the user choose how they want to interact with you (RSS, Twitter, Facebook)
Make sure that you don’t take advantage of the trust that is implicit in that interaction.
5. Use Natural Interactions of Objects
Natural Movements
Fluidity
Animate / Exaggerate: Slow things down so the eye can follow
16. Delivering the Art of Software
6. “Game-ification”
We Live in a Gaming Culture
The Digital Natives Expect a Contest
Even When Learning
It doesn’t fit all use cases, but if you can provide “game-fication” you’ll get engagement
7. Make it Intuitive and Easy to Use
Todays Generations, especially the Digital Natives, Don’t Read User Manuals.
Accessibility is expected in the Elderly Culture
The “Grandma Huckaby Theory”
If Grandma Huckaby walks up to a piece of software we have built and is
effective we have succeeded.
The 7 Tenets of an Engaging UX
17. Breakout Session
Delivering the Art of Software
Cincinnati Children's Hospital - The Power of Faces
Tim Huckaby
◦ Chairman / Founder, InterKnowlogy & Actus Software
18. Breakout Session
Delivering the Art of Software
EMN8 – Fast Food Kiosk
Tim Huckaby
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
19. 2
We have come a Long Way in User Interface…
The
“Green
Screen”
20Lessons from the Trenches of NUI: User Experience and Interaction Design
20. Delivering the Art of Software 2
Image Courtesy of Computer History Museum
Lessons from the Trenches of NUI: User Experience and Interaction Design
We have come a Long Way in User Interaction
21. Breakout Session
Delivering the Art of Software
Touch Capable Computing Devices Are Not New
Past
1972: PLATO IV
Touch Screen Terminal Perspective PixelApple iPad
Present Future (now)
22. The Engaging User Experience & the Natural User InterfaceDelivering the Art of Software
Multi-Touch
Inexpensive Touch Capable, Non Proprietary
Hardware
A Operating Systems designed for Multi-
Touch
A Consistent Multi-Touch API for the major
Hardware/Software Platforms
Developer Productivity in Touch Capable
Apps
The Forecast of Multi-Touch Hardware
Capability in Every New Computing Device
in the Short Term
What is Relatively New is:
Gesture & Voice
Recognition
Inexpensive 3D Cameras
Natural Movements / Natural
Interactions
Fidelity: Reliable and Dependable
Safe: anti-bacterial
Becoming a norm of using
computing systems
23. Breakout Session
Delivering the Art of Software
Barriers to Good Interaction Design in Touch on the
Windows Platform
• As Microsoft Matured their App-Dev Touch platform,
the Windows Team, Hardware OEMs and Software
ISVs ignored it
…While Apple & it’s App Builders Embraced it and
Designed for it.
• In WPF, touch is a first class citizen
• But, it’s ignored by the software control vendors
• Example: Originally, Google chrome was not built to
the Windows touch specification
• Ignored touch events
24. 2
HTML 5 Touch Events Spec
W3C Recommendation 10 October 2013
Source: http://www.w3.org/TR/touch-events/
Defines a set of low-level events that represent one or more points of
contact with a touch-sensitive surface
Changes of those points with respect to the surface and any DOM elements
displayed upon it (e.g. for touch screens) or associated with it (e.g. for
drawing tablets without displays).
Addresses pen-tablet devices, such as drawing tablets, with consideration
toward stylus capabilities.
HTML5 Touch Events Extensions
W3C Working Group Note 31 October 2013
https://dvcs.w3.org/hg/webevents/raw-file/tip/touchevents.html
Lessons from the Trenches of NUI: User Experience and Interaction Design
28. The Engaging User Experience & the Natural User InterfaceDelivering the Art of Software
Wireframe vs. Comp / Design
29. The Engaging User Experience & the Natural User InterfaceDelivering the Art of Software
User Interaction Design (IxD)
User Interaction Design:
• Extends Beyond the Screen
• Shapes Digital Things for People’s Use
• Is the practice of designing interactive digital products,
environments, systems, and services.
• Has a Main focus on Human Behavior and Interaction
Distinction:
• User Interaction Design: heavily focused on satisfying the needs
and desires of the majority of people who will use the product
• Application Programming: heavily focused on designing for
technical stakeholders of the product
• User Experience Design: is typically artwork, look and feel
30. The Engaging User Experience & the Natural User InterfaceDelivering the Art of Software
User Interaction Design Sketch
31. Delivering the Art of Software
Good User Interaction Design:
Fluid Layout, Naturally Behaving Objects
Large Form Factors demand
natural behaviors
Physical & Virtual Objects:
Some Touch Capable Hardware has
unique features
Software Physics Engines are
readily available in open source
Interaction Design & The User
Experience is “never done”
…It can always be improved with
User Interaction Design
Tablet Computing paved the
way…
32. Breakout Session
Delivering the Art of Software
Pixelsence Craps, LOC
Tim Huckaby
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
33. Breakout Session
Delivering the Art of Software
Fluid Layout / Natural Data Bench
Tim Huckaby
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
34. Breakout Session
Delivering the Art of Software
Good User Interaction Design:
Software Designed for Touch
Large Controls
Intuitive
Smoothing; Fidelity
Correction
Ubiquitous Touch
Gestures like
“pinch”
35. Breakout Session
Delivering the Art of Software
NASA Marsbound – Designed for Touch
Tim Huckaby
Founder / Chairman,
InterKnowlogy
Founder / CEO, Actus
Software
Microsoft RD & MVP
36. Breakout Session
Delivering the Art of Software
3D Modeler – Designed for Touch
Tim Huckaby
Founder / Chairman,
InterKnowlogy
Founder / CEO, Actus
Software
Microsoft RD & MVP
37. The Engaging User Experience & the Natural User InterfaceDelivering the Art of Software
Developer Pacification – “I can’t
handle it unless I can stare at code.”
Check out the
method level
support for Rotation
This is why I should
have paid attention
in high school
calculus..
38. The Engaging User Experience & the Natural User InterfaceDelivering the Art of Software
Kinect for Windows
$250
http://www.microsoft.com/en-us/kinectforwindows/Multi-
Platform
.NET; awesome SDK and samples
Leap Motion
$70
https://leapmotion.com
Multi-Platform
SoftKinetic
http://www.softkinetic.com
$249
Distance and Near Mode
Multiple Products; In the space for over a decade
Panasonic
Works outside in direct sunlight -$2,600
3D Cameras – Major Players
39. Breakout Session
Delivering the Art of Software
User Interaction Design with Gesture:
From Bad to Good
Using your hands as the
mouse
Intuitive
Smoothing; Fidelity
Correction
40. Delivering the Art of Software
• Kinect for Windows Human Interface Guidelines v1.8.0
•
•
• roadmap to building exciting human-computer interaction solutions you
once thought were impossible.
• “Living Document”- regularly updated it to reflect both ongoing findings
and the evolving capabilities of Kinect for Windows.
Human Interface Guidelines
Guidance Published by the K4W Team
41. Breakout Session
Delivering the Art of Software
Before & After: IK Physical Therapy, IK Touch-less Operating Room
Tim Huckaby
Founder / Chairman,
InterKnowlogy
Founder / CEO, Actus
Software
Microsoft RD & MVP
42. Delivering the Art of Software
The Future of Interaction Design with Gesture & Voice:
Facial Recognition, Identity, Demographics
Tim Huckaby
◦ Chairman / Founder, InterKnowlogy & Actus Software
43. User Interface Innovation: What’s on the Horizon?Delivering the Art of Software
Brand new device
• Not an upgrade
• Replaces the prime sense device purchased by Apple
• 2 to 3 times better functionality across the board
• Specs that are unmatched by any current 3D camera
• Fidelity so good it’s capable of detecting human pulse
Kinect for Windows v2
All-new:
• Active IR camera enables it to see in the dark. And by using advanced three-
dimensional geometry, it can even tell if you’re standing off balance.
• Multi-microphone array, the advanced noise isolation capability lets the
sensor know who to listen to, even in a crowded space.
Officially: “Kinect for Windows v2 worldwide launch next summer.”
Unofficially: “…we’ll ship in spring.”
44. Delivering the Art of Software
interknowlogy.com
1525 Faraday Ave, Suite 250, Carlsbad CA 92008
curious@interknowlogy.com
NUI provides a number of challenges to good User Interaction
Design:
• Large and Intuitive Controls
• Large Form Factors
• Naturalizing Interactions withVirtual and Physical Objects
• Fidelity and environment weaknesses of 3D Cameras
The hardware, tools, plumbing and platform for good user Interaction Design in
Applications are powerful and available now.
…and will just continue to get better, cheaper, powerful and more available
The concepts of engagement and Good User Interaction Design change the way we
design and build software in many Use Cases.
…when the goal is better user experiences
Summary
45. Delivering the Art of Software
More info on InterKnowlogy:
◦www.InterKnowlogy.com
◦www.vimeo.com/InterKnowlogy
More info on Actus Software:
◦www.Actus-Software.com
About Tim Huckaby…
Founder/Chairman, InterKnowlogy
Founder/Chairman, Actus Software
Microsoft® Regional Director – Southern California
Microsoft® .NET Partner Advisory Council Founder /
Member
Microsoft® MVP – Kinect for Windows
INETA Speaker – International .NET Users Group
Association
Board Member: Association for Competitive Technology
Board Member: DevIntersection
Board Member: Microsoft Healthcare Partner Advisory
Board
Author / Speaker
www.TimHuckaby.com
Contact Tim:
E-mail:
TimHuck@InterKnowlogy.com
TimH@Actus-Software.com
Phone:
• Office: 760-444-8640
• Mobile:: 619 990 9200
Blog:
http://team.interknowlogy.com/blogs/timhuckaby
FaceBook:
http://www.facebook.com/Tim.Huckaby.Carlsbad
LinkedIn:
http://www.linkedin.com/in/timhuckaby
Twitter: @TimHuckaby
Tim Huckaby