This document discusses natural user interfaces (NUI) and provides an overview of NUI technologies including the Kinect, Wiimote, and Surface. It describes how these devices track motion and gestures using technologies like infrared lasers and cameras. Examples are given of NUI uses in media, operating systems, and science. Tools for developing with these devices like the Kinect SDK are also mentioned. The document concludes with the direction of the author's own work building an API and gesture language to simplify NUI development.

1. NUX
in a
nutshell

3. agenda
• Introduction
• What is NUI/NUX
• NUX In the media and the wild
• Real-world devices
• Tools, SDK’s, …
• Where is this going?
• Demo using Kinect and Wiimote

4. about mike
• Principal global technologist, Advanced Technology
Services, SunGard Global Services
• Technology focus on .NET, cloud, parallel systems, rich
interfaces and NUX
• mike@heydt.org, michael.heydt@sungard.com
• @mikeheydt
• 42spikes.com

5. 55
Advanced Technology
What is Advanced Technology?
SunGard Consulting Services Approach
Typical Client Engagements
 Our consultants offer deep domain experience and a broad set of leading edge technology
skills, allowing companies to optimize their business processes
 We use a variety of Agile project management and execution techniques to facilitate quick
delivery and reduce risk
 We take an objective approach in the selection of tools and products, and are comfortable
working with a variety of vendor products and platforms
 We have applied our advanced technology skills to provide solutions to customers in finance,
energy and media sectors
 Typical assignments involve deploying highly skilled specialists with relevant domain
knowledge who work with customers in a collaborative way to design and build custom
solutions
MapReduce/Hadoop
Complex Events Processing
GPU Mathematical Pricing
Mashups
Web 2.0
Cloud & Grid Computing
Functional Languages
Mobile Applications
RIA (Silverlight/Flex)
Semantic Web
REST/JSON
Grails

7. opening questions…
• Who here has used a Kinect on the Xbox?
• Who has programmed a Kinect?
• Who has used a WiiMote on the Wii?
• Who has programmed a wiimote in their application?
• Who has seen Minority Report?

8. what is nui/x?
• In computing, a natural user
interface, or NUI, is the common
parlance used by designers and
developers of computer interfaces
to refer to a user interface that is
effectively invisible, or becomes
invisible with successive learned
interactions, to its users.
• NUI Covers the class of interfaces
where the user is enabled to
interact with the computer in a
more natural manner.

9. genesis
• Conceptually has been thought of for several decades
• Made popular by Minority Report
• iPad has changed the game with single-task gestural
apps
• Xbox Kinect has broken open the flood gates for NUX
• NUX is starting to show adoption in operating systems
through the use of tablet and trackpad gestures

10. in the media and the wild
• Xbox
• Operating systems
• Movies and television
• Science

11. XBox

12. os and mobility: osx lion

13. os and mobility: Windows 8

14. tv: Caprica

15. movie: District 9

16. movie: Disclosure

17. movie: Minority Report

18. movie: Minority Report

19. movie: Minority Report

20. science: world wide telescope

21. other: smart cubes

22. real world devices
• Surface
• Kinect
• PrimeSense
• Wiimote
• Tablets and SmartPhones

23. surface

24. surface capabilities
• Multi-user
• Multi-touch
• Also camera based identification and tracking
• Built in wireless to devices

25. surface 2.0

26. kinect
• Laser Depth Imager
• Video Camera
• Microphone Array
• Very XBox Specific Interface (non-standard USB)
• Microsoft SDK, non-commercial license
• $150

27. kinect: an evolution

28. primesense
• Only laser depth finder
• Standard USB
• OpenNI drivers
• Semi-open source
• commercial license

29. johnny 5

30. wiimote

31. tablets and smartphones
• Have driven
adoption of NUX
• Will be an integral
“augmentation” to
NUX

32. elements of nux
• Skeleton Tracking
• Gesture (Body, fingers) Identification
• Voice Recognition

33. the how of the kinect
• Perform initialization gesture (helps, but not always
necessary)
• You are scanned and tracked by an IR laser
• Depths are measured to the millimeter at 320 x 200 +
• From depths skeletons are identified and tracked

34. Kinect SDK
http://research.micro
soft.com/en-
us/um/redmond/proj
ects/kinectsdk/

35. OpenNI
• http://www.ope
nni.org/
• Pseudo-open
source

36. data from the kinect
• Multiple streams of data
• Video
• Audio
• Depth
• Skeletons

37. kinect - the invisible

38. depth mapping
• Distance to the millimeter
• Used to create maps for
“skeletons”

39. skeleton tracking
• Built from the depth map
• 1..n skeletons
• Done in software not the
hardware

40. the “psi” gesture

41. hand and finger tracking
Not a part of any SDK

42. the how of wiimote
• Connects to PC with Bluetooth
• With .NET, use WiimoteLib
(http://wiimotelib.codeplex.com/)

43. Wiimotelib
http://www.ted.com/talks/johnny_lee_demos_wii_remo
e_hacks.html

44. Thoughts on the Wiimote
• An interesting device, but…
• It’s first gen
• Looked good prior to the Kinect
• All capabilities can be duplicated with Kinect (point
tracking) and a cell phone (accelerometer, gyro)
• And a cell phone can provide additional uses such as
robust data entry

45. speech
• Kinect has a very good set of microphones
• Speech recognition is not in the kinect - use the .NET
libraries
• Must be bridged with Microsoft Speech Platform SDK
• I can’t stress this enough – this is excellent
augmentation

46. gesture recognition
• Not built into any of the
devices
• Typically implemented with
Dynamic Time Warping
algorithms (DTW)
• Record sequences of skeleton
locations, match against live
streams

47. difficulties
• Many devices
• Immature libraries
• Small body of work
• Many API’s, even for each
individual device
• Kinect is non-standard
USB
• All devices are separate
and don’t interoperate
• All work on a single
system – No support for
moving physical location
• No standard pattern
language for NUX
• No “canned” gesture
engines

48. what i’m building
• NuiDotNet (nuidotnet.codeplex.com, nuidot.net)
• Abstracts devices from the application
• Allows configuration and location based construction
of services
• Multiple and prioritized services and interactions
mapped to application functions
• Gesture engine and domain specific language for
gestures

49. directions
• Define useful set of gesture patterns for data
manipulation
• Build finger / hand tracking capabilities
• Make the DTW gesture engine robust
• Tracking of specific parts of the body
• Architecture and software to plug into business
applications
• Provide a library of pluggable UI elements and gestures

50. uses for business
• Interactive data manipulation and visualization
• Free-form visual application development
• Media presentations
• Home control
• Many still to be determined

51. demo and the tools
• Hardware: Kinect, WiiMote
• Libraries: Kinect SDK, Wiimotelib
• Language and GUI: C# / WPF

52. resources
• ^ Buxton, Bill (2010-01-06). "CES 2010: NUI with Bill Buxton". Microsoft Research.
• http://www.microsoft.com/surface/en/us/default.aspx
• http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/
• http://www.youtube.com/watch?v=1-tMp4WkQjA
• http://openni.org/
• http://silverlightkinect.codeplex.com/
• http://mono-mig.sourceforge.net/
• https://42spikes.com

53. Resources
• http://en.wikipedia.org/wiki/NUI
• http://kinectdtw.codeplex.com/
• http://en.wikipedia.org/wiki/Dynamic_time_warping
• http://tinyurl.com/6a64sqr
• http://wiimotelib.codeplex.com/

54. Resources
• http://ctmarionette.wordpress.com/ (marionette)
• http://www.ted.com/talks/johnny_lee_demos_wii_remot
e_hacks.html
• http://research.microsoft.com/en-
us/um/redmond/projects/kinectsdk/docs/speech_walkth
rough.pdf
• http://www.microsoft.com/download/en/details.aspx?dis
playlang=en&id=14373

55. resources
• http://www.springer.com/cda/content/document/cda_do
wnloaddocument/9783540740476-c1.pdf?SGWID=0-0-
45-452103-p173751818
• http://www.youtube.com/watch?v=tlLschoMhuE&featur
e=player_embedded
• http://www.youtube.com/watch?v=lCuItHQEgEQ&NR=
1

56. resources
• http://www.microsoft.com/presspass/press/2011/jan11/
01-06mssurfacesamsungpr.mspx
• http://blogs.oracle.com/mprove/entry/reipadsnui
• http://channel9.msdn.com/Shows/HanselminutesOn9/H
anselminutes-on-9-Johnny-Lee-on-Computer-Vision