As technology advances, more sophisticated ways of interfacing with it are emerging. Even though new tech strives to make our apps more intuitive and easy to use, designing interfaces for those apps is not quite as straight forward. We’ve learned a few rules and “gotchas” when working with gesture cameras that can help to make apps that use them easy and fun to use. In this talk Justin described: 1. Different data types you can get from Intel® RealSense™ and how to get them 2. Designing an interface for a gesture camera 3. Using your hands, face, and voice as an interface

1. Using Intel’s RealSense to create
games with Natural User Interfaces
Justin Link, Chronosapien Interactive

2. Overview
RealSense Overview
• Camera types
• System requirements
• SDK features
• Getting data
• Unity toolkit
Our Use Cases
• Getting the data
• Challenges
• What we did
Space Between
The Risen
Photo booth

3. RealSense Overview
Camera Types
F200 R200 SR300

4. RealSense Overview
System Requirements
• 4th generation (or later) Intel® Core™ processor
• 8GB free hard disk space
• Microsoft Windows* 8.1 or 10 OS 64-bit

5. RealSense Overview
SDK Features
• Hand tracking/gesture recognition
• Face tracking/recognition
• 3D Scanning
• Background removal (user segmentation)
• Object tracking
• Speech recognition

6. RealSense Overview
Getting Data
• The first step for getting any data from the RealSense SDK is creating a
session instance:
https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?pxcsession.html

7. RealSense Overview
Getting Data
• Next you can create an instance of the module you want to use and
(optionally) configure it:
https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_hand_initializing_and_configuring.html

8. RealSense Overview
Getting Data
• Once the module is created and set up, you can poll data:
https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_hand_implementing_the_main_loop.html

9. RealSense Overview
The Unity Toolkit
• Easy to use
• Quick to set up
• Covers a lot, but has limited functionality compared to the SDK
• Great for prototyping
https://www.youtube.com/watch?v=SK50ncrYq4w

10. Hand trackingin space between

11. Hand Tracking in Space Between
What is Space Between?
• Underwater themed game that lets you play as five different sea
creatures or machines
• Uses RealSense to let players control characters with their hands and
head
• Unique input for each character

12. Hand Tracking in Space Between
Getting Hand Data from RealSense
Let’s check out some of our code

13. Hand Tracking in Space Between
Challenges
User understanding of the technology
• Tracking has bounds
• Tracking requires a line of sight
• Tracking is based on silhouettes

14. Hand Tracking in Space Between
Challenges
“Gorilla Arm”, or Consumed Endurance
• Constant motion fatigues users’ arms after
about 60 seconds
• Might be ok for exercise, but isn’t that fun

15. Hand Tracking in Space Between
Challenges
Data is noisy
• Can create false positives
• Limits precise actions
• Can look visually unappealing

16. Hand Tracking in Space Between
What We Did
• Cursor for every game to always show the tracked hand/head position
• Game instructions to tell players what is being tracked and how it’s
used
• Depth feed of the player during gameplay
• Instructions/tutorials accessible at all times

17. Hand Tracking in Space Between
What We Did
Let’s jump into Space Between

18. VoiceRecognition in The Risen

19. Voice Recognition in The Risen
What is The Risen?
• A tech demo we created for the RealSense App Challenge
• Let’s players play as a skeleton lord who commands skeletons with
their voice
• Let’s players be IN the game via user segmentation
• Let’s players use their hands to help their skeleton minions

20. Voice Recognition in The Risen
Getting Speech Data from RealSense
Let’s check out some of our code

21. Voice Recognition in The Risen
Challenges
Recognition is based on the player’s
environment
• Loud environments kill recognition
• Quiet environments may prevent talking

22. Voice Recognition in The Risen
Challenges
Recognition is slower than players’
expectations
• RealSense uses a local solution
• Expectations are based on cloud
solutions

23. Voice Recognition in The Risen
Challenges
Recognition is often inaccurate
• False positives
• True negatives

24. Voice Recognition in The Risen
What We Did
• Step by step tutorial
• UI feedback for when speech was detected
• UI feedback for speech too loud or too quiet
• Microphone volume slider

25. Voice Recognition in The Risen
What We Did
Let’s check out The Risen

26. User Segmentation in our Photo booth

27. User Segmentation in our Photo booth
What is The Photo booth?
• A photo booth that puts users into a virtual environment without the
need for green screen
• Uses gestures to control the app
• Has print and share functionality

28. User Segmentation in our Photo booth
Getting Segmentation Data from RealSense
Let’s check out some of our code

29. User Segmentation in our Photo booth
Challenges
Segmentation quality is highly dependent
on lighting
• Be careful with external IR sources
• Dark environments create grainy color
feeds

30. User Segmentation in our Photo booth
Challenges
Segmentation often cuts off parts of the
user
• This is a result of the method of getting a
depth image
• Hair almost always gets lost
• Reflective materials can be invisible

31. User Segmentation in our Photo booth
Challenges
Segmentation only supports a single user
• This is mostly true, but sometimes
people can trick the software into
thinking they are one “blob”

32. User Segmentation in our Photo booth
Challenges
Color/lighting of the user often does not
match the virtual environment
• Not necessarily a fault of the SDK, but
still a challenge when making an app

33. User Segmentation in our Photo booth
What We Did
• Used bright lights to illuminate the user
• Used curtains/walls to block external light sources
• Used hats to hide chopping users’ hair off
• Used a plugin to control segmentation color, brightness, and contrast

34. Voice Recognition in The Risen
What We Did
Let’s jump into the photo booth

35. Wrapping Up
Take Away’s
• Users must be trained to understand how the camera works
• User feedback is critical to make up for the loss of haptic feedback
• Giving users control over the camera’s properties can improve
recognition and overall UX
• This is new tech – it has many quirks and working through them will
be a big part of the work you do

36. Wrapping Up
References
• RealSense SDK download page - https://software.intel.com/en-us/intel-realsense-sdk/download
• RealSense Documentation - https://software.intel.com/sites/landingpage/realsense/camera-
sdk/v1.1/documentation/html/index.html
• User Feedback for NUI’s with RealSense P1 - https://software.intel.com/en-us/articles/user-
feedback-for-natural-user-interfaces-with-intel-realsense-technology-part-1
• User Feedback for NUI’s with RealSense P2 - https://software.intel.com/en-us/articles/user-
feedback-for-natural-user-interfaces-with-intel-realsense-technology-part-2
• “Gorilla Arm”, or Consumed Endurance -
https://pdfs.semanticscholar.org/d852/e9749467b06f6cd516ea3f6c894408dea6d4.pdf
• Space Between on the RealSense App Showcase - https://appshowcase.intel.com/en-
us/realsense/space-between?cam=f200

37. Wrapping Up
Contact
• http://www.chronosapien.com/
• https://www.reddit.com/r/chronosapien
• @jlink__ + @_chronosapien
• jlink@chronosapien.com
• rclark@chronosapien.com

38. Thanks for Watching!
Time for QA!