This document provides an overview of the Kinect SDK, including hardware requirements, the development environment, image and depth APIs, skeletal tracking, audio processing, speech recognition, and sample applications. It discusses the Kinect sensor's depth sensors, RGB camera, and microphone array, and how to access depth data, skeletal data, and audio streams from the Kinect in code. It also introduces APIs for images, depth data, skeletal tracking, audio, and speech recognition.

1. SDK Overview Dr David Brown Microsoft Technology Centre

2. Agenda Installing and Using the Kinect Sensor Setting up your Development Environment Camera Fundamentals Working with Depth Data Skeletal Tracking Fundamentals Audio Fundamentals

3. Hardware Computer with a dual-core 2.66-GHz or faster processor 2GB RAM Windows 7-compatible graphics card that supports DirectX 9.0c

4. Kinect Sensor 3D DEPTH SENSORS RGB CAMERA MOTORIZED TILT MULTI-ARRAY MIC

5. Development Environment Microsoft Visual Studio 2010 Express or other Visual Studio 2010 edition .NET Framework 4.0 SDK, http://research.microsoft.com/kinectsdk DirectX Samples Microsoft DirectX® SDK - June 2010 or later version Current runtime for Microsoft DirectX® 9 Speech Samples Microsoft Speech Platform Runtime, version 10.2 (x86 edition) Microsoft Kinect Speech Platform (US-English version) Microsoft Speech Platform - Software Development Kit, version 10.2 (x86 edition)

6. Image API

7. Demo

8. Depth Image Array of bytes (ImageFrame.Image.Bits) Left to right, top to bottom Represents distance for pixel in mm (850 to 4,000mm) 0 means unknown Shadows, low reflectivity, and high reflectivity among the few reasons Player Index 0, No player 1, Skeleton 0 2, Skeleton 1

9. Depth Data 2 bytes per pixel (16 bits) Depth (Distance per pixel) Bitshiftsecond byte by 8 Distance (0,0) = (int)(Bits[0] | Bits[1] << 8); DepthAndPlayer Index (Includes Player index) Bitshift by 3 first byte (player index), 5 second byte Distance (0,0) =(int)(Bits[0] >> 3 | Bits[1] << 5);

10. Demo

11. Skeleton API

12. Skeleton Data

13. Joint Data Maximum two players tracked at once Six player proposals Each player with set of joints <x, y, z> in meters Tracking state Tracked Inferred Occluded, clipped, or low confidence joints Not tracked Rare, but your code must check for this state

14. Demo

15. Audio Processing Four microphone arraywith hardware-basedaudio processing Multichannel echo cancellation (MEC) Sound position tracking Other digital signal processing (noise suppression and reduction)

16. Audio API

17. Speech Recognition Grammar – What we are listening for Code – GrammarBuilder, Choices Speech Recognition Grammar Specification (SRGS) C:rogram Files (x86)icrosoft Speech Platform SDKamplesample Grammarsbr />Set AutomaticGainControl = false

18. Demo

19. Samples NUI Skeletal viewer, C++, C# Shape Game Demo, C# Audio Raw capture, C++ Audio filtering, C++ Echo cancellation, C++ Recording, C# Speech, C#

20. Resources SDK, http://research.microsoft.com/kinectsdk Channel 9 quick-starts, http://channel9.msdn.com/series/KinectSDKQuickstarts/ Coding4Fun gallery & blog, http://channel9.msdn.com/coding4fun/kinect

22. Architecture