Slides from the presentation made to the Flash/Flex User Group in Wellington. Introduction to the Kinect sensors and how to read their data with actionscript.

1. Get Moving!
Reading Kinect data from Actionscript

2. Kinect
๏ By Microsoft. Previously Natal.
๏ Sold to be used with the XBox.
๏ "Controller-free gaming and
entertainment experience"

3. You don’t have to buy a XBox to do any of the
stuff shown here. You can buy the camera on its
own for about NZ$200.

4. 1 Sensors

5. Kinect Sensor(s)
“The Kinect sensor is a horizontal bar connected to a small base with
a motorized pivot and is designed to be positioned lengthwise above
or below the video display. The device features an "RGB camera,
depth sensor and multi-array microphone running proprietary
software", which provide full-body 3D motion capture, facial
recognition and voice recognition capabilities.
The depth sensor consists of an infrared laser projector combined
with a monochrome CMOS sensor, which captures video data in 3D
under any ambient light conditions. The sensing range of the depth
sensor is adjustable“
Source: http://en.wikipedia.org/wiki/Kinect

6. Demo!
(CocoaKinect)

7. Sensors
Depth magic chip
(data stream)
Infrared Laser
Projector
Image chip
RGB Camera (video stream)
Infrared Camera
Multi-array mic
Motorised tilt

8. Sensors, #1 RGB
Image chip
480
@ 30 frames/sec
๏ The RBG camera can be
used for recording 640
video and assist with Horizontal field of view: 57º
Vertical field of view: 43º
Physical tilt range: 27º
tasks such as facial
recognition.

9. Sensors, #2 Depth
๏ Project a pattern of dots with a near
infrared laser over the scene. A
detector establishes the parallax shift Depth magic chip
of the dot pattern for each pixel. 480
12 bits (Data Array)
@ 30 frames/sec
640
Range: 1.2m -3.5 m

10. Depth. Uh?
Depth computed by comparing distance between Image 1
(uniquely identifiable) dot in image 1 and image 2.
The more distant, the more depth.
Magic Algorithm generates the pattern of dots
that will be sent by the Infrared projector.
In a sense a virtual image.
Image 2
Infrared camera
Depth magic chip
Stereo Image from: http://gigantico.squarespace.com/
336554365346/2007/6/21/virtual-reality-part-1.html

11. Sensors, #3 Infrared
Image chip
480
@ 30 frames/sec
๏ Images from the random
dot pattern projected by 640
the near-infrared laser.
๏ Switch off other infrared sources
(hallogen, sun light) to see them.

12. Sensors, #4 Voice
16 bits
@ 16kHz
Not covered in this talk

13. 2 Open Kinect

14. OpenKinect

15. BW + RGB images
๏ By default, provides depth
information as an RGB
image. Each color represents
a layer of depth. Red = close, blue = far away
๏ Good for Flash. Flash is good
with picture but doesn’t
excel at looping through
numbers.

16. Demo!
(Isometric Depth)

17. Demo
๏ Modified as3 server
• video2rgba: rbg to rgba mapping
(prefixing with 0xFF for alpha).
• depth2heatmap : depth to rgba
mapping.
• depth2grayscale : depth to grayscale
mapping, a scale from 0...2048 is
reduced to one of 0...255.
•

18. Resolution vs meters
๏ The depth data provided by kinect follow
some logarithmic curve :
• High resolution in the middle, low res
close and far away from the camera
๏ A linear estimate, in centimeters, can be
computed with:
0.1236 * tan(rawDisparity / 2842.5 + 1.1863)
http://openkinect.org/w/index.php?title=Imaging_Information

19. Info / Resources

20. 3 Tuio Kinect

21. Hands tracking
๏ TuioKinect tracks simple hand gestures
using the Kinect controller and sends
control data based on the TUIO protocol.
This allows the rapid creation of gesture
enabled applications

22. Demo!

23. Using TUIO from AS
https://github.com/silviopaganini/openKinect-as3-experiments
Documentation, Parameters
Note: getting 0 for
z or Z in current
implementation.
http://nuigroup.com/?ACT=28&fid=33&aid=703_FKSuKMcUkTAJrwBGk2Qv

24. Demo!
(Musical Hands)

25. 4 Advanced

26. Body-Pose
๏ What MS uses for his games is Machine
Learning
“The idea was that we would teach the computer with lots of
different people of lots of different shapes and sizes in lots of
different poses and the computer will learn how to
distinguish one part of your body from another part,” he said.
Since the Kinect camera includes depth information, it can
distinguish between big people a long way away and small
people up close.”
http://blogs.wsj.com/tech-europe/2010/11/08/key-
kinect-technology-devised-in-cambridge-lab/
For publicly available methods:
http://openkinect.org/wiki/Research_Material

27. Skeleton
http://www.openni.org/downloadfiles/
http://www.keyboardmods.com/2010/12/howto-
kinect-openninite-skeleton.html
http://www.freenect.com/kinect-skeleton-detection-
work-in-progress
http://www.modmykinect.com/2010/11/29/skeleton-
detection-part-2/

28. Frame differencing
http://actionsnippet.com/?p=2820

29. Blob Detection

30. UX

31. That’s it, folks!