Becoming a kinect hacker innovator v2
•Als PPTX, PDF herunterladen•
1 gefällt mir•2,549 views
This document provides an overview of the Kinect sensor and Kinect for Windows SDK. It describes the Kinect sensor's capabilities including depth sensing, skeletal tracking, and speech recognition. It explains how the Kinect SDK allows accessing the sensor's data streams and provides APIs for tasks like skeletal tracking and speech recognition. The document also outlines the tools included in the SDK and provides code examples for initializing the sensor, accessing sensor data, and using speech recognition features.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Becoming a kinect hacker innovator v2
Ähnlich wie Becoming a kinect hacker innovator v2 (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Becoming a kinect hacker innovator v2
- 1. Becoming a Kinect Hacker Innovator
- 3. Who We Are Jeff Sipko – SDE on KinectShare / KinectHack – Part of Good Science, working on Fun Labs KinectShare
- 4. Overview History Capabilities Code Demo
- 7. Atari 2600 Best video game system…ever…
- 24. Vision: Shift the world from thinking “We need to understand technology” to "Technology needs to understand us"
- 27. The Challenge • Find the people in the scene, ignore background • Find their limbs and joints, which person is which • Find and track their gestures • Map the gestures to meaning and commands • Also: recognize faces • Also: recognize voices and commands • Also: reduce hardware costs to consumer levels
- 28. “What are those things?” IR Camera RGB Camera Multi-array Microphone
- 29. IR Camera: Depth Computation
- 30. IR Camera: Depth Map
- 31. IR Camera: Provided Data Depth and segmentation map
- 32. Skeletal - Provided Data
- 33. Vision Algorithm (Summary) Quickly and accurately predict 3D positions of body joints From a single depth image, using no temporal information Object recognition approach Intermediate body parts representation that maps the difficult pose estimation problem into a simpler per-pixel classification problem Large and highly varied training dataset allows the classifier to estimate body parts invariant to pose, body shape, clothing, etc. Generate confidence-scored 3D proposals of several body joints by re- projecting the classification result and finding local modes System runs at 200 frames per second on consumer hardware Evaluation shows high accuracy on both synthetic and real test sets State of the art accuracy in comparison with related work and improved generalization over exact whole-skeleton nearest neighbor matching
- 34. In Practice • Collect training data – thousands of visits to global households, filming real users, the Hollywood motion capture studio generated billions of images • Apply state-of-the-art object recognition research • Apply state-of-the-art real-time semantic segmentation • Build a training set – classify each pixel’s probability of being in any of 32 body segments, determine probabilistic cluster of body configurations consistent with those, present the most probable • Millions of training images Millions of classifier parameters • Hard to parallelize New algorithm for distributed decision-tree training • Fun Fact: Major use of DryadLINQ (large-scale distributed cluster computing)
- 35. Motorized Tilt ±28° up / down
- 36. The Audio System
- 38. Beam Forming / Source Localization Automatically points to loudest sound source Manually steer the direction of the listening beam in 10° increments
- 40. Kinect Speech Recognition Languages (as of June 2011 SDK) Acoustic model Language Countries/Regions de-DE German Germany en-AU English Australia, New Zealand en-GB English Ireland, United Kingdom en-US English Canada, United States es-ES Spanish Spain es-MX Spanish Mexico fr-CA French Canada fr-FR French France it-IT Italian Italy ja-JP Japanese Japan
- 41. Sample Grammar (from Simple Speech Recognition sample) <rule id="rootrule" scope="public"> <one-of> <item> <tag>view inventory</tag></item> <item> <tag>show quests</tag></item> <item> <tag>pause game</tag></item> <item> <token sapi:pron="S P EH L B UH K"> </token><tag>open spellbook</tag></item> […] </one-of> </rule></grammar>
- 42. Sample Grammar: Localized (from Simple Speech Recognition sample) <rule id="rootrule" scope="public"> <one-of> <item> <tag>view inventory</tag></item> <item> <tag>show quests</tag></item> <item> <tag>pause game</tag></item> <item> <tag>open spellbook</tag></item> […] </one-of> </rule></grammar>
- 43. What does this allow? Watch, track, and render people’s motion Recognize faces and facial expressions Emotionally Recognize voices, words, and tone Aware What else? What is coming? Background Removal Seated Skeletal Tracking Finger Tracking Head Tracking Digital Object Creation ???? = You
- 44. The Kinect SDK Provides both Unmanaged and Managed API Unmanaged API – Concepts work in C++ Managed API – Concepts work in both VB/C# Samples & documentation to get you started Assumes some programming experience http://kinectforwindows.org
- 45. What’s in the SDK? Raw sensor streams Access to raw data streams from the depth sensor, color camera sensor, and four-element microphone array enables developers to build upon the low-level streams that are generated by the Kinect sensor. Skeletal tracking The capability to track the skeleton image of one or two people moving within the Kinect field of view make it easy to create gesture-driven applications. Advanced audio capabilities Audio processing capabilities include sophisticated acoustic noise suppression and echo cancellation, beam formation to identify the current sound source, and integration with the Windows speech recognition API. Sample code and documentation The SDK includes more than 100 pages of technical documentation. In addition to built-in help files, the documentation includes detailed walkthroughs for most samples provided with the SDK. Easy installation The SDK installs quickly, requires no complex configuration, and the complete installer size is less than 100 MB. Developers can get up and running in just a few minutes with a standard standalone Kinect sensor unit (widely available at retail outlets). Designed for non-commercial purposes; a commercial version is expected later. Windows 7 – C++, C#, or Visual Basic in Microsoft Visual Studio 2010.
- 47. The Tools Visual Studio 2010 (Express or other) .NET Framework 4.0 Kinect for Windows SDK Microsoft Speech Framework Coding4Fun Kinect Toolkit XNA Game Studio
- 48. Let’s Get Started! … Literally! Runtime nui = new Runtime(); Runtime.Initialize(); And nui.VideoStream.Open(…); nui.DepthStream.Open(…); … and then later: Runtime.Shutdown();
- 49. Gimme Data Polling Method nui.DepthStream.GetNextFrame(timeout) nui.VideoStream.GetNextFrame(timeout) nui.SkeletonEngine.GetNextFrame(timeout) Event Method nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(fn) nui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(fn) nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(fn)
- 50. What you get Color Image: Byte array in B8G8R8 format by default Also supports YUV Depth Image: 13 high-order bits contain the distance in mm 3 low-order bits contain the player index Only valid values are 0, 1, 2
- 51. Speech Start the Recognition Engine varsre = new SpeechRecognitionEngine(newSystem.Globalization.C ultureInfo("en-US")); Create a grammar Choices colors = new Choices(); colors.Add(new string[] {"red"}); colors.Add(new string[] {"green"}); colors.Add(new string[] {"blue"}); GrammarBuildergb = new GrammarBuilder(); gb.Append(colors); // Create the Grammar instance. Grammar g = new Grammar(gb); sre.LoadGrammar(g);
- 52. Speech – Grammar From File FileStreamfs = new FileStream(grammarPath + ”foobar.cfg", FileMode.Create); SrgsGrammarCompiler.Compile(grammarPath + ”foobar.grxml", (Stream)fs); fs.Close(); Grammar g = new Grammar(grammarPath + ”foobar.cfg", "rootrule");
- 53. Speech – cont Register for speech events sre.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(sre_S peechRecognized); and void sre_SpeechRecognized(object sender, SpeechRecognizedEventArgse) { MessageBox.Show(e.Result.Text); }
- 54. Show me the demo!
- 55. Kinect Fun Labs… + +
- 58. Thank You! Obrigado! Forums http://kinectforwindows.org/resources Email me jeff.sipko@microsoft.com
Hinweis der Redaktion
- Video.Who owns a Kinect? Who owns an Xbox 360? Can I get a show of hands? How many people have used the Kinect SDK?
- We are Microsoft Studios (formerly Microsoft Game Studios). We are a collection of, surprise, individual, autonomous studios. Our group is tackling kinectshare.com (for all your Kinect UGC needs) and now Kinect Hacks.
- MagnovoxOdesseyReleased in 1972The Magnavox Odyssey is the first home video game console, predating the Atari PONG home consoles by three years. The Odyssey was designed by Ralph Baer, who had a working prototype finished by 1968. This prototype is affectionately known as the "Brown Box" to classic video game hobbyists. Unlike most video game consoles, the Odyssey is analog rather than digital, which makes its invention all the more amazing in spite of its rather crude graphics and controller responsiveness. Also, unlike any conventional console today, this system was powered by batteries. The Odyssey and its variants also lack sound capability (hence a silent console), which was not uncommon in early PONG systems of that era.The Odyssey was released in May 1972. While it did not perform badly, it did not take long before it succumbed to poor marketing by Magnavox retail chains. One of their mistakes was misleading consumers into believing that the Odyssey would work only on Magnavox televisions. It did, however, prove that consoles for the home could be designed.
- Atari PONGReleased in 1975In 1973, after the success of the original PONG coin-op, an Atari engineer by the name of Harold Lee came up with the idea of a home PONG unit. Since the PONG coin-op that Alan Alcorn designed was nothing more than the game board connected to an actual television set, he thought it would be possible to scale it down a bit and modify it for use at home. This would be a new direction for the fledgling Atari consumer electronics. If they could pull it off, they would be one of the pioneers of using high tech custom integrated circuits in the consumer industry.In 1975 it was decided Sears would sell PONG under it's own specially created Tele-Games label, and production was initially projected at 50,000 units. This was soon raised to 150,000 for the 1975 Christmass season. Atari agreed to give Sears exclusive rights for the following year, and would continue to make custom Tele-Games versions for any future consoles. This was the beginning of a long relationship between Atari and Sears, which would continue even after Nolan Bushnell sold Atari to Warner.
- Atari VCS 2600Released in 1977The Atari 2600, released in 1977, is the first successful video game console to use plug-in cartridges instead of having one or more games built in. It was originally known as the Atari VCS, for Video Computer System, and the name "Atari 2600" (taken from the unit's Atari part number, CX2600) was first used in 1982, after the release of the more advanced Atari 5200.The initial price was $199 with a library of 9 titles. In a play to compete directly with the Channel F, Atari named the machine the Video Computer System (or VCS for short), as the Channel F was at that point known as the VES, for Video Entertainment System. When Fairchild learned of Atari's naming they quickly changed the name of their system to become the Channel F.Atari expanded the 2600 family with two other compatible consoles. The Atari 2700, a wireless version of the console was never released due to design flaws. The Sleek Atari 2800 released to the Japanese market in 1983 suffered from competition from the newly-released Nintendo Famicom.
- ColecovisionReleased in 1982The Colecovision is Coleco's third generation video game console, released in August 1982. It offered arcade-like graphics and controllers, and an initial catalog of 12 titles, with 10 more promised titles on the way. All told, approximately 170 titles were released on plug-in cartridges during its lifetime. The controller was a flat joystick, two side buttons, and a number-pad, which allowed the user to put inserts for customized buttons. The majority of titles in its catalog were conversions from coin-operated arcade games. The ColecoVision introduced two new concepts to the home video game industry - the ability to expand the hardware system, and the ability to play other video game system games.By Christmas of 1982, Coleco had sold 500,000 units, mainly on the strength of its bundled games. While Atari's fortune had risen on the popularity of Space Invaders, Colecovision was the first console to feature the hit Donkey Kong, by Nintendo. The Colecovision's main competitor in the next-generation console space was the arguably more advanced but less commercially successful Atari 5200.
- Nintendo Entertainment System (NES)Released in 1985Following a series of arcade game successes in the early 1980s, Nintendo made plans to produce its own console hardware that had removable cartridges, a feature not included with the company's earlier Color TV Games product. Designed by Masayuki Uemura and released in Japan on July 15, 1983, the Nintendo Family Computer (Famicom) was slow to gather momentum: during its first year, many criticized the system as unreliable, prone to programming errors and rampant freezing. Following a product recall and a reissue with a new motherboard, the Famicom's popularity soared, becoming the best-selling game console in Japan by the end of 1984. Encouraged by their successes, Nintendo soon turned their attentions to the North American markets.In June 1985, Nintendo unveiled its American version of the Famicom at the Consumer Electronics Show (CES). With a completely redesigned case and a new name, the Nintendo Entertainment System (NES) proved to be just as popular in America as the Famicom was in Japan, and played a major role in revitalizing interest in the video game industry.
- Sega Master System (SMS)Released in 1986After producing many games for early home video game consoles, Sega decided to develop a console system of its own. The SG-1000 and Mark III were available in Japan in the mid-1980s, but when Sega witnessed the early success of the Nintendo Entertainment System, the company knew it wanted a share of the American console market. So, Sega redesigned the Mark III, renamed it the Sega Master System (SMS for short), and released it in 1986, not long after the NES first came out.Technically, the Master System was superior to the NES, with better graphics and higher quality sound. The original SMS could play both cartridges and the credit card-sized "Sega Cards," which retailed for cheaper prices than carts but had less code. The SMS also had cooler accessories (like 3D glasses), but this didn't do much good when there weren't very many exciting games.The Master System technology lived on in Sega's Game Gear, which was basically a portable SMS with some enhancements.
- Sega GenesisReleased in 1989It was 1989. Nintendo's NES had reigned supreme in the video game market for nearly five years, and it was time for a new system to take over the throne. Sega's Master System, while graphically superior to the NES, failed to make any kind of lasting impression in the U.S. market (although it was very popular in Europe), and Sega knew that their next system would not only have to be superior to everything else out there, but they'd have to have a lot of third-party developers lined up.After two years of development, Sega introduced their "next generation" system to the world in late 1989. Known as the Genesis in the West, and the Mega Drive in the east, Sega began an aggressive marketing campaign, not only to customers, but also to developers.Although NEC's TurboGrafx-16 had beat the Genesis to market by nearly four months, Sega quickly regained lost ground, thanks to their line-up of quality arcade conversions, killer sports games, and most of all, the full support of Trip Hawkins and Electronic Arts.
- Super Nintendo Entertainment System (SNES)Released in 1991The Super Nintendo Entertainment System was Nintendo's second home console, following the Nintendo Entertainment System (often abbreviated to NES, released as the Famicom in Japan). Whereas the earlier system had struggled in Europe and large parts of Asia the SNES proved to be a global success, albeit one that could not match its predecessor's popularity in South East Asia and North America - due in part to increased competition from Sega's Mega Drive console (released in North America as the Genesis). Despite its relatively late start, the SNES became the best selling console of the 16-bit era but only after its competitor Sega had pulled out of the 16-bit market to focus on its 32-bit next generation console.Nintendo released the Super Nintendo Entertainment System which initially sold for a price of $200. The North American package included the game Super Mario World. The SNES was released in the United Kingdom and Ireland in April 1992 for £150, with a German release following a few weeks later. The PAL versions of the console looked identical to the Japanese Super Famicom, except for labelling.
- Atari JaguarReleased in 1993Competing with Sega and Nintendo's 16-bit consoles, the Jaguar was said to be 64-bit. Back then, bit width was a big deal in the gaming industry, just as polygon-pushing power is today. The Jaguar did not work off of a solitary 64-bit processor, but instead it had a collection of processors with bus widths ranging from 16 to 64 bits. The bit width of the Jaguar is still a source of considerable debate today, but consensus exists among those who are familiar with the system hardware that, because Jaguar's main data bus and some of the processors are 64-bit, the entire system can be considered 64 bit. It would otherwise be considered a 32-bit console.Nonetheless, it was technically superior to the leading 16-bit consoles at the time. Unfortunately, this last ditch effort by Atari to find room in the console market failed. A relatively small number of games were developed for the system, but Atari pulled the plug altogether in 1996.
- Sony PlayStation PS1Released in 1994The first new version was actually a revision in early 1996, produced in response to complaints that PlayStations were overheating. Sony did not change the technical aspects or the cosmetics, but did remove the S-video port left over from the Japanese release.Sony produced a redesigned version of the original console, called the "PSone", in a smaller (and more ergonomic) case which was introduced in September 2000. The original PlayStation was abbreviated in Japan to "PS" and was often abbreviated as "PSX" by American gamers, as this was Sony's internal code name for the system while it was under development. This led to some confusion in 2003, when Sony introduced a PS2-derived system in Japan actually called the PSX. The PlayStation is now officially abbreviated as the "PS1" or "PSone," although many people still abbreviate it "PS" or "PSX". There were only 2 differences between the "PSone" and the original, the first one being cosmetic change to the console, and the second one was the home menu's Graphical User Interface.
- Nintendo 64Released in 1996The Nintendo 64, commonly called the N64, is Nintendo's third home video game console. The N64 was released on June 23, 1996 in Japan, September 29, 1996 in North America, 1 March 1997 in Europe/Australia and September 1, 1997 in France. It was released with only two launch games in Japan and North America (Super Mario 64 and PilotWings 64) while Europe had a third launch title in the form of Star Wars: Shadows of the Empire (which was released earlier in the other markets). The Nintendo 64 cost $199 at launch in the United States.During the developmental stages the N64 was referred to by its code name, Project Reality. The name Project Reality came from the speculation within Nintendo that this console could produce CGI on par with then-current super computers. Once unveiled to the public the name changed to Nintendo Ultra 64, referring to its 64-bit processor, and Nintendo dropped "Ultra" from the name on February 1, 1996, just five months before its Japanese debut.
- Sega DreamcastReleased in 1999The Dreamcast was released on November 27, 1998 in Japan, on September 9, 1999 in the United States (the date 9/9/99 featured heavily in US promotion) and on October 14, 1999 in Europe. The tag line used to promote the console in the US was "It's thinking", and in Europe "Up to 6 Billion Players". (The vagueness of these campaigns and almost total lack of any in game footage has been touted as one of the reasons for the Dreamcast's eventual downfall. Many Americans knew that the Dreamcast was coming, but didn't know what one was.)The Dreamcast was the first console to include a built-in modem and Internet support for on-line gaming. It enjoyed brisk sales in its first season and was one of Sega's most successful hardware units. In the United States alone, a record 200,000 units had been pre-ordered before launch and Sega sold 500,000 consoles in just two weeks (including 225,000 sold on the first 24 hours which became a video game record until the PlayStation 2 launched a year later). In fact, due to brisk sales and hardware shortages, Sega was unable to fulfill all of the advance orders.
- Sony PlayStation 2 (SCPH-5000x)Released in 2000The PlayStation 2 had a difficult start. Only a few million users had obtained consoles by the end of 2000 due to manufacturing delays. The PlayStation 2 was such a hot item after its release that it was near impossible to find one on retailer shelves, leaving those wanting a PlayStation 2 to either wait or purchase the console on-line at sites such as eBay, where the console was being sold by many people for twice and sometimes five times as much as the manufacturer's listed price.The PlayStation brand's strength has lead to strong third-party support for the system. Although the launch titles for the PS2 were unimpressive in 2000, the holiday season of 2001 saw the release of several best-selling and critically acclaimed games. Those PS2 titles helped the PS2 maintain and extend its lead in the video game console market, despite increased competition from the launches of the Microsoft Xbox and Nintendo GameCube. In several cases, Sony made exclusivity deals with publishers in order to preempt its competitors.
- Nintendo Game CubeReleased in 2001Unveiled during Spaceworld 2000, the Nintendo GameCube was widely anticipated by many who were shocked by Nintendo's decision to design the Nintendo 64 as a cartridge-based system. Physically shaped similar to a geometric cube, the outside casing of the Nintendo GameCube comes in a variety of colors, such as indigo, platinum, and black (also a limited edition Resident Evil 4 platinum and black game console).The Nintendo GameCube uses a unique storage medium, the GameCube Optical Disc, a proprietary format based on Matsushita's optical-disc technology; the discs are approximately 8 centimeters (3 1/8 inches) in diameter (considerably smaller than the 12cm CDs or DVDs used in competitors' consoles), and the discs have a capacity of approximately 1.5 gigabytes. The disc is also read from the outer-most edge going inward, the opposite of a standard DVD. This move was mainly intended to prevent piracy of GCN titles, but like most anti-piracy technology, it was eventually cracked.
- Microsoft X-BoxReleased in 2001The Microsoft Xbox is a sixth generation era video game console first released on November 15, 2001 in North America, then released on February 22, 2002 in Japan, and later on March 14, 2002 in Europe. The Xbox was Microsoft's first independent venture into the video game console arena, after having developed the operating system and development tools for the MSX, and having collaborated with Sega in porting Windows CE to the Sega Dreamcast console. Notable launch titles for the console include Amped, Dead or Alive 3, Halo: Combat Evolved, Oddworld: Munch's Oddysee, and Project Gotham Racing.In November 2002 Microsoft released the Xbox Live on-line gaming service, allowing subscribers to play on-line Xbox games with (or against) other subscribers all around the world and download new content for their games to the hard drive. This on-line service works exclusively with broadband. 250,000 subscribers had signed on in 2 months since Live was launched. In July 2004, Microsoft announced that Xbox Live reached 1 million subscribers, and announced in July 2005 that Live had reached 2 million.
- XaviXPORTReleased in 2004In January at the Consumer Electronics Show 2004 (CES), SSD COMPANY LIMITED debuts their XaviX® technology to the American public. The XaviXPort console was officially released in the US in August of 2004.XaviXPort is a unique and innovative console that uses peripherals to interact with on screen games. The console contains image recognition and infrared sensors that can detect player movements. These movements are calculated by a proprietary multiprocessor that measures both velocity and angle. The multiprocessor then translates the actions into on screen movement.Getting players to immerse themselves into games with body movements is not something new. However, this is the first time a console has been dedicated to providing this "Get your butt off the couch" interactive gaming experience. What makes XaviXPort even more unique is that the console’s multiprocessor is not installed inside the system itself. The multiprocessor can be found in each game cartridge.
- Microsoft X-Box 360Released in 2005The Xbox 360 is Microsoft's newest video game console, the successor to their original Xbox. It was released on November 22, 2005 in North America, December 2 in Europe, and December 10 in Japan. It will be released on February 2, 2006 in Mexico, February 24 in South Korea, and March 2, 2006 in Australia, New Zealand, Hong Kong, Taiwan and Singapore.The Xbox 360 will compete against the upcoming generation of consoles, including the Sony PlayStation 3 and Nintendo Revolution, and was officially unveiled on MTV on May 12, 2005, a week before the E3 trade show.Except in Japan the console is sold in two different configurations: the "Xbox 360" and the "Xbox 360 Core System". The Xbox 360 configuration, often referred to as the "Premium Edition", includes a hard drive (required for backwards compatibility with original Xbox games), a wireless controller, a headset, an Ethernet cable, an Xbox Live silver subscription, and a component HD AV cable (which can also be used on non-HD TVs).
- Sony PlayStation 3Released in 2006The PlayStation 3 was released in North America on November 17, 2006. During its first week of release in the United States, PlayStation 3s were being sold on eBay for more than $2300 USD. Reports of violence surrounding the release of the PS3 include a customer shot, campers robbed at gunpoint, customers shot in a drive-by shooting with BB guns, and 60 campers fighting over 10 systems. Two GameStop employees fabricated a robbery to cover up their own theft of several PlayStation 3 and four Xbox 360 consoles.Sony stated every PlayStation and PlayStation 2 game that observes its respective system's TRC (Technical Requirements Checklist) will be playable on PS3 at launch. SCE president Ken Kutaragi asked developers to adhere to the TRC to facilitate compatibility with future PlayStations, stating that the company was having some difficulty getting backward compatibility with games that had not followed the TRCs. It has been confirmed (image) that initial PS3 units include the CPU/rasterizer combination chip used in slim PS2 (EE+GS) to achieve backward compatibility.
- Nintendo WiiReleased in 2006The console was known by the codename of "Revolution" until April 27, 2006, when it was renamed Wii, spelled with two "i"s to imply an image of players gathering together, as well as to represent the console's controllers. It is said Wii sounds like 'we', which emphasizes that the console is for everyone. Wii can easily be remembered by people around the world, no matter what language they speak. No confusion. No need to abbreviate. Just Wii"The Wii Remote is a one-handed controller that uses a combination of accelerometers and infrared detection to sense its position in 3D space. This allows users to control the game using physical gestures as well as traditional button presses. The controller connects to the console using Bluetooth, and features force feedback, 4KB non-volatile memory and an internal speaker. Perhaps the most important of these devices is the Nunchuk unit, which features an accelerometer and a traditional analog stick with two trigger buttons. In addition, an attachable wrist strap can be used to prevent the player from unintentionally dropping or throwing the device.
- A few inspiration points from the creators of Kinect.
- So who likes playing video games? Who thinks gaming controllers are really easy to use? How long do you think it would take for you to become an expert at all of these buttons and win games? If you could just turn on the game and play and be pretty good at the game, do you think you’d probably play more video games?The purpose of Kinect is to make XBox more accessible to a broader audience. The Kinect team focused on making XBox so easy to use that anyone could jump in and play and not have to worry about reading any instructions or learning all the different controller buttons and permutations to be great at the game. They wanted to make beginners feel like experts.Kinect is designed so anyone can play, whether they are a kid, an adult, no matter how much gaming experience you have, how old you are -- you can jump in a play right away. Imagine your little brother or sister, or your grandparents trying to play an Xbox game without having to learn which button does what?
- So, as we said in the last slide, instead of learning all the right buttons to click on the console, make the game understand YOU. That’s Kinect!Make gaming more accessible. Open up gaming to others. Use what you know. Don’t need to learn.But there’s also another unique element to Kinect and that is making gaming more social. Traditionally you would have your hard core gamers sitting alone in front of their game with their console firing away at the next alien, racing away in their own world for hours, etc. With Kinect, gaming is actually bringing people together in a fun, collaborative way, where watching your friends and family play is actually really entertaining. And playing with others using Xbox Live is a very social gaming experience. People are laughing and joining in even if they aren’t playing, so much that they want to get up and play themselves.
- Let’s have a look at the Kinect Sensor. What are those things on the sensor? There’s a RGB camera, a depth sensor and a multi-array microphone. When you first start up Kinect, it reads the layout of your room and configures the play space you'll be moving in. Then, Kinect detects and tracks 32 points on each player's body, mapping them to a digital reproduction of that player's body shape and skeletal structure, including facial detail.Let’s take a look at each component separately to help you understand how it all works together…[next few slides go into more detail]
- Source: http://nuit-blanche.blogspot.com/2010/11/unsing-kinect-for-compressive-sensing.html
- Algorithms built in to allow control of which joints are moving. Which means the ability to remove noise.The point is you can fine tune how the position of all those joints are represented.So you can imagine the skeleton engine drawing 30 fps, if there’s a slight shift, and you’re using your hand as a cursor, it may make the cursor jump, producing jitter. You can remove that jitter using TransformSmooth.
- 32 points of dataExpresses it in X, Y, Z coordinate with Z determine positioning and/tilt 12 Tracking Points – 20 joint pointsSkeletal MirroringIs avatar facing you or away? By default, the skeleton system always mirrors the user who is being tracked. For applications that use an avatar to represent the user, such mirroring could be desirable if the avatar is shown facing into the screen. However, if the avatar faces the user, mirroring would present the avatar as backwards. Depending on its requirements, an application can create a transformation matrix to mirror the skeleton and then apply the matrix to the points in the array that contains the skeleton positions for that skeleton. The application is responsible for choosing the proper plane for reflection.
- This wall of text basically says that the algorithm works by:Performing object separation on the players in the playspace (left column)Perform object segmentation to find major chunks of the body (second column)Machine-learning trained classifier turns segmentation data into skeletal positioning data (right-most columns)
- See segmentation example in this image.Really, lots of people were recorded to build the training data (me included)
- The sensor automatically tilts down to find the floor, and then back up to see the players in the play space. On the Xbox, the play space can be fine tuned via the Kinect tuner. Programmatic control over the tilt level is also offered, to support a variety of different scenarios.
- There are 4 Microphones in the Kinect array.Enables some nifty features, like the following:
- By meas
- Leverages Microsoft Speech TechnologiesTitle supplies one or more grammarsRules for which words/phrases should be recognized in a given contextTitle grammars are XML (subset of W3C standard)Contain one or more rules of active phrases to recognizeTitle can toggle individual rule states at runtime (or just use one grammar file with a single rule per game state)
- This SDK should not be used to develop prototype applications with the intent of porting those applications to the Xbox 360 console. There are numerous architectural, performance, and behavioral differences between Xbox 360 and Windows. These differences affect how the Kinect technology is implemented for Windows 7. As a result, design decisions and performance characteristics are quite often very different between the two platforms. Microsoft recommends using Xbox 360 development kits (XDK hardware and software) to create Xbox 360 applications.
- Kinect hardwareMicrosoft Kinect drivers: Windows 7 drivers for the Kinect sensor are installed as part of the SDK Beta setup process as described in this document.The Microsoft Kinectdrivers support:• The Kinectsensor’s microphone array as a kernel-mode audio device that you can access through the standard audio APIs in Windows.•Streaming image and depth data.• Device enumeration functions that enable an application to use more than one Kinect sensor that is connected to the computer. Up to 4 Kinect devices for apps can be connected at concurrently3.NUI API: A set of APIs that retrieves data from the image sensors and controls the Kinect devices.4. Audio API: Exposes additional functionality like Beamforming. KinectAudio DMO:The Kinect DMO that extends the microphone array support in Windows 7 to expose beamforming and source localization functionality. Beamforming allows applications to determine the direction of the audio source and use the microphone array as a steerable directional microphone.5. Windows 7 standard APIs: The audio, speech, and media APIs in Windows 7, as described in the Windows 7 SDK and the Microsoft Speech SDK.
- These are the ones that I’ll be using, listed for the sake of completeness
- Drivers install automatically after you install the SDK and then plug in the Kinect sensor. Covering C#, because that’s what I like, but you can check docs for the analogous C++ functions. So far, easy as pie.
- Polling method is the easiest, but can be inefficient. Event method also pretty easy, and doesn’t use resources unnecessarily, but in the current state, skeletal performance can be slow.
- Actual array wrapped inside a PlanarImage structure inside an ImageFrame structure. Like a mystery wrapped in an enigma.
- Kinect Fun Labs. This new hub exhibits never before seen Kinect technology, including player capture, object scanning and finger tracking. The Microsoft Studios team has created the first Gadgets available on Kinect Fun Labs – from turning yourself into a bobblehead to bringing a favorite toy to life and more. http://kinectfunlabs.comThrough KinectShare.com not only will you be able to share your pictures and videos from Kinect Fun Labs with friends and family, but you will also be able to vote for your favorite Kinect innovations. Innovations with the most votes cast will have the opportunity to be integrated and released through Kinect Fun Labs.http://kinectshare.com