In this presentation we explore three different approaches to Augmented Reality and we see examples of commercially available titles that use each. We explore the principals behind each approach and the steps required by developers to build AR applications on their platform of choice. Finally, we take a look at libraries that developers can leverage to build AR applications quickly on Windows Phone.

1. Augmented Reality
for Consumer Devices
(and shortcuts for Windows Phone)
Jared Bienz
jbienz@microsoft.com
jaredbienz.wordpress.com
@jbienz

2. What will be covered
 What is Augmented Reality?
 How AR is being used today
 Different approaches to AR (platform agnostic)
 Tools for building AR on Windows Phone
 Getting “crafty”
Windows Phone Microsoft Corporation.

3. What is Augmented Reality?
“A live direct or indirect view of a physical, real-world
environment whose elements are augmented by computer-
generated sensory input such as sound, video, graphics or
GPS data.”
- Wikipedia
Windows Phone Microsoft Corporation.

4. What is Augmented Reality?
“A view of the world through a digital “lens”, which allows the
scene to be changed or appear differently than the real-
world environment.”
- Jared Bienz
Windows Phone Microsoft Corporation.

5. AR Approaches
Windows Phone Microsoft Corporation.

6. Vision-Based (Games)
“These Magic Cards Are The „Holy Shit‟ Moment of
Nintendo‟s New Toy” - Gizmodo
(March2011) Windows Phone Microsoft Corporation.

7. Vision-Based (Apps)
Kogan concept app
Windows Phone Microsoft Corporation.

8. Sensor-Based (Games)
Live Butterflies Viewer / Game
Windows Phone Microsoft Corporation.

9. Sensor-Based (Apps)
PhotoSynth for iPhone (yes, we know)
Windows Phone Microsoft Corporation.

10. Vision vs. Sensor
Vision
+ No sensors required (only access to camera feed)
+ Existing libraries for tracking tags (code is fairly simple)
- Tied to specially printed tags (in range, good angle)
- CPU intensive
Sensor
+ Requires no printed tags
+ Not nearly as CPU intensive
- Code is more complex (due to dealing with sensors)
- Most current examples are tied to a fixed location
- Requires sensors
Windows Phone Microsoft Corporation.

11. Steps for Building Vision-Based AR
1. Capture video from the camera
2. Using a vision library to locate tag in scene (position)
and determine its scale and skew
3. Working backward, calculate the real-world camera
location and direction
4. Transform virtual scene (possibly using a virtual camera)
5. Draw virtual scene on top of camera feed
6. Repeat
Windows Phone Microsoft Corporation.

12. Vision-Based AR on Windows Phone 00:37
00:37
SLAR Toolkit - slartoolkit.codeplex.com
Windows Phone Microsoft Corporation.

13. Sensor-Based AR
Virtual World + Camera and Sensors = Augmented World
Windows Phone Microsoft Corporation.

14. Steps for Building Sensor-Based AR
1. Position items in “virtual world” (WP use XNA)
2. Determine how device is rotated using Motion APIs
3. Rotate virtual world to match device rotation
4. Capture video from camera
5. Draw virtual world on top of camera feed
a) (WP) Use XNA to render in XNA objects – or –
b) (WP) Convert XNA space to Silverlight transforms
6. Repeat
Windows Phone Microsoft Corporation.

15. What Sensors are Required?
 Accelerometer
 Camera
 Compass
 Gyro (best)
Windows Phone Microsoft Corporation.

16. Connecting the camera to UI (WP)
// Create PhotoCamera
camera = new PhotoCamera(CameraType.Primary);
// Create a VideoBrush and connect it to the camera
cameraBrush = new VideoBrush();
cameraBrush.SetSource(camera);
// Fill rectangle with video from the camera
someRectangle.Fill = cameraBrush;
Windows Phone Microsoft Corporation.

17. Sensors
Accelerometer Gyro
+ Very fast + Fast
+ 3 Axis (X, Y, Z)
- Not a good reference point + 3 Axis (X, Y, Z)
in motion + Always a reference point,
even in motion
Compass - Drifts
+ Simple to use
- Jitters
- Not always available
- 1 Axis only (heading)
Windows Phone Microsoft Corporation.

18. Motion “Sensor” (WP)
Windows Phone Microsoft Corporation.

19. Using Motion “sensor” (WP)
// Create Motion “sensor”
if (Motion.IsSupported)
{
motion = newMotion();
motion.TimeBetweenUpdates = TimeSpan.FromMilliseconds(20);
motion.CurrentValueChanged += motion_CurrentValueChanged;
motion.Start();
}
// Handle movement
void motion_CurrentValueChanged(object sender, SensorReadingEventArgs<MotionReading> e)
{
…
}
Windows Phone Microsoft Corporation.

20. Hardware Availability (WP)
Accelerometer & Compass are required for motion. Highest
quality if Gyro is available.
 All 7.0 devices have Accelerometer
 Some 7.0 devices have Compass (Motion enabled)
 All 7.5 devices will have Compass (Motion enabled)
 Many 7.5 devices will have Gyro (Best motion)
Windows Phone Microsoft Corporation.

21. Debugging and Testing
 Only Accelerometer supported in the emulator today
 Test for Motion.IsSupported and fail gracefully!
 AR should always be tested on a physical device
Windows Phone Microsoft Corporation.

22. Sensor-Based AR on Windows Phone
Step-by-step video - bit.ly/WPARBasic
Windows Phone Microsoft Corporation.

23. Geo Augmented Reality
TexMex
TexMex
Automatically Augmented
Real World Object
+
Your Location and
+ Generated
= World
Location Direction (sensors)
Virtual World
Windows Phone Microsoft Corporation.

24. Steps for Building Geo-Based AR
1. Get location of objects in real world (web service, etc.)
2. Get location of the user (Location APIs)
3. Calculate actual distance between user and objects
4. Generate “virtual world” where user is at center
5. Capture the direction the user is looking (Motion APIs)
6. Rotate virtual world to match device rotation
7. Capture video from camera
8. Draw virtual items on top of camera feed
9. Repeat
Windows Phone Microsoft Corporation.

25. Introducing: GART 00:18
Geo AR Toolkit - gart.codeplex.com
Windows Phone Microsoft Corporation.

26. Steps for using GART
1. New Windows Phone project
2. Add a reference to GART.dll
3. Add ARDisplay control to page (use Expression Blend)
4. Add child views (map, heading, camera, 3D)
5. Create an ARItem to represent a real-world thing
6. Add ARItem to ARDisplay.ARItems collection
Windows Phone Microsoft Corporation.

27. GART Manages Sensors
GART automatically manages the camera, motion and
location APIs for you!
 Call ARDisplay.StartServices in Page.OnNavigatedTo
 Call ARDisplay.StopServices in Page.OnNavigatedFrom
If there is a problem starting or stopping any sensor it’s
bubbled through the ARDisplay.ServiceError event.
Windows Phone Microsoft Corporation.

28. GART Demo
Windows Phone Microsoft Corporation.

29. Getting Crafty 
Windows Phone Microsoft Corporation.

30. Targus TG-42TT $14
Cable Ties: 30 cm $3
Cable Ties: 10 cm $3
Friction Mount $16
Spray Grip $3
Mainstays Fabric Bin $6
Total: $45
Windows Phone Microsoft Corporation.

31. Please let me help you build Windows Phone apps
 gart.codeplex.com
 jbienz@microsoft.com
 jaredbienz.wordpress.com
 @jbienz on Twitter
Windows Phone Microsoft Corporation.

32. The information herein is for informational purposes only and represents the
current view of Microsoft Corporation as of the date of this presentation. Because
Microsoft must respond to changing market conditions, it should not be
interpreted to be a commitment on the part of Microsoft, and Microsoft cannot
guarantee the accuracy of any information provided after the date of this
presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO
THE INFORMATION IN THIS PRESENTATION.
© 2011 Microsoft Corporation.
All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S.
and/or other countries. Windows Phone Microsoft Corporation.