COSC 426 Graduate class in Augmented Reality, lecture on AR tracking. Taught by Mark Billinghurst of the HIT Lab NZ at the University of Canterbury, July 25th 2012

1. COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
July 25th 2012
Lecture 3: AR Tracking

2. Tracking Requirements
Head Stabilized Body Stabilized World Stabilized
 Augmented Reality Information Display
 World Stabilized
Increasing Tracking
 Body Stabilized
Requirements
 Head Stabilized

3. Tracking Technologies
• Mechanical
• Electromagnetic
• Optical
• Acoustic
• Inertial and dead reckoning
• GPS
• Hybrid

4. AR Tracking Taxonomy
AR
TRACKING
Indoor Outdoor
Environment Environment
Limited Range Extended Range Low Accuracy & High Accuracy
Not Robust & Robust
Low Accuracy High Accuracy Many Fiducials Not Hybridized Hybrid Tracking
at 15-60 Hz & High Speed in space/time GPS or GPS and
Hybrid but Camera or Camera and
Tracking no GPS Compass Compass
e.g. AR Toolkit e.g. IVRD e.g. HiBall e.g. WLVA e.g. BARS

5. Tracking Types
Magnetic Inertial Ultrasonic Optical Mechanical
Tracker Tracker Tracker Tracker Tracker
Specialized Marker-Based Markerless
Tracking Tracking Tracking
Edge-Based Template-Based Interest Point
Tracking Tracking Tracking

6. Tracking Systems
 Mechanical Tracker
 Magnetic Tracker
 Ultrasonic Tracker
 Inertial Tracker
 Computer Vision (Optical Tracking)
 Specialized (Infrared, Retro-Reflective)
 Monocular (DVCam, Webcam)

7. Mechanical Tracker
 Idea: mechanical arms with joint sensors
Microscribe
 ++: high accuracy, haptic feedback
 -- : cumbersome, expensive

8. Magnetic Tracker
 Idea: difference between a magnetic transmitter
and a receiver
Flock of Birds (Ascension)
 ++: 6DOF, robust
 -- : wired, sensible to metal, noisy, expensive

9. Magnetic Tracking Error

10. Ultrasonics Tracker
 Idea: Time of Flight or Phase-Coherence Sound Waves
Ultrasonic
Logitech IS600
 ++: Small, Cheap
 -- : 3DOF, Line of Sight, Low resolution, Affected
Environment Conditon (pressure, temperature)

11. Inertial Tracker
 Idea: measuring linear and angular orientation rates
(accelerometer/gyroscope)
IS300 (Intersense)
Wii Remote
 ++: no transmitter, cheap, small, high frequency, wireless
 -- : drift, hysteris only 3DOF

12. Mobile Sensors
 Inertial compass
 Earth’s magnetic field
 Measures absolute orientation
 Accelerometers
 Measures acceleration about axis
 Used for tilt, relative rotation
 Can drift over time

13. Global Positioning System (GPS)
 Created by US in 1978
 Currently 29 satellites
 Satellites send position + time
 GPS Receiver positioning
 4 satellites need to be visible
 Differential time of arrival
 Triangulation
 Accuracy
 5-30m+, blocked by weather, buildings etc

15. Problems with GPS
 Takes time to get satellite fix
 Satellites moving around
 Earths atmosphere affects signal
 Assumes consistent speed (the speed of light).
 Delay depends where you are on Earth
 Weather effects
 Signal reflection
 Multi-path reflection off buildings
 Signal blocking
 Trees, buildings, mountains
 Satellites send out bad data
 Misreport their own position

16. Accurate to < 5cm close to base station (22m/100 km)
Expensive - $20-40,000 USD

17. Assisted-GPS (A-GPS)
 Use external location server to send GPS signal
 GPS receivers on cell towers, etc
 Sends precise satellite position (Ephemeris)
 Speeds up GPS Tracking
 Makes it faster to search for satellites
 Provides navigation data (don’t decode on phone)
 Other benefits
 Provides support for indoor positioning
 Can use cheaper GPS hardware
 Uses less battery power on device

18. Assisted GPS

19. Cell Tower Triangulation
 Calculate phone position
from signal strength
 < 50 m in cities
 > 1 km in rural

20. WiFi Positioning
 Estimate location by using WiFi access points
 Can use know locations of WiFi access points
 Triangulate through signal strength
 Eg. PlaceEngine (www.placeengine.com)
 Client software for PC and mobiles
 SDK returns position
 Accuracy
 5 – 100m (depends on WiFi density)

21. WiFi Hotspots in New York

23. Indoor WiFi Location Sensing
 Indoor Location
 Asset, people tracking
 Aeroscout
 http://aeroscout.com/
 WiFi + RFID
 Ekahau
 http://www.ekahau.com/
 WiFi + LED tracking

25. Integrated Systems
 Combine GPS, Cell tower, WiFi signals
 Skyhook (www.skyhookwireless.com)
 Core Engine
 Database of known locations
 700 million Wi-Fi access points and cellular towers.

33. Comparative Accuracies
 Study testing iPhone 3GS cf. low cost GPS
 A-GPS
 8 m error
 WiFi
 74 m error
 Cell Tower Positioning
 600 m error
Accuracy of iPhone Locations: A Comparison of Assisted GPS, WiFi, and
Cellular Positioning
In GIScience on July 15, 2009 at 8:11 pm By Paul A Zandbergen
Transactions in GIS, Volume 13 Issue s1, Pages 5 - 25

34. Optical Tracking

35. Optical Tracker
 Idea: Image Processing and Computer Vision
 Specialized
 Infrared, Retro-Reflective, Stereoscopic
ART Hi-Ball
 Monocular Based Vision Tracking

36. Outside-In vs. Inside-Out Tracking

37. Optical Tracking Technologies
 Scalable active trackers
 InterSense IS-900, 3rd Tech HiBall
3rd Tech, Inc.
 Passive optical computer vision
 Line of sight, may require landmarks
 Can be brittle.
 Computer vision is computationally-intensive

38. HiBall Tracking System (3rd Tech)
 Inside-Out Tracker
 $50K USD
 Scalable over large area
 Fast update (2000Hz)
 Latency Less than 1 ms.
 Accurate
 Position 0.4mm RMS
 Orientation 0.02° RMS

40. Starting simple: Marker tracking
 Has been done for more than 10 years
 Several open source solutions exist
 Fairly simple to implement
 Standard computer vision methods
 A rectangular marker provides 4 corner points
 Enough for pose estimation!

41. Marker Based Tracking: ARToolKit
http://artoolkit.sourceforge.net/

42. Coordinate Systems

43. Tracking Range with Pattern Size
Rule of thumb – range = 10 x pattern width

44. Tracking Error with Range

45. Tracking Error with Angle

46. Tracking challenges in ARToolKit
Occlusion Unfocused camera, Dark/unevenly lit Jittering
(image by M. Fiala) motion blur scene, vignetting (Photoshop illustration)
Image noise
False positives and inter-marker confusion (e.g. poor lens, block coding /
compression, neon tube)
(image by M. Fiala)

47. Limitations of ARToolKit
 Partial occlusions cause tracking failure
 Affected by lighting and shadows
 Tracking range depends on marker size
 Performance depends on number of markers
 cf artTag, ARToolKitPlus
 Pose accuracy depends on distance to marker
 Pose accuracy depends on angle to marker

48. Tracking, Tracking, Tracking

49. Other Marker Tracking Libraries
 arTag
 http://www.artag.net/
 ARToolKitPlus [Discontinued]
 http://studierstube.icg.tu-graz.ac.at/handheld_ar/
artoolkitplus.php
 stbTracker
 http://studierstube.icg.tu-graz.ac.at/handheld_ar/
stbtracker.php
 MXRToolKit
 http://sourceforge.net/projects/mxrtoolkit/

50. Markerless Tracking

52. Markerless Tracking
 No more Markers! Markerless Tracking
Magnetic Tracker Inertial Ultrasonic Optical
Tracker Tracker Tracker
Specialized Marker-Based Markerless
Tracking Tracking Tracking
Edge-Based Template-Based Interest Point
Tracking Tracking Tracking

53. Natural feature tracking
 Tracking from features of the surrounding
environment
 Corners, edges, blobs, ...
 Generally more difficult than marker tracking
 Markers are designed for their purpose
 The natural environment is not…
 Less well-established methods
 Usually much slower than marker tracking

54. Natural Feature Tracking
Features Points
 Use Natural Cues of Real Elements
Contours
 Edges
 Surface Texture
 Interest Points
 Model or Model-Free
 ++: no visual pollution
Surfaces

55. Texture Tracking

56. Edge Based Tracking
 RAPiD [Drummond et al. 02]
 Initialization, Control Points, Pose Prediction (Global Method)

57. Line Based Tracking
 Visual Servoing [Comport et al. 2004]

58. Model Based Tracking
 Track from 3D model
 Eg OpenTL - www.opentl.org
 General purpose library for model based visual tracking

59. Marker vs. natural feature tracking
 Marker tracking
 + Can require no image database to be stored
 + Markers can be an eye-catcher
 + Tracking is less demanding
 - The environment must be instrumented with markers
 - Markers usually work only when fully in view
 Natural feature tracking
 - A database of keypoints must be stored/downloaded
 + Natural feature targets might catch the attention less
 + Natural feature targets are potentially everywhere
 + Natural feature targets work also if partially in view

60. Hybrid Tracking

61. Hybrid Tracking
Combining several tracking modalities together

62. Sensor tracking
 Used by many “AR browsers”
 GPS, Compass, Accelerometer, (Gyroscope)
 Not sufficient alone (drift, interference)

63. Outdoor Hybrid Tracking
 Combines
 computer vision
- natural feature tracking
 inertial gyroscope sensors
 Both correct for each other
 Inertial gyro - provides frame to frame
prediction of camera orientation
 Computer vision - correct for gyro drift

64. Combining Sensors and Vision
 Sensors
- Produce noisy output (= jittering augmentations)
- Are not sufficiently accurate (= wrongly placed augmentations)
- Gives us first information on where we are in the world,
and what we are looking at
 Vision
- Is more accurate (= stable and correct augmentations)
- Requires choosing the correct keypoint database to track from
- Requires registering our local coordinate frame (online-
generated model) to the global one (world)

65. Outdoor AR Tracking System
You, Neumann, Azuma outdoor AR system (1999)

66. Robust Outdoor Tracking
 Hybrid Tracking
 Computer Vision, GPS, inertial
 Going Out
 Reitmayer & Drummond (Univ. Cambridge)

67. Handheld Display

68. Wrap-up
 Tracking and Registration are key problems
 Registration error
 Measures against static error
 Measures against dynamic error
 AR typically requires multiple tracking technologies
 Research Areas: Hybrid Markerless Techniques,
Deformable Surface, Mobile, Outdoors

69. More Information
• Mark Billinghurst
– mark.billinghurst@hitlabnz.org
• Websites
– www.hitlabnz.org