A presentation given by Mark Billinghurst on April 21st 2015 at the CHI 2015 conference. This talk presents highlights from the journal paper:
M. Billinghurst, A. Clark, and G. Lee. A Survey
of Augmented Reality, Foundations and
Trends in Human-Computer Interaction.
Vol. 8, No. 1 (2015) 1–202, 2015
Available at :http://www.nowpublishers.com/article/Details/HCI-049
Handwritten Text Recognition for manuscripts and early printed texts
A Survey of Augmented Reality
1. A Survey of Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
April 21st 2014
mark.billinghurst@hitlabnz.org
2. Highlights from:
M. Billinghurst, A. Clark, and G. Lee. A Survey
of Augmented Reality, Foundations and
Trends in Human-Computer Interaction.
Vol. 8, No. 1 (2015) 1–202, 2015
http://www.nowpublishers.com/article/Details/HCI-049
5. Augmented Reality Definition
! Defining Characteristics [Azuma 97]
! Combines Real and Virtual Images
- Both can be seen at the same time
! Interactive in real-time
- The virtual content can be interacted with
! Registered in 3D
- Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
10. Making Interfaces Invisible
Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented
interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on
User interface and Software Technology. UIST '95. ACM, New York, NY, 29-36.
11. From Reality to Virtual Reality
Ubiquitous Computing Augmented Reality Virtual Reality
12. Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
13. Metaverse
! Neal Stephenson’s “SnowCrash”
! The Metaverse is the convergence of:
! 1) virtually enhanced physical reality
! 2) physically persistent virtual space
! Metaverse Roadmap
! http://metaverseroadmap.org/
14.
15. Summary
! Augmented Reality has three key features
! Combines Real and Virtual Images
! Interactive in real-time
! Registered in 3D
! AR can be classified alongside other technologies
! Invisible Interfaces
! Milgram’s Mixed Reality continuum
! Stephenson’s MetaVerse
18. Early HMDs and HUDs (1960’s)
Sutherland / Sproull’s see-
through HMD (1965)
Bucaneer HUD (1958)
19. AR History
! 1960’s – 80’s: Early Experimentation
• Military, Academic labs
! 1980’s – 90’s: Basic Research
• Tracking, Displays
! 1995 – 2005: Tools/Applications
• Interaction, Usability, Theory
! 2005 - : Commercial Applications
• Games, Medical, Industry, Mobile
20. Important Milestones
! 1992: T. Caudell coined term “AR” (Boeing)
! 1996: First collaborative AR systems
! 1999: ARToolKit released – tracking library
! 1997: Feiner First outdoor AR system
! 1997 – 2001: Mixed Reality Systems Lab (Japan)
! 1999 – 2003: ARVIKA project (Germany)
! 2002: ISMAR conference started
21. AR Enters Mainstream (2007 -)
! Magazines
! MIT Tech. Review (Mar 2007)
- 10 most exciting technologies
! Economist (Dec. 2007)
- Reality, only better
! Games
! Sony “Eye of Judgement”
- 300,000+ units shipped
! Broadcast TV
! Sports broadcasting
23. Browser Based AR (2008 - )
! Flash + Camera + 3D graphics
! High impact
! High marketing value
! Large potential install base
! 1.6 Billion web users
! Ease of development
! Lots of developers, mature tools
! Low cost of entry
! Browser, web camera
24. Mobile AR (2005 - )
! Mobile Phones
! Camera, processor, display
! Computer vision based AR
! Advertising
! HIT Lab NZ (2007)
! AR print advertisement
! Txt to download app
25. Mobile Outdoor AR (2009 - )
! Mobile phones with GPS
! Tag real world locations
! GPS + Compass input
! Overlay graphics data on live video
! Applications
! Travel guide, Advertising, etc
! Wikitude, Layar, Junaio, etc..
! Android/iOS based, Public API released
Motorola Droid
26. AR Today
! Key Technologies Available
- Robust tracking (Computer Vision, GPS/sensors)
- Display (Handheld, HMDs)
- Input Devices (Kinect, etc)
- Developer tools (Qualcomm, Metaio, Wikitude)
! Commercial Business Growing
- Gaming, GPS/Mobile, Online Advertisement
• >$5 Billion USD by 2016 (Markets andMarkets)
• >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)
27. AR Business Today
! Marketing
! Web-based, mobile
! Mobile AR
! Geo-located information and service
! Gaming
! Mobile, Physical input (Kinect, PS Move)
! Upcoming areas
! Manufacturing, Medical, Military
28. Summary
! Augmented Reality has a long history going
back to the 1960’s
! Interest in AR has exploded over the last few
years and is being commercialized quickly
! AR is growing in a number of areas
! Mobile AR
! Web based AR
! Marketing experiences
37. Strengths of optical see-through
! Simpler (cheaper)
! Direct view of real world
! Full resolution, no time delay (for real world)
! Safety
! Lower distortion
! No eye displacement
! see directly through display
39. Vuzix Wrap 1200DXAR
▪ Stereo video see-through display ($1500)
■ Twin 852 x 480 LCD displays, 35 deg. FOV
■ Stereo VGA cameras
■ 3 DOF head tracking
40. Strengths of Video See-Through
! True occlusion
! Block image of real world
! Digitized image of real world
! Flexibility in composition
! Matchable time delays
! More registration, calibration strategies
! Wide FOV is easier to support
! wide FOV camera
41. Multiplexed Displays
! Above or below line of sight
! Strengths
! User has unobstructed view of real world
! Simple optics/cheap
! Weaknesses
! Direct information overlay difficult
• Display/camera offset from eyeline
! Wide FOV difficult
43. Display Technology
! Curved Mirror
! off-axis projection
! curved mirrors in front of eye
! high distortion, small eye-box
! Waveguide
! use internal reflection
! unobstructed view of world
! large eye-box
44. See-through thin displays
! Waveguide techniques for thin see-through displays
! Wider FOV, enable AR applications
! Social acceptability
Opinvent Ora
Lumus DK40
56. Objects Registered in 3D
! Registration
! Positioning virtual object wrt real world
! Tracking
! Continually locating the users viewpoint
- Position (x,y,z), Orientation (r,p,y)
59. Example: Marker tracking
! Available for more than 10 years
! Several open source solutions exist
! ARToolKit, ARTag, ATK+, etc
! Fairly simple to implement
! Standard computer vision methods
! A rectangle provides 4 corner points
! Enough for pose estimation!
62. Tracking challenges in ARToolKit
False positives and inter-marker confusion
(image by M. Fiala)
Image noise
(e.g. poor lens, block coding /
compression, neon tube)
Unfocused camera,
motion blur
Dark/unevenly lit
scene, vignetting
Jittering
(Photoshop illustration)
Occlusion
(image by M. Fiala)
63.
64. Markerless Tracking
Magnetic Tracker Inertial
Tracker
Ultrasonic
Tracker
Optical
Tracker
Marker-Based
Tracking
Markerless
Tracking
Specialized
Tracking
Edge-Based
Tracking
Template-Based
Tracking
Interest Point
Tracking
! No more Markers! #Markerless Tracking
Mechanical
Tracker
65. Natural Feature Tracking
! Use Natural Cues of Real Elements
! Edges
! Surface Texture
! Interest Points
! Model or Model-Free
! No visual pollution
Contours
Features Points
Surfaces
69. Model Based Tracking
! OpenTL - www.opentl.org
! General purpose library for model based visual tracking
70. Marker vs. natural feature tracking
! Marker tracking
! ++ Markers can be an eye-catcher
! ++ Tracking is less demanding
! -- The environment must be instrumented with markers
! -- Markers usually work only when fully in view
! Natural feature tracking
! -- A database of keypoints must be stored/downloaded
! ++ Natural feature targets might catch the attention less
! ++ Natural feature targets are potentially everywhere
! ++ Natural feature targets work also if partially in view
71. Example: Outdoor Hybrid Tracking
! Combines
! computer vision
- natural feature tracking
! inertial gyroscope sensors
! Both correct for each other
! Inertial gyro - provides frame to frame
prediction of camera orientation
! Computer vision - correct for gyro drift
72. Robust Outdoor Tracking
! Hybrid Tracking
! Computer Vision, GPS, inertial
! Going Out
! Reitmayr & Drummond (Univ. Cambridge)
Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented
reality. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp.
109-118). IEEE.
77. Interface Design Path
1/ Prototype Demonstration
2/ Adoption of Interaction Techniques from other
interface metaphors
3/ Development of new interface metaphors
appropriate to the medium
4/ Development of formal theoretical models for
predicting and modeling user actions
Desktop WIMP
Virtual Reality
Augmented Reality
78. Interaction Development
! Information Browsing
! Camera movement
! Limited interaction
! 3D AR Interaction
! HMD, hand tracking
! 3D UI/VR techniques
! Specialized input devices
83. Tangible AR Interaction
! AR overcomes limitation of TUIs
! enhance display possibilities
! merge task/display space
! provide public and private views
! TUI + AR = Tangible AR
! Apply TUI methods to AR interface design
84. Tangible AR Design Principles
! Tangible AR Interfaces use TUI principles
! Physical controllers for moving virtual content
! Support for spatial 3D interaction techniques
! Support for multi-handed interaction
! Match object affordances to task requirements
! Support parallel activity with multiple objects
! Allow collaboration between multiple users
85. VOMAR - Tangible AR Interface
! Use of natural physical object
to control virtual objects
! Physical objects
! Catalog book:
- Turn over the page
! Paddle operation:
- Push, shake, incline, hit, scoop
Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000). Virtual object manipulation on
a table-top AR environment. In Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM
International Symposium on (pp. 111-119). Ieee.
86. Wrap-up
! Browsing Interfaces
! simple (conceptually!), unobtrusive
! 3D AR Interfaces
! expressive, creative, require attention
! Tangible Interfaces
! Embedded into conventional environments
! Tangible AR
! Combines TUI input + AR display
88. ! Web based AR
! Flash, HTML 5 based AR
! Marketing, education
! Outdoor Mobile AR
! GPS, compass tracking
! Viewing Points of Interest in real world
! Eg: Junaio, Layar, Wikitude
! Handheld AR
! Vision based tracking
! Marketing, gaming
! Location Based Experiences
! HMD, fixed screens
! Museums, point of sale, advertising
Typical AR Experiences
92. Demo: colAR
! Turn colouring books pages into AR scenes
! Markerless tracking, use your own colours..
! Try it yourself: http://www.colARapp.com/
93. What Makes a Good AR Experience?
! Compelling
! Engaging, ‘Magic’ moment
! Intuitive, ease of use
! Uses existing skills
! Anchored in physical world
! Seamless combination of real and digital
95. Previous Research Themes
! Zhou et. al. (2008) analyzed all ISMAR papers
Zhou, F., Duh, H. B. L., & Billinghurst, M. (2008). Trends in augmented reality tracking, interaction and display:
A review of ten years of ISMAR. In Proceedings ISMAR 2008, (pp. 193-202). IEEE Computer Society.
96. Possible Research Directions
! Tracking
! Wide area, Reliable indoor, Ubiquitous tracking
! Interaction
! Intelligent systems, Gesture, Collaborative systems
! Displays
! Wide FOV, Retinal Scanning, Contact Lens
! Social Acceptance
! Wearable AR, Handheld AR, Social Interactions
97. Wide Area Tracking
! Process
! Combine panorama’s into point cloud model (offline)
! Initialize camera tracking from point cloud
! Update pose by aligning camera image to point cloud
! Accurate to 25 cm, 0.5 degree over wide area
Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking.In Mixed
and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.
98. Gesture Based Interaction
! Use free hand gestures to interact
! Depth camera, scene capture
! Multimodal input
! Combining speech and gesture
HIT Lab NZ Microsoft Hololens
Meta SpaceGlasses
99. Wide FOV Displays
! Wide FOV see-through display for AR
! LCD panel + edge light point light sources
! 110 degree FOV
Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays: wide
field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH 2014
Emerging Technologies (p. 20). ACM.
100. Social Acceptance
! People don’t want to look silly
! Only 12% of 4,600 adults would be willing to wear AR glasses
! 20% of mobile AR browser users experience social issues
! Acceptance more due to Social than Technical issues
! Needs further study (ethnographic, field tests, longitudinal)
102. Conclusion
! AR seamlessly blends real and virtual imagery
! Interactive in real time, fixed in space
! AR has developed into a mass market technology
! Education, engineering, entertainment
! The technologies to create AR are available
! Display, tracking, interaction
! There are opportunities for significant research