1. COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
July 17th 2014
Lecture 1: Introduction
mark.billinghurst@hitlabnz.org

2. Mark Billinghurst
 PhD Electrical Engineering
 University of Washington
 Interaction Design
 Museum experiences
 Tools for designers
 Augmented Reality
 Mobile AR, Evaluation,
 Multimodal Interfaces, Collaborative
 Collaboration
 Enhanced FtF and remote collaboration
 Social networking

3. Overview
 One two hour lecture a week
 Thursday 11am – 1pm
 You will learn
 Introduction to Augmented Reality
 Augmented Reality technology
 AR Interaction techniques
 Interaction Design
 AR authoring tools
 Research directions in AR
 Complete a simple project

4. Course Outline
 Wk 1 (July 17th): Introduction to Augmented Reality (AR)
 Wk 2 (July 24th): AR Technology (Assign 1)
 Wk 3 (July 31st): AR Tracking
 Wk 4 (Aug 7th): Designing AR Interfaces (Assign 2)
 Wk 5 (Aug 14th): Project presentation + AR Interaction
 Wk 6 (Aug 21st): Mobile AR (Assign 3)
 Wk 9 (Sept 11th): Evaluating AR Interfaces (Assign 4)
 Wk 10 (Sept 18th): AR research Directions
 Wk 11 (Sept 25th): Final Project Presentations

5. Assessment - Update
 Research project – 40%
 Group work (2-4 people)
 Due Sept 27th
 Four Class Assignments – 20 %
 Design, programming, individual work
 Final Exam – 40%
 Exam date TBD

6. Introduction

7. A Brief History of Time
 Trend
 smaller, cheaper, more functions, more intimate
 Technology becomes invisible
 Intuitive to use
 Interface over internals
 Form more important than function
 Human centered design

8. A Brief History of Computing
 Trend
 smaller, cheaper, faster, more intimate, intelligent objects
 Computers need to become invisible
 hide the computer in the real world
- Ubiquitous / Tangible Computing
 put the user inside the computer
- Virtual Reality

9. Invisible Interfaces
Jun Rekimoto, Sony CSL

10. Graphical User Interfaces
 Separation between real and digital worlds
 WIMP (Windows, Icons, Menus, Pointer) metaphor

11. Ubiquitous Computing
 Computing and sensing embedded in real world
 Particle devices, RFID, motes, arduino, etc

12. Virtual Reality
 1985…

13. Virtual Reality
 Immersive VR
 Head mounted display, gloves
 Separation from the real world

14. Occulus Rift
 $300 USD
 360 degree head tracking
 100 degree field of view

15. 1977 – Star Wars

16. Augmented Reality Definition
 Defining Characteristics [Azuma 97]
 Combines Real and Virtual Images
- Both can be seen at the same time
 Interactive in real-time
- The virtual content can be interacted with
 Registered in 3D
- Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

17. 2008 - CNN

18.  Put AR pictures here
Augmented Reality Examples

19. AR vs VR
 Virtual Reality: Replaces Reality
 Scene Generation: requires realistic images
 Display Device: fully immersive, wide FOV
 Tracking and Sensing: low accuracy is okay
 Augmented Reality: Enhances Reality
 Scene Generation: minimal rendering okay
 Display Device: non-immersive, small FOV
 Tracking and Sensing: high accuracy needed

20. Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

21. Augmented Virtuality
 VR with windows into the real world

22. Metaverse
 Neal Stephenson’s “SnowCrash”
 The Metaverse is the convergence of:
 1) virtually enhanced physical reality
 2) physically persistent virtual space
 Metaverse Roadmap
 http://metaverseroadmap.org/

23. Metaverse Dimensions
• Augmentation technologies that layer information onto
our perception of the physical environment.
• Simulation refers to technologies that model reality
• Intimate technologies are focused inwardly, on the
identity and actions of the individual or object;
• External technologies are focused outwardly, towards
the world at large;

24. Metaverse Components
 Four Key Components
 Virtual Worlds
 Augmented Reality
 Mirror Worlds
 Lifelogging

26. Mirror Worlds
 Mirror worlds are informationally-enhanced
virtual models of the physical world.
 Google Earth, MS Street View, Google Maps

28. LifeLogging
 Technologies record and report the intimate
states and life histories of objects and users
 Nokia LifeBlog, Nike+, FitBits

29.  Steve Mann - LifeLogging

30. Gordon Bell: LifeLogging
1 TB to store 65 years of data

31. Narrative Clip
 Wearable camera
 Automatic picture capture - 2 pics/minute
 http://getnarrative.com

32. Summary
 Augmented Reality has three key features
 Combines Real and Virtual Images
 Interactive in real-time
 Registered in 3D
 AR can be classified alongside other technologies
 Invisible Interfaces
 Milgram’s Mixed Reality continuum
 MetaVerse

33. AR History

34. A Brief History of AR (1)
 1960’s: Sutherland / Sproull’s
first HMD system was see-
through

35. A Brief History of AR (2)
F16 – Head Up Display

36. A Brief History of AR (3)
1960 - 70’s: US Air Force helmet mounted
displays (T. Furness)

37. A Brief History of AR (4)
1970 - 80’s: US Air Force Super Cockpit (T. Furness)

38. A Brief History of AR (5)
 Early 1990’s: Boeing coined the term “AR.” Wire harness
assembly application begun (T. Caudell, D. Mizell).

39. A Brief History of AR (6)
 1994: Motion stabilized display [Azuma]
 1995: Fiducial tracking in video see-through [Bajura / Neumann]
 1996: UNC hybrid magnetic-vision tracker

40. A Brief History of AR (7)
 1996: MIT Wearable Computing efforts
 1998: Dedicated conferences begin (ISMAR)
 Late 90’s: Collaboration, outdoor, interaction
 Late 90’s: Augmented sports broadcasts

41. History Summary
 1960’s – 80’s: Early Experimentation
 1980’s – 90’s: Basic Research
 Tracking, displays
 1995 – 2005: Tools/Applications
 Interaction, usability, theory
 2005 - : Commercial Applications
 Games, Medical, Industry

42. 2007 - AR Reaches Mainstream
 MIT Technology Review
 March 2007
 list of the 10 most
exciting technologies
 Economist
 Dec 6th 2007
 Reality, only better

43. Gartner Hype Cycle

44. 2009 - AR in Magazines
 Esquire Magazine
 Dec 2009 issue
 12 pages AR content
 Many Others
 Wired
 Colors
 Red Bull
 Etc

45. Google Searches for AR

46. 2008 - Browser Based AR
 Flash + camera + 3D graphics
 High impact
 High marketing value
 Large potential install base
 1.6 Billion web users
 Ease of development
 Lots of developers, mature tools
 Low cost of entry
 Browser, web camera

49. Impact of Web-based AR
 Boffswana Living Sasquatch
 http://www.boffswana.com/news/?p=605
 In first month
 100K unique visits
 500K page views
 6 minutes on page

50. 2005 - Mobile Phone AR
 Mobile Phones
 camera
 processor
 display
 AR on Mobile Phones
 Simple graphics
 Optimized computer vision
 Collaborative Interaction

51. AR Advertising (HIT Lab NZ 2007)
 Txt message to download AR application (200K)
 See virtual content popping out of real paper advert
 Tested May 2007 by Saatchi and Saatchi

52. 2008: Location Aware Phones
Nokia NavigatorMotorola Droid

53. 2009 - Outdoor Information Overlay
 Mobile phone based
 Tag real world locations
 GPS + Compass input
 Overlay graphics data on live video
 Applications
 Travel guide, Advertising, etc
 Wikitude, Layar, Junaio, etc..
 Android based, Public API released

54. Layar (www.layar.com)
 Location based data
 GPS + compass location
 Map + camera view
 AR Layers on real world
 Customized data
 Audio, 3D, 2D content
 Easy authoring
 Android, iPhone

57. AR Today
 Key Technologies Available
- Robust tracking (Computer Vision, GPS/sensors)
- Display (Handheld, HMDs)
- Input Devices (Kinect, etc)
- Developer tools (Qualcomm, Metaio, ARTW)
 Commercial Business Growing
- Gaming, GPS/Mobile, Online Advertisement
• >$5 Billion USD by 2016 (Markets andMarkets)
• >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)

58. Google Glass (2011 - )

59. Epson Moverio BT-200
▪ Stereo see-through display ($700)
▪ 960 x 540 pixels, 23 degree FOV, 60Hz, 88g
▪ Android Powered, separate controller
▪ VGA camera, GPS, gyro, accelerometer

60.  Web based AR
 Flash, HTML 5 based AR
 Marketing, education
 Outdoor Mobile AR
 GPS, compass tracking
 Viewing Points of Interest in real world
 Eg: Junaio, Layar, Wikitude
 Handheld AR
 Vision based tracking
 Marketing, gaming
 Location Based Experiences
 HMD, fixed screens
 Museums, point of sale, advertising
Typical AR Experiences

61. AR Business Today
 Marketing
 Web-based, mobile
 Mobile AR
 Geo-located information and service
 Driving demand for high end phones
 Gaming
 Mobile, Physical input (Kinect, PS Move)
 Upcoming areas
 Manufacturing, Medical, Military

62. Some Commercial AR Companies
 ARToolworks (http://www.artoolworks.com/)
 ARToolKit, FLARToolKit, SDKs
 Metaio (http://www.metaio.com/)
 Marketing, Industry, SDKs
 Total Immersion (http://www.t-immersion.com/)
 Marketing, Theme Parks, AR Experiences
 Qualcomm (http://www.vuforia.com/)
 Mobile AR, Vuforia SDK
 Many small start-ups

64. Summary
 Augmented Reality has a long history going
back to the 1960’s
 Interest in AR has exploded over the last few
years and is being commercialized quickly
 AR is growing in a number of areas
 Mobile AR
 Web based AR
 Marketing experiences

65. Sample AR Applications

66. Applications
 Medicine
 Manufacturing
 Information overlay
 Architecture
 Museum
 Marketing
 Gaming

67. Applications: medical
 “X-ray vision” for surgeons
 Aid visualization, minimally-invasive operations.
Training. MRI, CT data.
 Ultrasound project, UNC Chapel Hill.
Courtesy
UNC
Chapel
Hill

68. Medical AR Trials
 Sauer et al. 2000 at Siemens
Corporate Research, NJ
 Stereo video see through
F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a
Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial,
MICCAI 2002

69. Assembly and maintenance
© 1993 S. Feiner, B. MacIntyre, &
D. Seligmann, Columbia University
© 1996 S. Feiner, B. MacIntyre, &
A. Webster, Columbia University

70. PS3 - Eye of Judgment (2007)
 Computer Vision Tracking
 Card based battle game
 Collaborative AR
 October 24th 2007

72. AR Books – Markerless Tracking

73. AR Annotations
Columbia
University
HRL
© 1993 S. Feiner, B. MacIntyre,
M. Haupt, & E. Solomon,
Columbia University
© 1997 S. Feiner, B. MacIntyre,
T. Höllerer, & A. Webster,
Columbia University

74. Broadcast TV

75. Interactive Museum Experiences
 BlackMagic
 Virtual America’s Cup
 410,000 people in six months
 MagicPlanet
 TeManawa science museum
 Virtual Astronomy
 Collaborative AR experience
 ARVolcano
 Interactive AR kiosk
 Scienceworks museum, Melbourne

76. Digital Binocular Station
http://www.DigitalBinocularStation.com/

77. Museum Archeology
 LifePlus (2002-2004)
 Natural feature tracking
 Virtual characters
 Mobile AR system
 Archeoguide (2000-2002)
 Cultural heritage on-site guide
 Hybrid tracking
 Virtual overlay

78. Sales and Marketing
 Connect with brands and branded objects
 Location Based Experiences
 Lynx Angels
 Web based
 Rayban glasses
 Mobile
 Ford Ka campaign
 Print based
 Red Bull Magazine

79. Summary
 AR technology can be used to develop a wide
range of applications
 Promising application areas include
 Games
 Education
 Engineering
 Medicine
 Museums
 Etc..

80. Things to Do..
 Find your favourite YouTube video showing
an AR interface – send to Mark
 Try AR for yourself
 Install Junaio, or other mobile AR application
 Read Articles
 Find a friend for the project
 Think about project ideas