Is Augmented Reality Hype? Where are we now and going? What are right next directions?

1. MOBAS 2012@ 숙명여대 (2012.2.2)
(모바일) 증강현실은Hype(거품)인가?
Woontack Woo (禹 雲澤), Ph.D.
http://twitter.com/wwoo_ct
GIST CTI/U-VR Lab
Gwangju, Korea

2. Outline
VR vs. AR : Is AR Hype?
Where are we now?
What’s Next?
Why Ubiquitous VR, DigiLog & CAMAR?
Summary and Q&A

3. Virtual Reality vs. Augmented Reality
Virtual Reality (out there):
Puts people in a computer-generated world
Augmented Reality (here):
Puts virtual content in a real physical world
Ubiquitous VR : VR over ubiComp Env (here & right now):
Forces the VR/MR/AR to live out here in the world with people
Johnny Mnemonic Minority Report

4. Augmented Reality vs. Virtual Reality
AR VR
Supplements reality, rather than Completely immerse a user inside
completely replacing it a synthetic environment
User maintains a sense of presence While immersed, the user cannot
in real world (Here and Right Now!) see the real world around him (Out
there!)
• Apply to all senses, not just sight: sound, haptics, etc.
• Extend and enhance user’s ability (performance/perception of the
world)

5. What’s AR?
What’s Augmented Reality?
It is really an old idea, e.g. “Terminator vision in 1990”
Milgram's Reality-Virtuality Continuum [94]
Azuma's definition on AR [97]
combines real and virtual
is interactive in real time
is registered in real 3D world

6. AR is Hype?
The web is dead. Long Live the Internet!
Web
Aug. 2010

7. AR is Hype?
Google Trend (VR vs. AR)
A: Virtual Reality Embraced by Businesses
B: Another use for your phone: 'augmented reality
C: Qualcomm Opens Austria Research Center to Focus on Augmented Reality
D: Qualcomm Launches Augmented Reality Application Developer Challenge
E: Review: mTrip iPhone app uses augmented reality
F: Toyota demos augmented-reality-enhanced car windows

8. AR is Hype?
Half century after its birth, the VR is in decline.
Two decades after its birth, the AR is in saturation.
 AR over web for interoperability?
 Smart AR Apps for just-in-time services?
 AR authoring and sharing for sustainable eco-system?
So, will AR be dead?
@ Wired September 2010

9. Hype Cycle of AR
LBS-based AR I might be able to
make money
Mobile 2D AR Nope, no I am wondering
money here how I lived
without AR
3D AR • Mobile IT Infra
• CV HW/SW Tech
I hate AR • AI (context) Tech
Mobile 3D AR
Marker-based AR
CAMAR 2.0
What is CAMAR? U-VR?

10. Hype Cycle of AR 2011
Augmented Reality
• MIT’s annual review; “10 Emerging Tech.s 2007”
• Gartner: top 10 disruptive tech 2008-12
2010 • Juniper: mAR 1.4B downloads/y, revenue $1.5B/y
by 2015 (11M in 2010)
2009
2008

11. Outline
VR vs. AR : Is AR Hype?
Where are we now?
What’s Next?
Why Ubiquitous VR, DigiLog & CAMAR?
Summary and Q&A

12. Augmented Reality
What is involved in the AR process?
Get video from
camera
Estimate
Update the position and
status orientation of
the camera
Render the
Process the
augmented
interaction
scene

13. AR Trend
3D vs. Informative AR
Items 3D AR Informative AR
Main focus Realistic 3D augmentation Information augmentation
Input Camera Sensors (GPS, Compass, etc.)
CoI Object Location (POI)
Core Tech Object recognition/tracking POI detection
3D registration Overlay
Realistic 3D rendering Information filtering
Interaction 3D virtual objects vs. 2D text vs. user
physical space & user
Authoring Offline In-situ
Killer apps Games, training, education Navigation, adv, sns
Platform Standalone Networked
PC-based Sony, Total Immersion WearableAR with HMD
Phone-based Vuforia(Qualcomm), Layar, wikitude, sekaiCam,
Junaio/iLiving(metaio), Argon

14. Tracking @ U-VR Lab 2007-8
BilliARd (2005)
2D Picture Tracking (K.Kim) 3D Vase Tracking (Y. Park) 3D Vase Tracking (Y. Park)
Tracking (W. Baek) Tracking MO (W.Back) Multiple object Tracking
(Y.Park)

15. Tracking @ U-VR Lab 2008-11
ISMAR2008 ISMAR 2009 VC2010
3D objects Tracking, Motion Blur in 3D tracking, Scalable Tracking, VC 2010
ISMAR2008 (K.Kim) ISMAR2009 (Y. Park) (KKim)
ISMAR2010 ISMAR 2011 ICAT2011
Modeling & Tracking, Depth-assisted Tracking, Depth-assisted Detection,
ISMAR2010 (KKim) ISMAR2011 (Y.Park) ICAT2011 (W.Lee)

16. Digilog Apps 2010
Enhance Experience, Engage, Educate & Entertain
Hongkil Dong Technologies in Chosun
Storytelling application Storytelling application
Integrated with virtools* Integrated with virtools*

17. Digilog Apps 2011
DigiLog Miniature
Hong-Kildong Asadal-Asaneo
Storytelling application Storytelling application
Desktop AR Desktop AR + mobile interaction
Integrated with virtools* Integrated with virtools*

18. Why Mobile AR?
Why phone-based AR ?
968M(40M) m-phones worldwide (Korea) in 09
64M(15M) s-phone worldwide(Korea) in 11
Low cost, ubiquity, robust, self-contained
From terminal to gateway:
With advances in tracking and increased
computing power, mobile AR systems are
developing
Leading graphics board vendors release new
chipset that will enable HW-accelerated 3D CG http://www.capv.com/home/Multiclient/MobileImaging.html
on mobile phones
AR Apps in iPhone AppStore
54 in July, 2010 and 605 total. (260K Apps in
Oct., 2010)

19. Mobile UI/UX
Camera for Physical Interaction
Connecting People, Places, Things, and Ideas
 The environment as part of the interface
 Camera phones as “bridging” devices
 Integration with the user’s activities
Click real world !!!

20. Mobile AR : Informative
Informative AR Applications
Wikitude Layar TwitAround
Junaio Aurasma vuforia

21. Informative mobile AR
How does it work?
Illustrations courtesy of GeoVector (left)
and Layar & Junaio by Metaio (right)

22. Mobile AR SDK
QualComm SDK:
http://developer.qualcomm.com/dev/augmented-reality
https://ar.qualcomm.com/qdevnet/

23. Mobile AR SDK
KHARMA by GATECH
KML/HTML Augmented Reality Mobile Architecture
An Open Platform for Delivering Mobile Augmented Reality Experiences
https://research.cc.gatech.edu/polaris/
CAMAR 2.0 by GIST
Context-aware mobile AR
CAMAR + LBS + SNS
CONTEXT
SNS
LBS
CONTENT
Embedded Sensors

24. Outline
VR vs. AR : Is AR Hype?
Where are we now?
What’s Next?
Why Ubiquitous VR, DigiLog & CAMAR?
Summary and Q&A

25. What is still holding us back?
Processing power <10fps ~30fps
Registration accuracy 1cm 1deg
Screen size Limited FOV Optical-see-through
Interaction Limited to touching NUI with 5 senses
screen
GPS Limited speed and AGPS and wifi with
accuracy Vision-assisted
Network BW/latency 400ms pings are not so <50ms for cloud
responsive
Authoring Just emerging In-situ mash-up
Battery power Devour batteries Full day usage
Platform Dual core smart phone Quad core smart phone
Apps Navigation M-commerce

26. What to Expect in 2012?
Matthias Galica (CEO of ShareSquare)
 Quick Response (QR) Codes : simple yet effective…
 Augmented Reality (AR)
 Near Field Communication (NFC)
 The Field (Everybody Else)
 The effects of AR
 Increased transparency
 Reduced privacy
 Timely access to information
 Juniper Research report on AR
 predicted that annual revenues from mobile AR apps will reach
$732M by 2014,
$1.5B by 2015,
up from less than $1M in 2009.


27. What to Expect in 2012?
Where is Digital World headed?
 Wearable, Cloud computing
 Internet of things with sensor proliferation
 Digital immersion with Rich UI/UX over ubiComp
 SNS over networked urbanism
SNS needs AR!
 More than AR needs SNS, SNS needs AR to improve the UI/UX!
 Augmented Content is a King, then Context is a queen consort controlling the King!
So, ubiComp and AR are tightly coupled ‘Outernet’!
 Mobile : smart phone for mobility of people
 Social : layers of AR for cultural/collaborative experience
 Cloud : bringing down all the resources of the internet to reality
 eco-system: publishing sharable cloud content

28. How to make Mobile AR App’s useful?
Beyond the hype:
Moving from simple gimmicky experiments to real world applications
Current AR experience
shows generally low social maturity
Is stalling, waiting for breakthrough either technological or conceptual
Future Impractical AR
AR needs to Useful AR
•3D models placed in a webcam of replacing them with AR gimmicks
Enhance social interactions, instead •Connectivity is always ON
with optional for social interaction
Be little or no interactivity •The cloud is the computer
•Layered animation with little or no •Engaging, persistent experience for
feedback the user
•MAR that uses solely GPS, compass,
and accelerometer input •[LBS + SNS + MAR] drawing from
networked large DBs with
•MAR where geo-tagging doesn't customization features
serve an everyday purpose

29. What’s Next?

30. Social AR?
Issues of Social AR
Physical self along with a digital profile
Unauthorized Augmented Advertising
Privacy: Augmented Behavioral Targeting
Safety: Physical danger
Spam

31. Social AR?
Cases against AR

32. Outline
VR vs. AR : Is AR Hype?
Where are we now?
What’s Next?
Why Ubiquitous VR, DigiLog & CAMAR?
Summary and Q&A

33. Smart Holistic Space
Convergence
Virtual space
Between Real & Virtual
Seams btw two worlds How to
combine
Seamlessly?
Real space
Space for experience and expression
Personal space (3rd skin): an individual has a complete control of data/info flow, i.e.,
incoming, filtering, selecting, outgoing (using a user profile)
Social space: a group has some control, ownership and regulatory power, but as a
result of negotiations and transactions
General space: a temporary quality and free access, but regulated by a public social
contract

34. What’s VR, AR and UVR?
Dual space {R, R’}
RE RE’
V
RE
RE
R R’
VE’
RE RE’
VE’

35. What’s Ubiquitous VR?
Woo’s Definition [11] : U-VR is
3D Link btw dual (real & virtual) spaces with
additional info
CoI augmentation, not just sight: sound,
haptics, smell, taste, etc.
Bidirectional UI for H2H/H2S/S2H/S2S
communication in dual spaces
Virtual space
How to U-Content
Seamless Augmentation
Link btw dual spaces
Seamlessly? LINK
CoI
Real space
Social Networks

36. AR for DigiLog Experience
AR platform for DigiLog Experience will be
How to avoiding the fate of VR ?
"to look for DigiLog, the intersections • Multicore/GPGPU
between the digital world (3D, SNS) • Object recognition/tracking
and analog world (CoI)” • Light source estimation
Realistic 3D • Physics engine
•
augmentation AI engine
Context-
awareness
• Interaction
• CoI-aware
• Mash-up authoring
• Environmental context: User • SNS
• User context: knowledge, participation • Extension of human
experience, preference
perception
• Just-in-time visualization

37. Technical Challenges
CoI Localization:
Context of Interest (CoI): Space vs. Object
Accurate CoI Recognition and Tracking
3D Interaction
Ubiquitous Augmentation
LBS/SNS-based Authoring and Mash-up
Smart UI for Intuitive Visualization
AR-Infography + Organic UI
Networking and public DB management
U-VR ecosystem with SNS, LBS, CaS
Interoperable representation
HW wish list
Better camera/GPS/compass, CPU/GPU, I/O, battery

38. CAMAR 2.0: Context-aware mobile AR
Sharing
Direct
response
Mashup
Reflective
response Planned
response

39. CAMAR 2009-10
Mobile AR: WHERE to augment?
Concept Context-aware Annotation (H. Kim)
Plan Recognition (Y. Jang) Multi-page Recognition (J.Park) LBS + mobile AR (W. Lee)
[Paper] Y. Jang and W. Woo, “Stroke-based semi-automatic region of interest detection for in-situ painting recognition", 14th International Conference on Human-Computer Interaction (HCII 2011), Jul. 9-14,
Orlando, USA, accepted.
[Patent] W. Woo, Y. Jang, “현장에서 그림 인식을 위해 선긋기 상호작용을 통한 반자동식 관심영역 검출 알고리즘 ,” 2010. (출원 중)

40. Context @ U-VR Lab 2010-11
Adaptive Content Recommendation
Recommend user-preferred content
Retrieve content efficiently using hierarchal
context model
J. Han, H. Schmidtke, X. Xie, and W. Woo, “Adaptive Content Recommendation using Hierarchical Context Model with Granularity for Mobile Consumer,” in Pers. Ubiqu. Comp
ut., pp.000-000, 2012. (Submitted)

41. Interaction @ U-VR Lab 2011
ARWand: Phone-based 3D Object Manipulation in AR
Exploits a 2D touch screen, a 3DOF accelerometer, and compass sensors information
to manipulate 3D objects in 3D space
Design transfer functions to map the control space of mobile phones to an AR display
space
Taejin Ha, Woontack Woo, "ARWand: Phone-based 3D Object Manipulation in Augmented Reality Environment," ISUVR, pp. 44-47, 2011.

42. Interaction @ U-VR Lab 2011
ARWand: Phone-based 3D Object Manipulation in AR
Experiment and application
◦ Low control-to-display gain: a sophisticated translation could be possible but this requires a significant amount of
clutching
◦ High gain could reduce the frequent clutching, but accurate manipulation could be difficult
◦ Therefore, we need to consider an optimal control function that satisfies both fast and accurate manipulation
Taejin Ha, Woontack Woo, "ARWand: Phone-based 3D Object Manipulation in Augmented Reality Environment," ISUVR, pp. 44-47, 2011.

43. CAMAR @ U-VR Lab 2010
In Situ Video Tagging on Mobile Phones
In situ Planar Target Learning on Mobile Phones
Sensor-based Automatic Fronto-parallel View Generation
Fast Vanishing Point Computation
Input Image
Horizontal Vanishing Points
target ? Estimation
Fronto-parallel View
Generation
Target Learning
on the mobile GPU
Real-time Detection
W. Lee, Y. Park, V. Lepetit, W. Woo, "In-Situ Video Tagging on Mobile Phones," Circuit Systems and Video Technology, IEEE Trans. on, Vol. 21, No. 10, pp. 1487-1496, 2011.
W. Lee, Y. Park, V. Lepetit, W. Woo, "Point-and-Shoot for Ubiquitous Tagging on Mobile Phones," ISMAR10, pp. 57-64, 2010.

44. CAMAR @ U-VR Lab 2010
In Situ Video Tagging on Mobile Phones
In situ Augmentation of Real World Objects Vertical Target Learning & Detection
데모비디오
- In situ augmentation of real world objects - Learning a vertical target from an arbitrary
without pre-trained database viewpoint
- Fast target learning in a few seconds - Vanishing point-based fronto-parallel view
- Real-time detection from novel viewpoints generation
- Real-time detection from unseen viewpoints
Available at : http://youtu.be/vaaFhvfwet8
W. Lee, Y. Park, V. Lepetit, W. Woo, "In-Situ Video Tagging on Mobile Phones," Circuit Systems and Video Technology, IEEE Trans. on, Vol. 21, No. 10, pp. 1487-1496, 2011.
W. Lee, Y. Park, V. Lepetit, W. Woo, "Point-and-Shoot for Ubiquitous Tagging on Mobile Phones," ISMAR10, pp. 57-64, 2010.

45. CAMAR @ U-VR Lab 2011
Interactive Annotation on Mobile Phones for Real and Virtual
Space Registration
Allows to quickly capture the dimensions of a room
Operates at interactive frame-rates on mobile device
and provides simple touch-interaction
Serves as anchors for linking virtual information to
the real space represented by the room
H. Kim, G. Reitmayr and W.Woo, “Interactive Annotation on Mobile Phones for Real and Virtual Space Registration,” in Proc. ISMAR 2011, pp.265-266, Oct. 2011.

46. CAMAR @ U-VR Lab 2011
Interactive Annotation on Mobile Phones for Real and Virtual
Space Registration
Demo #1 데모비디오 Demo #2
In office room and seminar room, In ART gallery,
- Capture the dimensions of a room, - Load an AR zone-based room model
approximated as a room - Annotate a virtual content on rectangular areas
- Annotate a virtual content on rectangular on the room’s surface
areas on the room’s surface
Youtube share link http://www.youtube.com/watch?v=I00I-phmPbI

47. CAMAR @ U-VR Lab 2011
In-situ AR Mashup for AR Content Authoring
Easily create AR contents from Web contents
Context-based content recommendation
 User-similarity, item similarity, social relationship
Configure AR content sharing setting
 To Whom, When, in What conditions
H.Yoon and W.Woo, “CAMAR Mashup: Empowering End-user Participation in U-VR Environment,” in Proc. ISUVR 2009, pp.33-36, July. 2009. (Best Paper Award)
H.Yoon and W.Woo, “Concept and Applications of In-situ AR Mashup Content,” in Proc. SCI 2011, pp. 25-30, Sept. 2011.

48. CAMAR @ U-VR Lab 2011
In-situ AR Mashup for AR Content Authoring
In-situ Content Mashup
• Extract query keywords based on context of object
• Content recommendation based on personal context and social context
• Access related Flickr, Twitter, Picasa contents in-situ
H.Yoon and W.Woo, “CAMAR Mashup: Empowering End-user Participation in U-VR Environment,” in Proc. ISUVR 2009, pp.33-36, July. 2009. (Best Paper Award)
H.Yoon and W.Woo, “Concept and Applications of In-situ AR Mashup Content,” in Proc. SCI 2011, pp. 25-30, Sept. 2011.

49. Some Issues: Standard
Interoperability (Standard)
W3C : HTML5 (ETRI): http://www.w3.org/2010/06/w3car/report.html
web3D : X3D (Fraunhofer)
OGC : ARML (GATECH)
ISO/IEC JTC1: SC24 WG9 [X3D(KU), XML(GIST)] vs. SC29 [X3D(ETRI)]
Real Content, Virtual Content, ***Link***
POI: Metadata Metadata Format
MR
id ID
2
id ID
who author
who owner, author
when creation time
when creation time
where location Real where real-virtual relation
what ID of real and virtual
what tag, description Contents contents
how movable, access how access
AR/MR Link Lookup Table Link Link Link Link
1
Virtual Virtual Virtual Virtual
Contents Content Contents Contents
#1 #2 #3 #4
id ID
AR Content: Metadata who owner, author
3
when creation time
where location
what tag, type
how movable, access

50. Standard by GIST
Server/DB Mobile Device Real World
Feature Points or Image Acquisition (Grab)
Content ID
-FP Recognition AR
-AR Content Management Viewer Object of Interest (OOI)
Context of Interest
WHER WHAT
LID WHO WHEN HOW
E
V-
R-ID
Image
V-ID R-ID, R-Pose (COI)
GUID1 … … VID1 … Point of Interest (POI)
Pose ID1
GUID2 …
V- Image
GUID3 … … Pose ID1
VID4 …
Link DB
TTAK.KO-10.0434 TTAK.KO-10.0428
Data Representation for QR Code Data Representation
AR/MR (=Link) for (Mobile) AR

51. What’s Next?
Next?
People don’t necessarily want to walk around holding phone in front of their faces
CAMAR 2.0 vs. Wearable CAMAR 2.0
<Open + Participate + Share>
LBS + In-situ Mashup + SNS + CAMAR for sustainable AR eco-system

52. Outline
VR vs. AR : Is AR Hype?
Where are we now?
What’s Next?
Why Ubiquitous VR, DigiLog & CAMAR?
Summary and Q&A

53. Q&A
“The future is already here. It is just not uniformly distributed”
by William Gibson (SF writer)
More Information
Woontack Woo, Ph.D.
Twitter: @wwoo_ct
FB: @wtwoo
Mail: wwoo@gist.ac.kr
Web: http://uvr.gist.ac.kr
1st UVR Workshop @ KAIST GSCT, Mar. 16-17, 2012
7th ISUVR 2012 @ KAIST GSCT, Aug. 22 - 25, 2012