Uplin q'14 giuliano maciocci - bringing mobile vision to wearables digital eyewear

1. 1

2. Bringing Mobile Vision
to Digital Eyewear
Giuliano Maciocci, Staff Product Manager
Andrew Everitt, Principal Engineer
Qualcomm Cambridge Ltd.
The Qualcomm Vuforia SDK is a product of Qualcomm Connected Experiences, Inc.

3. 3
Digital play
Enabling apps
to see

4. 4
Digital Eyewear Shipments
(Millions of units)
Most shipped to North America
2013/2014
North America, Western Europe
and Asia Pacific are projected to
be the main markets through 2019
Source: ABI Research
0
5
10
15
20
25
30
35
40
2013 2014 2015 2016 2017 2018 2019
Middle East/Africa Latin America Asia Pacific
Eastern Europe Western Europe North America

5. 5
“Augmented Reality Will Be the
Magic Key for Smart Glasses”
ABI Research, October 2013

6. 6
Key use cases
The potential impact of vision-based Digital Eyewear
Enterprise Education Shopping Gaming!
Industrial and Logistics
Step-by step instructions
Maintenance tasks
Data visualization
Employee training
First responders
Schools and homes
Virtual experiments
Augmented textbooks
3D visualization
Better product previews
In-situ product previews
Enhanced print advertising
More immersive experiences
Environmental interaction
Live tabletop gaming
AR + Toys
Hybrid reality gaming
Real time data + accurate
physical tracking for field
and site-based applications
Facilitating active learning Immersive, scale-accurate product
visualization. “Place before you buy.”
Taking vision-based gaming
and toy companion applications to the
next level of interaction.

7. 7
Challenges
Taking vision-based experiences beyond mobile
“Look-to” displays effectively place a screen in the corner of your vision.
− Limited scope for moving beyond traditional mobile AR use cases.
See-through displays (both actual and camera-based) present new opportunities
for immersive, line-of-sight AR – but also serious challenges.
− Need to augment the real world, not a camera feed.
− Each user’s eyes are different.
− Binocular vision needs to be handled carefully (comfort, eye strain).
− Tracking latency becomes more apparent.
New ways of interacting need to be accommodated.
− The touchscreen is gone.

8. 8
10,000+unique applications
100,000+registered developers
An industry leading vision platform in mobile
Source: Qualcomm Connected Experiences, Inc.
125,000,000+
Vuforia-powered app installs

9. 9
With a network of
preferred
developers
around the world
Global developer adoption
North
America
Latin
America
Europe
Middle East
& Africa
India
SE / Asia Pacific
Japan & Korea
China
13%
17%
26%
6%
7%
7%
9%
15%
Numbers reflect percentages of total Vuforia developers as of May 2014

10. 10
And now, in stereo!

11. 11
Introducing: Vuforia
for Digital Eyewear

12. 12
VDE Intro video
Placeholder for VDE video

13. 13
Vuforia SDK for Digital Eyewear
Feature overview
User calibration
API calculates
the complex
relationship
between target,
camera, display
and eye
Brings the Vuforia vision platform to the latest Digital Eyewear
Video background
removal for
augmentation directly
on real
world targets in
see-through
Digital Eyewear
Stereoscopic
content
authoring on
binocular
Digital Eyewear
Unity extension and
scene camera prefab
supporting
drag-and-drop stereo
scene rendering
Virtual Buttons
functionality
allows direct on-
target interaction
Samples
for both OpenGL
and Unity
tailored
to Digital
Eyewear

14. 14

15. 15
Target Camera Display Eye

16. 16
User calibration
One-time calibration for all Vuforia SDK for Digital Eyewear applications
Solves a key challenge of AR for see-through eyewear: precise line-of-sight alignment of
augmented content.
4-step calibration app (2 steps x eye).
− Simple one-time user-driven task: aligning a virtual rectangle to a real-world Vuforia target.
− Generates a projection matrix for each eye, encapsulating camera-display-eye relationship.
− Projection matrices are stored locally in the user’s calibration profile.
− Stored projection matrices accessible via VDE API.
Calibration API: roll your own calibration app.
− Calibration API lets you embed calibration routine into your own applications.
− Personalize the calibration process with your own targets, graphics, GUI.
− Store and access multiple calibration profiles for multiple users.

17. 17

18. 18
Stereoscopic rendering
Rendering 3D content for binocular Digital Eyewear
Left/Right eye pose matrices supported in OpenGL.
Drag and Drop Unity prefab replaces ARCamera.
− Side-by-side split-screen rendering.
− Supports simple conversion of Unity content to stereo.
− Leverages stored projection matrices in user’s calibration profile to
ensure correct alignment and perspective correction of AR content
over real world target.
Mobile AR (top) vs. Eyewear AR (bottom). Black background
appears transparent on Digital Eyewear.

19. 19

20. 20
Video background removal
Support for see-through Digital eyewear
Traditional mobile use case aligns AR content over camera’s video feed.
In see-through Digital eyewear, we need to align AR content directly over an actual real-
world target, as seen by the user.
Easily toggle video background on or off depending on target device: off for see-through
or on for occluded.
Video background removal + calibration achieves correct alignment over real-world
targets.

21. 21

22. 22
Virtual Buttons
Direct on-target interaction
Allows developers to designate regions of a
physical target as buttons.
Touch the target to interact.
Placing a finger over the designated target
region triggers an event.
Provides compelling interaction when
Eyewear input methods are limited or not
convenient.

23. 23

24. 24
OpenGL and Unity Samples
Bootstrapping development for VDE
OpenGL samples:
− Planar target sample + stereo
Unity samples:
− Planar target sample with stereo rendering
− Virtual Buttons sample application
− Step-by-step instructional sample application

25. 25
Supported Devices

26. 26
Epson Moverio BT-200

27. 27
ODG R-7

28. 28
Samsung Gear VR Innovator Edition

29. 29
API Overview

30. 30
Digital Eyewear API
New classes: QCAR::Eyewear, QCAR::EyewearCalibrationProfileManager, QCAR::EyewearUserCalibrator
A set of APIs to configure and provide rendering support for the Digital Eyewear device that
your app is running on.
Query device for stereo support and enablement.
Switch between the device’s stereo and mono display modes (if supported).
Query whether device is see-thru.
Retrieve projection matrices for each eye using calibration profile (see-thru devices) or
correction for stereo rendering (occluded devices).
Profile manager to manage up to 10 user profiles for calibration (see-thru devices only).
Calibrator class to support creating your own calibration experience (see-thru devices only)

31. 31
Unity extension overview

32. 32

33. 33

34. 34

35. 35

36. 36

37. 37

38. 38

39. 39

40. 40
Running on device
In Unity simply ‘Build And Run’ for Android to create an APK.
APK works on any Vuforia SDK for Digital Eyewear supported device.
Calibration for see-thru devices handled transparently by SDK.
Unity Extension handles stereo vs. mono rendering and removal of video background on see-
thru devices.
On see-thru devices user calibration shared between all SDK applications.

41. 41
Takeaways

42. 42
Takeaways
Vuforia SDK for Digital Eyewear
Delivers powerful, immersive 3D content to an emerging generation of stereoscopic see-
through digital eyewear – not just a small screen in the corner of your field of view.
Enables enterprise applications on see-though digital eyewear such as the Epson Moverio
BT-200 and powerful gaming, shopping and educational experiences on the new mobile
VR/AR solution, the Samsung GearVR.
Addresses key challenges of developing AR experiences for digital eyewear, such as precise
calibration and object alignment, enabling the display of content that is anchored to the real
world and dynamically adjusts to the wearer’s eyes.

43. 43
Call to action
Join the vanguard of Digital Eyewear developers.
Think of a great vision-based experience for see-through or mobile AR/VR Digital Eyewear
− Enterprise
− Education
− Shopping
− Gaming

44. 44
Sign up for the beta program
vuforia.com/beta

45. 45
For more information on Qualcomm, visit us at:
www.qualcomm.com & www.qualcomm.com/blog
©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Qualcomm and Vuforia are trademarks of Qualcomm Incorporated, registered in the United
States and other countries. Uplinq and Smart Terrain are trademarks of Qualcomm Incorporated.
All Qualcomm Incorporated trademarks are used with permission. Other products and brand
names may be trademarks or registered trademarks of their respective owners.
References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm
Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate
structure, as applicable.
Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of
its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm
Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering,
research and development functions, and substantially all of its product and services businesses,
including its semiconductor business, QCT.
Thank you FOLLOW US ON: