Workshop slides for Hack Reality event at the State Library of NSW October 2017 by https://awe.media - Mixed Reality. In the web. On any device.

2. The promise…
https://www.disneyresearch.com/publication/magic-bench/

3. The reality
https://www.disneyresearch.com/publication/magic-bench/

4. A bit about us

5. What we’ll
be doing
today
WHAT IS MR/XR/AR/VR
o History – know what’s come before
o Making sense of the language & labels
o Different technologies & approaches
THE UX LAYER
THE EXPERIENCE BUILD IT
AND WHAT WE
WON’T…
o The first use experience & roadblocks
o Making sense of a spatial environment
o Guiding principles
o Planning a mixed reality experience
o Content
o ACTIVITY - Capturing & creating content
o ACTIVITY – build a prototype MR
experience using data from the
State Library NSW

6. STRANGE BEASTS
https://vimeo.com/209070629
HYPER-REALITY
https://vimeo.com/166807261
UNCANNY VALLEY
https://www.youtube.com/watch?v=UXX0TRtg5Vk
Pick a future…

7. What are VR & AR?
“It takes 20 years
to become an
overnight success”
EDDIE CANTOR

8. A short
history…
1962

9. 1960 1966
TELESPHERE MASK SWORD OF DAMOCLES

10. 1982
Tron

11. 1990’s
Jaron Lanier

12. 20031995
SECOND LIFENINTENDO VIRTUAL BOY

13. 2010
Kinect
AND FROM THERE WE KNOW WHAT HAPPENS FOR VR…

14. So where did AR come from?
1990
Tom Caudell

15. 1994Milgram’s Mixed
Reality Continuum
Image source: https://cdn2.hubspot.net/hubfs/479277/blog-files/Reality_Continuum.png

16. Milgram’s Mixed
Reality Continuum
Demo running entirely in a web browser
https://www.youtube.com/watch?v=fhJSfqBYjKE

17. Definition of AR
Augmented Reality allows the user to see the real world,
with virtual objects superimposed upon or composited with
the real world. Augmented Reality:
RON AZUMA (1997)
1. Combines real and virtual
2. Interactive in real time
3. Registered in 3-D

18. 360º
WHAT DO WE MEAN BY 360º OR PANOSPHERE?
TYPE OF “360º” – PANORAMA, ECQUIRECTANGULAR, SPHERICAL
STATIC VS INTERACTIVE
USE CASES – TELEPORTING, VIRTUAL TOUR-GUIDING
EVEREST IN 360
TOURISM - SKI RESORT
EDUCATION
CAN BE LINKED TO PHYSICAL LOCATIONS

19. VR
HOW IS VR DIFFERENT TO 360º?
DISCOVERY VS. LED THROUGH
CHALLENGES
MORE COMPLEX STORYTELLING, CREATION
MORE EXPENSIVE THAN 360º CONTENT

20. The multi-flavoured
landscape of AR
LOCATION
o Relative
o Geo or “Fixed”
o Sensor
COMPUTER VISION
o Recognition / Visual Search
o Tracking
o SLAM

21. Content is relative to the viewer rather than a physical location
Location-based AR
RELATIVE

22. HOW DOES IT WORK?
HOW ACCURATE IS IT?
USE CASES
Location-based AR
FIXED or GEO

23. WHAT IS IT?
WHY WOULD YOU USE IT?
USE CASES
Location-based AR
SENSOR-BASED

24. Image
Object
Facial
HOW DOES IT WORK?
USE CASES
Computer vision
VISUAL SEARCH

25. 2D planes
Spaces – SLAM (e.g. ARKit & ARCore)
DEPTH SENSING
Computer vision
TRACKING

26. Touch
Controllers
Sensors – motion / gesture / depth
Physical
input
Image credit: https://gest.co/

27. Haptics
Audio
Physical
output
Image credit: https://twitter.com/MixedReality11/status/881846276328968192

28. SLIDE:Modes of mixed reality
PUBLIC PERSONAL PRIVATE INTIMATE
Credit: awe.media

29. PUBLIC
WHOLE BODY IN FOV MORE SPECTATORS
MR
MODES
LIMITED DISTRIBUTION

30. INTIMATE
MR MODES
WHOLE BODY / UPPER TORSO
IN FOV
FEWER SPECTATORS
WIDER DISTRIBUTION

31. PERSONAL
MR MODES
ONLY EXTREMETIES OR HEAD
IN FOV
1-2 PARTICIPANTS
WIDE DISTRIBUTION & LOCATIONS

32. INTIMATE
MR MODES
ONLY EXTREMETIES IN FOV MOSTLY 1 PARTICIPANT *
STILL LIMITED DISTRIBUTION ALTHOUGH GROWING
* Except for shared VR and AR experiences

33. HOW VALUABLE IS THE MARKET GOING TO BE?
ELLA
PRESENTATION
0
10
20
30
40
50
60
70
80
90
2016 2017 2018 2019 2020 2021 2022 2023 2024 2025
Revenue(US$bns)
Hardware Software
The Goldman Sachs Group, Inc. 2016, Profiles in Innovation: Virtual & Augmented Reality
Goldman Sachs Global Investment Research
$35B
$45B
expects Virtual &
Augmented Reality
to become an
US$80+ billion
dollar market by
2025, roughly the
size of the desktop
market today.
Goldman Sachs
Research

34. Samsung
21%
Sony
19%
HTC
8%
Oculus
4%
Others
47%
HMDs shipped Jan-Mar this year
Source: IDC Worldwide Quarterly Augmented and Virtual Reality Headset Tracker report
Note: “Others” is predominantly Daydream and
cardboard devices.

35. Samsung
21%
Sony
19%
HTC
8%
Oculus
4%
Others
47%
Source: IDC Worldwide Quarterly Augmented and Virtual Reality Headset Tracker report

36. 37
iOS is slipping markedly
The smartphone market – UNITS SHIPPED

37. How many people can
use AR & VR right now?
Installed web browsers
26 MILLION (END 2017)
Includes both the mobile and
HMD users above
2 BILLION
3 BILLION
Mobile devices
HMD only

38. How do you view VR & AR?
Have a suitable HMD
v
1
2Have a valid account
with the HMD vendor's
app store
3
Complete the HMD
vendor's first use
4
v
Search for & find the app
in their app store
5 Download/install
the app through
their app store
6Often requires system
or app updates also
v
7 Select/play the
downloaded app
!
MOBILE ONLY APP
Have a valid account
with the Mobile OS
vendor's app store
v
1
2
Find the app in
their app store3
Download/install the
app through their
app store
4
v
Often requires
system or app
updates also
5
Download/install
the app through
their app store
6
Select/play the
downloaded app
v
7 !
View promo for app on
their website
AWE APP
1 Click on a web link to
open it in your
browser on any
device
HMD ONLY APP

39. How do you create VR & AR?
MOBILE ONLY APPs AWE APPSHMD ONLY APPS
Hire or be a developer
v
1
2
Download & get to know (&
deal with the various
issues/quirks) the developer
SDK for the target platform
3 Develop your app/s
4
v
Test it on the target HMD
vendor platform/s
5
Convert media and
3D assets to the
app/s format
6
Sign & submit your
app/s to the app store/s
& wait for approval/s
v
7
Publish & distribute
links to one or more
app stores
!
Download & get to know (&
deal with the various
issues/quirks) the developer
SDK for the target mobile OS
v
1
2
Develop your app/s
3
Test it on the target HMD
vendor platform/s 4
v
Convert media and
3D assets to the
app/s format
5
Sign & submit your app/s
to the app store/s & wait
for approval/s
6
Publish & distribute
links to one or more
app stores
v
7
Hire or be a developer
8
Create promotional web
content to drive people
to the app/s ! 8
Create promotional web
content to drive people
to the app/s
Get an account with
awe.media
*blatant plug J
v
1
2
Upload any types of
common media you
need (photos, videos,
3D & more)
3
Manipulate these objects
and add rich interactions
to create your VR & AR
experiences
4
Open your browser on
any type of device
v
5 Share a single link that
points directly to your web-
based VR/AR experience

40. CREATING MR
EXPERIENCES IN AWE
https://awe.media
360° LOCATION IMAGE

41. THE MR CREATION TOOLKIT
Smartphones, tablets, PCs
Head-mounted displays (cardboard & dedicated)
360 Cameras

42. Guiding principles
Planning – discovery, storyboarding, prototyping, user testing, etc.
You’re introducing a spatial environment that relates to the real world around people, or if not completely
then introduces real world elements and interactions.
Make interaction patterns consistent and strive to make it feel familiar and non-threatening (unless that is
your intent!)
Inclusiveness
If you’re developing for a single platform, don’t aim for fragmented silos. Think about how your audience on
other platforms or unsupported devices can also take part or be involved in the experience.
Smartphones & tablets will be the dominant form factor for at least the next 5+ years. HMDs are expensive.
Example from education
Are you solving a real problem, or just using the technology because you can?
You need to make the complex simple.

43. Constraints in MR environments
On-site experience
Have you actually been to the locations?
Physical obstacles
Factors impacting location accuracy
Line of sight
Interior of buildings
Gathering location data
Location source not allowed by the user
Have you considered what the experience will be like for people accessing off-site?
Friction when moving between modes (motion switcher demo)
LOCATION SPECIFIC

44. Constraints in MR environments
Image / object quality
Image print quality
Light source
Overlaid file sizes / object complexity
RECOGNITION & TRACKING SPECIFIC

45. Quick tips for 360°
Capturing
o Physically walk around the environment and plan shots
o What height are you shooting at e.g. person eye level, helicopter view?
o Take in the whole scene – foreground, mid-ground, background.
o If outdoor, where is the the sun & what other environmental factors may impact the scene. Beware
shadows!
o What obstacles are there that may obscure the meaning of the scene?
o If indoors, how large is the space, will the detail be lost?
o If using a tripod, make sure it’s stable and level.
Camera quality
o Every 360° camera / camera + app controller has it’s quirks
o Unless you need Hollywood quality, go for a <$800 camera e.g.:
Theta is great for prototyping / blocking out quick shots and sequences
Kodak PixPro works well with drones but is fragile
Nikon KeyMission best resolution, portable and robust

46. Quick tips
Default view direction
o Which part of your scene do you want to display in a user’s field of view first?
o Or, do you want a part of the background to always load to a specific compass direction?
Adding interactive objects to your view (360° or AR)
o What points of interest do you want to draw people’s attention to?
o How do you want to do this?
o Block it out with simple placeholder objects and test it in the real world.
o Refine the look and feel once the mechanics are right.
Video or image background?
o Static images can be evocative if you stage it for the context. Great for teleporting.
o Short video and audio loops capture the atmosphere of a scene.
o How do you camouflage the tripod arms?
o Long-form video should be used sparingly, unless the narrative is compelling.