1. APPLICATIONS IN AUGMENTED
AND VIRTUAL REALITY
Mark Billinghurst
mark.billinghurst@auckland.ac.nz
Bajaj Finance Limited, May 9th 2020

2. IBM 360 - $10,000 a month to rent in 1965

3. The Incredible Disappearing Computer
1960-70’s
Room
1970-80’s
Desk
1980-90’s
Lap
1990-2000’s
Hand

5. Digital
Physical

6. Graphical User Interfaces
• Separation between real and digital worlds
• WIMP (Windows, Icons, Menus, Pointer) metaphor

7. Ubiquitous Computing/IoT
• Embed computing and sensing in real world
• Smart objects, sensors, etc..

8. Virtual Reality
• Users immersed in Computer Generated environment
• HMD, gloves, 3D graphics, body tracking

9. https://www.youtube.com/watch?v=pAC5SeNH8jw

10. Virtual Reality
•Defining Characteristics
•Immersion
•Interaction
•Imagination
Grigore Burdea (1993). Virtual Reality Technology

11. Augmented Reality
• Virtual Images blended with the real world
• See-through HMD, handheld display, viewpoint tracking, etc..

12. https://www.youtube.com/watch?v=aUPMDwypBkA

13. Augmented Reality
•Defining Characteristics
•Combines Real andVirtual Images
•Interactive in real-time
•Registered in 3D
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

14. AR vsVR

15. Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments.
Making Interfaces Invisible

16. From Reality toVirtual Reality
Ubiquitous Computing Augmented Reality Virtual Reality
Real World Virtual World

17. Milgram’s Mixed Reality Continuum
Ubiquitous Computing Augmented Reality Virtual Reality
Real World Virtual World
Mixed Reality
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays

18. Mixed Reality Applications

19. Mixed Reality in 2020
• Large growing market
• $11B in 2017, $19B in 2020
• Many available devices
• HMD, phones, tablets, HUDs
• Robust developer tools
• Vuforia, WebGL, Unity, Wikitude, etc
• Large number of applications
• > 250K developers, > 100K mobile apps
• Strong research/business communities
• ISMAR, AWE, AugmentedReality.org, IEEE VR, etc

20. Market Growth

21. Market Breakdown

23. Using Mixed Reality in the Workplace
• Blending digital/physical worlds..
• Productivity
• Marketing
• Training
• Collaboration

24. Example: Productivity
• MR technology for hands busy tasks
• Warehouse picking (DHL, SAP)
• Smart glasses show pick location
• Simple user interface
• Hands-free interaction
• Improved performance
RW VR

25. https://www.youtube.com/watch?v=ZWsBHISOqjA

26. Improved Performance Time

27. https://www.youtube.com/watch?v=qTblKJjTadQ

28. Example: Marketing
• Add magic to company products
• Snapchat lenses
• Face tracking technology
• Simple interaction
• Animated content
• Significant ROI
• $750K for 224 million views
• c.f. $5 million for SuperBowl ad
RW VR

29. https://www.youtube.com/watch?v=wwmmgAw8ZAE

30. Example: Training
• AR/VR training for difficult/dangerous tasks
• Train with real objects, controls
• GlobalSim Crane Simulator
• Virtual Reality simulation
• Train with real controls
• Virtual body/hands
• Realistic physics
RW VR

31. https://www.youtube.com/watch?v=1Nej7iZjuRQ

32. Example: Collaboration
• Use MR technology for remote collaboration
• See through another’s eyes, virtual avatars
• Microsoft Remote Assist
• HoloLens see-through display
• Remote video sharing
• AR drawing/annotation
• Document sharing
• Group calling
RW VR

33. https://www.youtube.com/watch?v=V732PXZHLiU

34. FINTECH AR AND VR APP

35. Introducing AR/VR into the Workplace
•Identify key real-world tasks
• Involve spatial ability, physical interaction, etc.
•Justify Return on Investment
• Small technology investment leads to big return
•Focus on Human Needs
• User centered design
•Work with Experts
• Combine AR/VR experts with domain experts

36. Application Areas in Financial Services
• Customer Experience
• Customer Support
• Agent Training
• On-site Agent Support
• Data Visualization
• Meeting Support
• Etc..

37. Enhancing Customer Experience – Westpac NZ
https://www.youtube.com/watch?v=_dnKD2m_mPE

38. Customer Support – Virtual Agent – FidelityLabs
https://www.youtube.com/watch?v=FwNsDQLiazU

39. Customer Support – Remote Insurance Claims
https://www.youtube.com/watch?v=S59Xttnp2RY

40. Agent Training – Insurance Agents
https://www.youtube.com/watch?v=IlSt4p5XTmY

41. On Site Agent Support – Building Assessment
https://www.youtube.com/watch?v=a3_AtLFDisE

42. Data Visualization - Citibank
https://www.youtube.com/watch?v=0NogltmewmQ

43. Meeting Support - Spatial
https://www.youtube.com/watch?v=PG3tQYlZ6JQ

44. RESEARCH DIRECTIONS

45. Key Enabling Technologies
• Combines Real and Virtual Images
• Display Technology
• Interactive in real-time
• Interaction Technologies
• Registered in 3D
• Tracking Technologies

46. • Past
• Bulky Head mounted displays
• Current
• Handheld, lightweight head mounted
• Future
• Wide FOV see through
• Retinal displays
• Contact lens
Display Technology

47. Wide FOV See-Through (3+ years)
•Waveguide techniques
• Thin, wider FOV
• Socially acceptable
•Pinlight Displays
• LCD panel + point light sources
• 110 degree FOV
Lumus DK40

48. https://www.youtube.com/watch?v=P407DFm0PFQ

49. Retinal Displays (5+ years)
• Photons scanned into eye
• Infinite depth of field
• Bright outdoor performance
• True 3D stereo with depth modulation
• Microvision (1993-)
• Head mounted monochrome
• MagicLeap (2013-)
• Projecting light field into eye

50. Contact Lens (10 – 15 + years)
• Contact Lens only
• Unobtrusive
• Significant technical challenges
• Babak Parviz (2008)
• Mojo Vision
• https://www.mojo.vision/
• 14K ppi display
• Over $100m investment
http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/

51. Interaction
• Past
• Limited interaction
• Viewpoint manipulation
• Present
• Screen based, simple gesture
• Tangible interaction
• Future
• Natural gesture, Multimodal
• Intelligent Interfaces, Sensor based

52. Natural Gesture (2+ years)
• Freehand gesture input
• Depth sensors for gesture capture
• Rich two handed gestures
• E.g. Microsoft Research Hand Tracker
• 3D hand tracking, 30 fps, single sensor
• Commercial Systems
• Hololens2, Occulus, Leap Motion, Intel, etc..

53. https://www.youtube.com/watch?v=LblxKvbfEoo

54. Multimodal Input (5-10+ years)
• Combine gesture and speech input
• Gesture good for qualitative input
• Speech good for quantitative input
• Support combined commands
• “Put that there” + pointing
• E.g. HIT Lab NZ multimodal input
• 3D hand tracking, speech, multimodal fusion
• Complete tasks faster with MMI, less errors
Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with
Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.

55. HIT Lab NZ Multimodal Input

56. Tracking
• Past
• Location-based, marker based
• magnetic/mechanical
• Present
• Image based, hybrid tracking
• Future
• Ubiquitous
• Model based
• Environmental

57. Environmental Tracking (1-3+ yrs)
• Environment capture
• Use depth sensors to capture scene & track from model
• InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/)
• Real time scene capture on mobiles, dense or sparse capture
• Dynamic memory swapping allows large environment capture

58. https://www.youtube.com/watch?v=47zTHHxJjQU

59. 6D.ai AR Cloud Tracking
https://www.youtube.com/watch?v=AwwU14gllS0

60. Wide Area Outdoor Tracking (5 + yrs)
• Combine panorama’s into point cloud model (offline)
• Initialize camera tracking from point cloud
• Update pose by aligning camera image to point cloud
• Accurate to 25 cm, 0.5 degree over very wide area
Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed
and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.

61. Wide Area Outdoor Tracking

62. Example – Holoscape Outdoor AR Game
https://www.youtube.com/watch?v=Ndjdp9I6tFs

63. Other Research Areas
• Social Acceptance
• Overcome social problems with AR
• Cloud Services
• Cloud based storage/processing
• AR Authoring Tools
• Easy content creation for non-experts
• Collaborative Experiences
• AR teleconferencing
• Etc..

64. Example:SocialAcceptance
• People don’t want to look silly
• Only 12% of 4,600 adults would be willing to wear AR glasses
• 20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues
• Needs further study (ethnographic, field tests, longitudinal)

65. EMPATHIC COMPUTING

66. Modern Technology Trends
1. Improved Content Capture
• Move from sharing faces to sharing places
2. Increased Network Bandwidth
• Sharing natural communication cues
3. Implicit Understanding
• Recognizing behaviour and emotion

67. Natural
Collaboration
Implicit
Understanding
Experience
Capture

68. Natural
Collaboration
Implicit
Understanding
Experience
Capture
Empathic
Computing

69. Empathic Computing
Can we develop systems
that allow us to share what
we are seeing, hearing and
feeling with others?

70. Empathy Glasses
• Combine together eye-tracking, display, face expression
• Implicit cues – eye gaze, face expression
++
Pupil Labs Epson BT-200 AffectiveWear

71. Remote Collaboration
• Eye gaze pointer and remote pointing
• Face expression display
• Implicit cues for remote collaboration

72. https://www.youtube.com/watch?v=CdgWVDbMwp4

73. Shared Sphere – 360 Video Sharing
Shared
Live 360 Video
Host User Guest User

74. https://www.youtube.com/watch?v=FFF6qP5Ap44

75. Shared Sphere System Features
•AR / VR view collaboration
• VR user feels that they are in AR users location
•View independence
• Allow one user to look around independently from another
•View awareness cues
• Show where the other user is looking
•Hand gesture communication cues
• Sharing hand gesture cues

76. 3D Live Scene Capture
• Use cluster of RGBD sensors
• Fuse together 3D point cloud

78. View Sharing Evolution
• Increased immersion
• Improved scene understanding
• Better collaboration
2D 360 3D

79. Technology Trends
• Advanced displays
• Wide FOV, high resolution
• Real time space capture
• 3D scanning, stitching, segmentation
• Natural gesture interaction
• Hand tracking, pose recognition
• Robust eye-tracking
• Gaze points, focus depth
• Emotion sensing/sharing
• Physiological sensing, emotion mapping

80. Emotion Sensing HMDs are Coming
•Looxid
• EEG/Emotion sensing
• Eye tracking
•Emteq
• EMG sensing
•Neurable
• EEG/BCI for VR

81. Technology Trends
•Advanced displays
•Real time space capture
•Natural gesture interaction
•Robust eye-tracking
•Emotion sensing/sharing
Empathic
Tele-Existence

82. Empathic Tele-Existence
• Move from Observer to Participant
• Explicit to Implicit communication
• Experiential collaboration – doing together

83. CONCLUSIONS

84. Conclusions
•AR/VR in the workspace
•Breaks barrier between real/digital worlds
•Proven technology (display, input, sensing)
•Ready for implementation
•Future Research
•Display, tracking, interaction, user experience
•Empathic Computing

85. www.empathiccomputing.org
@marknb00
mark.billinghurst@auckland.ac.nz