FIFTY SHADES OF AUGMENTED REALITY:
CREATING CONNECTION USING AR
Mark Billinghurst
mark.billinghurst@unisa.edu.au
March 24th 2017
Laval Virtual

• jg

Ancient Communication Tools

Evolution of Communication Tools

TECHNOLOGY TRENDS

Modern Technology Trends
1. Improved Content Capture
• Move from sharing faces to sharing places
2. Increased Network Bandwidth
• Sharing natural communication cues
3. Implicit Understanding
• Recognizing behaviour and emotion
These three trends combined with AR/MR technology
can create new types of collaborative experiences

Content Capture
Realism
Time
Photo
Film
Live
Video
Panorama
360 Video
3D Space
1850 1900 1940 1990 2000 2010

Capturing Reality
• Panoramas (1790s)
• Immersive paintings
• Photography (1820-30s)
• Oldest surviving photo (Niépce, 1826)
• Stereo imagery (1830s)
• Wheatstone (1832)
• Brewster (1851)
• Movies (1870s)
• Muybridge (1878)
• Roundhay Garden Scene (1888)

Sharing Faces to Sharing Spaces
• Moving from desktop to
handheld cameras

Capturing 360 images
Kodak 360 Fly 360 Gear 360 Theta S Nikon
LG 360 Pointgrey Ladybug Panono 360 Bublcam

360 Stereo Capture
Samsung Project Beyond
Jaunt VR camera
https://www.youtube.com/watch?v=FnBIvq8jzds

3D Space Capture
Google Project Tango
Occipital Structure Sensor
https://www.youtube.com/watch?v=XA7FMoNAK9M

Content Capture
Realism
Time
Photo
Film
Live
Video
Panorama
360 Video
3D Space
1850 1900 1940 1990 2000 2010
2D Static
Immersive
Live
Experience

Networking Speeds
Log (b/s)
Time
100 b/s
10 Kb/s
1 Mb/s
1980 1985 1990 1995 2000 2010
100 Mb/s
2005

Networking Technology
• Telephone (1876)
• Bell Picture Phone (1955)
• B+W live video calls
• Mobile Phones (1985)
• Desktop Video Conferencing (1990’s)
• Netmeeting, Skype
https://www.youtube.com/watch?v=OBkcYyenNBM

Mobile Conferencing (2010-)
• Video conferencing
• Skype, facebook
• Live broadcasting
• Meerkat, Periscope
• Support for 360 video

Network Innovation
Universal Connectivity

Networking Speeds
Log (b/s)
Time
100 b/s
10 Kb/s
1 Mb/s
1980 1985 1990 1995 2000 2010
100 Mb/s
2005
Text
Audio
Natural
Video

Interaction Technology
Natural
Time
Punch
Card
Keyboard
Mouse
Speech
Gesture
Emotion
1950 1960 1980 1990 2000 2010
Thought

Natural Gesture Input
• Freehand input
• Depth sensors, rich two handed gestures
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S.
(2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).
https://www.youtube.com/watch?v=A-xXrMpOHyc

Physiological Sensing
Emotiv Empatica

Interaction Technology
Natural
Time
Punch
Card
Keyboard
Mouse
Speech
Gesture
Emotion
1950 1960 1980 1990 2000 2010
Thought
Implicit
Explicit

MIXED REALITY
CONFERENCING

Milgram’s Reality-Virtuality Continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

Mixed Reality Conferencing in SciFi

AR Video Conferencing (2001)
• Bringing conferencing into real world
• Using AR video textures of remote people
• Attaching AR video to real objects
Billinghurst, M., & Kato, H. (2002). Collaborative augmented reality. Communications
of the ACM, 45(7), 64-70.

Demo: AR Video Conferencing

MR Point Cloud Sharing (2016)
• Use video-see through HMD
• Occulus Rift
• Depth camera on HMD
• Softkinetic
• Create point cloud
• Remote hand gestures
• User’s space

Demo: Point Cloud Capture

Holoportation (2016)
• Augmented Reality + 3D capture + high bandwidth
• http://research.microsoft.com/en-us/projects/holoportation/

Holoportation Video
https://www.youtube.com/watch?v=7d59O6cfaM0

Social VR
• Facebook Social Virtual Reality, AltspaceVR
• Bringing Avatars into VR space
• Natural social interaction

Demo: Facebook Social VR (2016)
https://www.youtube.com/watch?v=yxHwWHHg4Vs

AR/VR Collaboration (2017)
• Make 3D copy of real space
• AR user in real world, VR user in 3D copy of real space
• Hololens for AR user, HTC Vive for VR user
• Share virtual body cues (head, hands, gaze information)
Real World Virtual World

Virtual Cues
• AR/VR displays
• Gesture input
• Leap Motion
• Room scale tracking

Demo: AR/VR Conferencing

EMPATHIC COMPUTING

Natural
Collaboration
Implicit
Understanding
Experience
Capture

Natural
Collaboration
Implicit
Understanding
Experience
Capture
Empathic
Computing

Empathy
“Seeing with the Eyes of another,
Listening with the Ears of another,
and Feeling with the Heart of another..”
Alfred Adler

Empathic Computing
Systems that allow us to
share what we are seeing,
hearing and feeling with
others..

Example: Google Glass
• Camera + Processing + Display + Connectivity
• Ego-Vision Collaboration (But with FixedView)

Current Collaboration onWearables
• First person remote conferencing/hangouts
• Limitations
• Single POV, no spatial cues, no annotations, etc

Social Panoramas (ISMAR 2014)
• Capture and share social spaces in real time
• Supports independent views into Panorama
Reichherzer, C., Nassani, A., & Billinghurst, M. (2014, September). [Poster] Social panoramas using
wearable computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium
on (pp. 303-304). IEEE.

Implementation
• Google Glass
• Capture live image panorama (compass + camera)
• Remote device (tablet)
• Immersive viewing, live annotation

User Interfaces
Glass View
Tablet View

Social Panorama Demo
https://www.youtube.com/watch?v=vdC0-UV3hmY&t=1s

JackIn – Live Immersive Video Streaming
• Jun Rekimoto – University of Tokyo/Sony CSL

JackIn Hardware
• Wide angle cameras – 360 degree video capture
• Live video stitching

JackIn Demo
https://www.youtube.com/watch?v=Lkuz6LhlBf8

• zxcz

JospehTame – Tokyo Marathon
• Live streaming from Tokyo marathon
• http://josephta.me/en/tokyo-marathon/

Empathy Glasses (CHI 2016)
• Combine together eye-tracking, display, face expression
• Impicit cues – eye gaze, face expression
++
Pupil Labs Epson BT-200 AffectiveWear
Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of
the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.

AffectiveWear – Emotion Glasses
• Photo sensors to recognize expression
• User calibration
• Machine learning
• Recognizing 8 face expressions

Empathy Glasses in Use
• Eye gaze pointer and remote pointing
• Face expression display
• In future integrated eye-tracking/display

Empathy Glasses Demo
https://www.youtube.com/watch?v=CdgWVDbMwp4

Empathic VR Environments (2017)
• Player and Viewer
• Viewer slaved to player
• Share emotional signals
• Heart rate, GSR
• Remote affect measuring

Sharing VR Experiences
• HTC Vive HMD
• Empathic glove
• Empatica E4

VR Environments
• Butterfly World: calm scene, collect butterflies
• Zombie Attack: scary scene, fighting zombies

Demo Video
https://www.youtube.com/watch?v=SaiHxps-Ofw

AR and VR for Empathic Computing
• VR systems are ideal for trying experiences:
• Strong story telling medium
• Provide total immersion/3D experience
• Easy to change virtual body scale and representation
• AR systems are idea for live sharing:
• Allow overlay on real world view/can share viewpoints
• Support remote annotation/communication
• Enhance real world task

CONCLUSIONS

Conclusions
•Trend towards Empathic Computing
• Understanding, Experiencing, Sharing
•AR/VR Enables Empathic Experiences
• Changes perspective
• Sharing space/experience
• Supports annotation/communication
•Many directions for future research

www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au