Presentation on how to design great AR experiences. Given by Mark Billinghurst at the Huawei STW 2016 conference on May 17th, 2016 in Shenzen, China.

1. DESIGNING OUTSTANDING
AUGMENTED REALITY
EXPERIENCES
Mark Billinghurst
mark.billinghurst@unisa.edu.au
May 17th 2016
Shenzen, China

2. • adsf

3. • jg

4. The Digital Divide
• Screens are windows into digital spaces
• Separation between digital and physical world

5. Augmented Reality
Vuforia Smart Terrain

6. Augmented Reality
Technology that:
1. Combines Real andVirtual Images
• Both can be seen at the same time
2. Interactive in real-time
• The virtual content can be interacted with
3. Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

7. • Web based AR
• Flash, HTML 5
• Marketing, education
• Outdoor Mobile AR
• GPS, compass tracking
• Viewing Points of Interest in real world
• Handheld AR
• Vision based tracking
• Marketing, gaming
• Location Based Experiences
• HMD, fixed screens
• Museums, point of sale, advertising
Typical AR Experiences

8. AR Business Today
• Around $600 Million USD in 2014 (>$2B 2016)
• 70-80+% Games and Marketing

9. Market Projections
cf. 2014 computer game market = $84 Billion USD

10. DESIGNING AR
EXPERIENCES

11. What Makes a Good AR Experience?
• Compelling
• Engaging,‘Magic’ moment
• Intuitive, ease of use
• Uses existing skills
• Anchored in physical world
• Seamless combination of real and digital

12. Example: colAR (Quiver)
• Turn colouring book pages into AR scenes
• Markerless tracking, use your own colours..
• Try it yourself: http://www.colARapp.com/

13. • Interface Components
• Physical components
• Display elements
• Visual/audio
• Interaction metaphors
Physical
Elements
Display
ElementsInteraction
MetaphorInput Output
AR Interface Elements

14. AR Interface Design
• Combines physical + virtual object design
• Physical
• Tangible controllers and objects
• Virtual
• Virtual graphics and audio

15. Physical Design
• Industrial Design
• Type of Objects
• Purposely built – affordances
• “Found” – repurposed
• Existing – already at use in marketplace

16. How are these used?

17. ”… the perceived and actual properties of the
thing, primarily those fundamental properties
that determine just how the thing could
possibly be used. [...]
Affordances provide strong clues to the
operations of things.”
(Norman, The Psychology of EverydayThings 1988, p.9)
Affordance

18. Physical vs. Virtual Affordances
• Physical Affordances
• Look and feel of real objects
• Shape, texture, colour, weight, etc
• Industrial Design
• Virtual Affordance
• Look of virtual objects
• Copy real objects
• Interface Design

19. Design of AR Enhanced Objects
• Make affordances obvious
• Object affordances visible
• Give feedback
• Provide constraints
• Use natural mapping
• Use good cognitive model

20. Example: Haunted Book/AR Book
• Camera hidden in lamp object
• AR content seamlessly integrated into real book
• Natural page turning/manipulation interaction

21. Interface Design Path
1/ Demonstration:Working Prototype
2/ Copying:Adoption of Interaction Techniques
from other interface metaphors
3/ Creation: Development of new interface
metaphors appropriate to the medium
4/ Theory: Development of formal theoretical
models for predicting and modeling user actions
Desktop WIMP
Virtual Reality
Augmented Reality

22. Example: VR Interfaces
• Copying: Virtual Windows/keyboards
• Creation: World in Miniature

23. AR Interaction Metaphors
• AR Lens/Window
• simple (conceptually!), unobtrusive
• 3D User Interfaces (VR)
• expressive, creative, require attention
• Tangible Interfaces (TUI)
• Embedded into conventional environments
• Tangible AR
• Combines TUI input + AR display

24. AR Lens
• Information is registered to
real-world context
• Hand held AR displays
• Interaction
• 2D/3D virtual viewpoint control
• Limited input/interactivity
• Applications
• Context-aware information
Tourism, gaming

25. 3D AR Interfaces
• Virtual objects displayed in 3D
physical space and manipulated
• HMDs and 6DOF head-tracking
• 6DOF hand trackers for input
• Interaction
• Viewpoint control
• Traditional 3D interaction:
• manipulation, selection, etc.
• VR techniques

26. Tangible User Interfaces (Ishii 97)
• Create digital shadows
for physical objects
• Foreground
• graspable UI
• Background
• ambient interfaces

27. Tangible Interface: Augmented Groove
• Collaborative Instrument
• Physically Based Interaction
• Map actions to Midi output
• Translation, rotation
• Tilt, shake

28. Lessons from Tangible Interfaces
• Advantages
• Physical objects make us smart
• Objects aid collaboration
• Objects increase understanding
• Disadvantages
• Difficult to change object properties
• Limited display capabilities
• Separation between object and display

29. Back to the Real World
• AR overcomes limitation of TUIs
• enhance display possibilities
• merge task/display space
• provide public and private views
• TUI + AR = Tangible AR
• Apply TUI methods to AR interface design
• TUI for input, AR for output

30. Tangible AR Design Principles
• Tangible AR Interfaces use TUI principles
• Physical controllers for moving virtual content
• Support for spatial 3D interaction techniques
• Time and space multiplexed interaction
• Support for multi-handed interaction
• Match object affordances to task requirements
• Support parallel activity with multiple objects
• Allow collaboration between multiple users

31. Tangible AR: Tiles
• Tiles semantics
• data tiles
• operation tiles
• Operation on tiles
• proximity
• spatial arrangements
• space-multiplexed
Poupyrev, I., Tan, D., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2001). Tiles: A
mixed reality authoring interface. In INTERACT 2001 Conference on Human Computer
Interaction (pp. 334-341).

32. Example:LevelHead

33. Case Study:LevelHead
• Physical Components
• Real blocks
• Display Elements
• Virtual person and rooms
• Interaction Metaphor
• Blocks are rooms

35. Design for Technology Limitations
• Understand the platforms used and design for limitations
• Hardware, software platforms
• Eg Handheld AR game with visual tracking
• Use large screen icons
• Consider screen reflectivity
• Support one-hand interaction
• Consider the natural viewing angle
• Do not tire users out physically
• Do not encourage fast actions
• Keep at least one tracking surface in view
Art of Defense Game

36. Seamless Design
• Design to reduce seams in the user experience
• Eg: AR tracking failure, change in interaction mode
• Paparazzi Game
• Change between AR tracking to accelerometer input
Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality
games, Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--
Arts, Media, and Humanities, p.19-28, October 26-29, 2011

37. Consider the Whole User

38. Consider Your User
• Mobile Phone AR User
• Probably Mobile
• One hand interaction
• Short application use
• Need to be able to multitask
• Use in outdoor or indoor environment
• Want to enhance interaction with real world

39. Social Acceptance
• People don’t want to look silly
• Only 12% of 4,600 adults would be willing to wear AR glasses
• 20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues
• Needs further study (ethnographic, field tests, longitudinal)

40. TAT Augmented ID

43. Google Glass Anyone?

44. Designing AR Experiences
• Create a compelling experience
• Intuitive and ease of use
• Anchor in the real world
• Design affordances for physical + virtual elements
• Create the appropriate Interaction Metaphor
• Design for technology limitations
• Consider the whole user (Social, cultural, ..)

45. RESEARCH DIRECTIONS

46. Looking to the Future
What’s Next?

47. Research Enables New Experiences
• Gesture/multimodal Input
• Natural user interaction
• Collaborative experiences
• Capturing rich communication cues
• Experience capture
• Sharing surroundings
• Empathic Computing
• Creating understanding
• Etc..

48. Gesture Based Interaction
• Use free hand gestures to interact
• Depth camera, scene capture
• Multimodal input
• Combining speech and gesture
HIT Lab NZ Microsoft Hololens
Meta SpaceGlasses

49. Social Panoramas
• Google Glass
• Capture live image panorama (compass + camera)
• Remote device (tablet)
• Immersive viewing, live annotation
Reichherzer, C., Nassani, A., & Billinghurst, M. (2014). Social panoramas using wearable
computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on
(pp. 303-304). IEEE.

50. Empathy Glasses
• Combine together eye-tracking, display, face expression
• Implicit cues – eye gaze, face expression
++
Pupil Labs Epson BT-200 AffectiveWear

51. Remote Collboration
• Eye gaze pointer and remote pointing
• Face expression display
• Implicit cues for remote collaboration

52. Holoportation
• Augmented Reality + 3D capture + high bandwidth
• http://research.microsoft.com/en-us/projects/holoportation/

53. CONCLUSION

54. Conclusions
• AR enables connection back to real world
• Great AR experiences need good design
• Physical + virtual components, interaction metaphor
• Design for technology limitations
• Need to consider whole user
• Many directions for future research
• Natural interaction, collaboration, experience capture

55. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au