Talk on Rapid Prototyping for Augmented Reality, given by Mark Billinghurst on April 5th 2016. Given to students at Stanford University's Augmented Reality class

1. RAPID PROTOTYPING FOR
AUGMENTED REALITY
Mark Billinghurst
mark.billinghurst@unisa.edu.au
April 5th 2016

2. How do you Design and Test this?

3. Or this..

4. Google Glass

5. View Through Glass

6. Google Glass Prototyping
http://ed.ted.com/lessons/rapid-prototyping-google-glass-tom-chi

7. Rapid Prototyping Glass
• Q: How long would it take you to make the first
working version of the Glass headset display?
• A: One day..

8. Interacting with Glass

9. Prototyping Interaction
• Q: How long would it take you to prototype a
Minority Report interaction technique?
• A: 45 minutes..

10. Tom Chi’s Prototyping Rules
1. Find the quickest path to experience
2. Doing is the best kind of thinking
3. Use materials that move at the speed of
thought to maximize your rate of learning

12. The Road to Glass

13. Earlyprototyping

14. DESIGN PROCESSES

15. Interaction Design Process

16. DesignThinking -Three Phase Model
http://dschool.stanford.edu/redesigningtheater/the-design-thinking-process/

17. Elaboration and Reduction
• Elaborate - generate solutions.These are the opportunities
• Reduce - decide on the ones worth pursuing
• Repeat - elaborate and reduce again on those solutions
Source: Laseau,P. (1980) Graphic Thinking for Architects & Designers. John Wiley and Sons

18. Elaboration vs.Reduction
• Elaboration: Sketching/Storytelling
• Creation, exploration
• Quick, conceptual
• Reduction: Prototyping/Testing
• Functional
• Idea presentation
• Decision making

19. RAPID PROTOTYPING

20. How can we quickly
prototype Augmented
Reality experiences
with little or no coding?

21. Why Prototype?
▪ Quick visual design
▪ Capture key interactions
▪ Focus on user experience
▪ Communicate design ideas
▪ “Learn by doing/experiencing”

22. Design/PrototypingTools

23. Typical Development Steps
▪ Sketching
▪ Storyboards
▪ UI Mockups
▪ Interaction Flows
▪ Video Prototypes
▪ Interactive Prototypes
▪ Final Native Application
Increased
Fidelity &
Interactivity

24. Sketching
Sketching is not about drawing
It is about design.
Sketching is a tool to help you:
• express
• develop, and
• communicate design ideas
Sketching is part of a process:
• idea generation,
• design elaboration
• design choices,
• engineering

25. Figure 41: A Sketch of a Dialogue with a Sketch
The “conversation” between the sketch (right bubble) and the mind (left
bubble). A sketch is created from current knowledge (top arrow). Reading,
or interpreting the resulting representation (bottom arrow), creates new
knowledge. The creation results from what Goldschmidt calls “seeing
that” reasoning, and the extraction of new knowledge results from what
she calls “seeing as.”
sketch
representation
mind
(new) knowledge
Create
(seeing that)
Read
(seeing as)
Sketching is about the
activity not the result
– Bill Buxton
25

26. AR Hockey Concept
https://stereoscopica.wordpress.com/2013/05/29/augmented-reality-hockey-concept/

27. Sketched Interfaces
▪ Sketch + Powerpoint/Photoshop/Illustrator

28. Sketched Concepts
http://www.objecttheory.com/

29. Example Storyboard

30. Example Storyboard p.2

31. Persona + Storyboard

32. Paper Prototyping (Low Fidelity)
Quick and simple means of sketching interfaces
Use office materials
Easier to criticize, quick to change
Creative process (develop in team)
Can also use for usability test (focus on interaction flow)
Used a lot to test out concepts before real design begins.

33. Paper Prototyping

34. The Basic Materials
• Post-its
• 5x8 in. index cards
• Scissors, X-acto knives
• Overhead transparencies
• Large, heavy, white paper (11 x 17)
• Tape, stick glue, correction tape
• Pens & markers (many colors & sizes)

35. Physical Prototype

36. • Early ideation
• Think through ideas
• Force you to visualize how things come together
• Communicate ideas to others to inspire new designs
• Active brainstorming
Why is Sketching/Paper Prototyping Useful?

37. Paper Prototypes Pros and Cons
• Good
• Quick and dirty
• No software needed
• Very inclusive
• Bad
• Not quick enough
• Too dirty (lack context and scale)
• Hard to share among team
• Still needs a separate documentation step

38. From Sketches toWorking Systems
Early design
Late design
Brainstorm different ideas and
representations
Choose a representation
Rough out interface style
Multitude of sketches
Sketch variations and details
Sketch or low fidelity prototypesTask centered walkthrough and redesign
Fine tune interface, screen design
Heuristic evaluation and redesign
Usability testing and redesign
Low to medium fidelity prototypes
Limited field testing
Alpha/Beta tests
High fidelity prototypes
Working systems

39. From Sketches to Prototypes
• Sketches: early ideation stages of design
• Prototypes: capturing /detailing the actual design

40. Prototyping
• Create physical form of ideas
• Allow people to experience and interact with them
• Why Prototyping?
• Empathy gaining- deepen understanding of design space
• Exploration – build to think
• Testing – test solutions with end users
• Inspiration – help others catch your vision

41. Sketch vs.Prototype
Sketch Prototype
Invite A)end
Suggest Describe
Explore Refine
Ques;on Answer
Propose Test
Provoke Resolve
Tenta;ve, non commi)al Specific Depic;on
The primary differences are in the intent

42. Sketching to Prototype

43. Wireframe
• It’s about
• Functional specs
• Navigational systems
• Functionality and layout
• Notes about the intended functionality
• How interface elements work together
• Lack of typographic style, colour or graphics
• Leaving room for the design to be created

44. Wireframes

46. Mockup
• It’s about
• Look and feel
• Build on the wireframe with
graphics and polish
• May adjust layout slightly but stays
within the general guide of the
wireframe

47. FunMe Mobile AR Wireframe/Mockup
http://www.yunnuocheng.com/funme/

48. GlassSim – http://glasssim.com/
▪ Simulate the view through Google Glass
▪ Multiple card templates

49. GlassSim Card Builder
▪ Use HTML for card details
▪ Multiple templates
▪ Change background
▪ Own image
▪ Camera view

50. GlassSim Samples

51. Application Storyboard
▪ http://dsky9.com/glassfaq/google-glass-
storyboard-template-download/

52. Application Flow

53. Acting/Role Playing

54. Wireframe/Mockup Pros and Cons
▪ Positives
▪ Good for documenting screens
▪ Can show application flow
▪ Negatives
▪ No interactivity/transitions
▪ Can’t be used for testing
▪ Can’t deploy on device
▪ Can be time consuming to create

55. Transitions

57. ▪ Series of still photos in a movie format.
▪ Demonstrates the experience of the product
▪ Discover where concept needs fleshing out.
▪ Communicate experience and interface
▪ You can use whatever tools, from Flash to iMovie.
Video Sketching

58. UI Concept Movies

59. Video Sketch Pros and Cons
• Good
• Excellent communication tool
• Shows interactions
• High fidelity concept
• Bad
• No interaction
• Difficult to use for user testing
• May not be able to implement vision
• Too good to be true

60. EXAMPLE:
SKETCH TO VIDEO

61. AR Hockey
• Concept – Air Hockey in the real world
• Mix materiality and real surfaces with digital forms
• React to real body movement and player interaction
https://stereoscopica.wordpress.com/2013/05/29/augmented-reality-hockey-concept/

62. AR Hockey Concept
https://stereoscopica.wordpress.com/2013/05/29/augmented-reality-hockey-concept/

63. Sketched Projected Content

64. Game Setup

65. Paper/Experience Prototypes
• Having people act out the game

66. Physical Mockup

67. Concept Video
https://vimeo.com/79285725

68. WORKING PROTOTYPES

69. Interactive Sketching
• Pop App
• Pop - https://popapp.in/
• Combining sketching and interactivity on mobiles
• Take pictures of sketches, link pictures together

70. Using Pop

71. Interactive Wireframing
▪ Developing interactive interfaces/wireframes
▪ Transitions, user feedback, interface design
▪ Web based tools
▪ UXpin - http://www.uxpin.com/
▪ proto.io - http://www.proto.io/
▪ Native tools
▪ Justinmind - http://www.justinmind.com/
▪ Axure - http://www.axure.com/

72. Proto.io - http://www.proto.io/
▪ Web based mobile prototyping tool
▪ Features
▪ Prototype for multiple devices
▪ Gesture input, touch events, animations
▪ Share with collaborators
▪ Test on device

73. Proto.io - Interface

74. Limitations
▪ No access to sensor data
▪ Camera, orientation sensor
▪ No multimedia playback
▪ Audio, video
▪ Simple transitions
▪ No conditional logic
▪ No networking

75. Wireframe vs.Prototype vs.Mockup
• Wireframe
• Low fidelity representation of design
• What UI elements, where UI are placed
• Mockup
• High fidelity static design
• Visual design draft
• Prototype
• Medium to high fidelity
• Supports user interaction

76. Interactive Coding - Processing
▪ Programming tool for Artists/Designers
▪ http://processing.org
▪ Easy to code, Free, Open source, Java based
▪ 2D, 3D, audio/video support
▪ Processing For Android
▪ http://wiki.processing.org/w/Android
▪ Strong Android support, builds .apk file

79. Processing + Augmented Reality
• Support for Augmented Reality
• Camera input
• GPS/Sensor input
• 3D Graphics
• Books
• Prototyping Augmented Reality (Mullins)
• Rapid Android Development (Sauter)
• Software
• Ketai Library - http://ketai.org/
• Android library support – all sensors on Android devices

80. http://toxiclibs.org/

81. Other Tools
▪ Wireframing
▪ pidoco
▪ FluidUI
▪ Rapid Development
▪ Phone Gap
▪ AppMachine
▪ Interactive
▪ App Inventor
▪ WearScript

82. EXAMPLE:
CONCEPT TO DEMO

83. NASA Hololens Concept Demo
• Vision: Work on Mars from your office
• Story and sketches based on vision
• Led to working Demo

84. Chesley Bonestell (1940s)

85. Hololens Story

86. HoloLens Concept Sketches

87. Final NASA HoloLens OnSight Demo
https://www.youtube.com/watch?v=o-GP3Kx6-CE

88. HARDWARE PROTOTYPING

89. Build Your Own Wearable
▪ MyVu display + phone + sensors

90. Beady-i
▪ http://www.instructables.com/id/DIY-
Google-Glasses-AKA-the-Beady-i/

91. Physical Input Devices
▪ Can we develop unobtrusive input
devices ?
▪ Reduce need for speech, touch pad input
▪ Socially more acceptable
▪ Examples
▪ Ring,
▪ pendant,
▪ bracelet,
▪ gloves, etc

92. Prototyping Platform
Arduino Kit Bluetooth Shield Google Glass

93. Example: Glove Input
▪ Buttons on fingertips
▪ Map touches to commands

94. Example: Ring Input
▪ Touch strip, button, accelerometer
▪ Tap, swipe, flick actions

95. How it works
Bracelet
Armband
Gloves
1,2,
3,4
Values/
output

96. CASE STUDY

97. ITERATIVE DESIGN PROCESS
Design
(redesign)
Prototype
Test Design with Users (Note
problems)
Evaluate
(Fix Issues)

98. MOBILE AUGMENTED REALITY FOR
SPATIAL NAVIGATION
Sharon Brosnan
Bachelor of Science in Digital Media Design
http://www.sharonbrosnan.com/portfolio/mobile-augmented-reality.php

99. BUNRATTY FOLK PARK
• Irish visitor attraction run by Shannon Heritage
• 19th century life is recreated
• Buildings from the mid-west have been relocated to the
26-land surrounding Bunratty Castle
• 30 buildings are set in a rural or village setting there.

101. AUGMENTED REALITY
In Bunratty Folk Park:
• Allows the visitor to point a camera at an exhibit, the
device recognises its by it’s location and layers digital
information on to the display
• 3-dimensional virtual objects can be positioned with
real ones on display
• Leads to dynamic combination of a live camera view and
information

102. NAVIGATIONALAID
Smartphone Platform
Most people carry mobile phones
Ideal Augmented Reality Technology
• Global tracking tools
• Wireless communication capabilities
• Location based computing
• Large display for interaction

103. DESIGNING
FOR THEVISITOR

104. HUMAN CENTRED DESIGN
Goal of the Navigational Aid
• Easy to use, clear and understandable
• Useful to visitors
• Creating interaction between the visitor and the aid
through the user interface
Engage the visitor
• It is necessary to understand the visitor to Bunratty
• Identify visitor motives and goals while going through
the Folk Park.

105. HUMAN CENTRED DESIGN
Understanding the technology and Related Works
! Literature
! Similar Technologies
! Electronic Tours in Museum Settings
! Interactions design in Outdoor Museums
Understanding the User over time
! Observations
! Interviews

106. HUMAN CENTRED DESIGN
Findings
• Most visitors do not use the map
• Most visitors have mobile phones
• Visitors want more information
• View the Folk Park at their own pace
• Information should be straight to the point
• Large social interaction within groups

107. FUNCTIONALITY
• View Options
• CameraView
• MapView
• ListView
• Sub-Options
• Places
• Events
• Restaurants
! AR Features
! Text Information
! 3D Objects
! 3D Tour Guide
! 3D Placement of Buildings

108. STORYBOARD

111. ITERATIVE DESIGN PROCESS

112. ITERATIVE DESIGN PROCESS
Prototyping and User Testing
• Low Fidelity Prototyping
• Sketches
• Paper Prototyping
• Post-It Prototyping
• PowerPoint Prototyping
• High Fidelity Prototyping
• Wikitude

113. INITIAL SKETCHES - Photoshop
Pros:
• Good for idea generation
• Cheap
• Concepts seem feasible
Cons:
• Not great feedback gained
• Photoshop not fast enough
for making changes

114. POST IT PROTOTYPING
First DraJ
Camera View with 3D
Second DraJ Third DraJ
• Selec;on highlighted in blue • Home bu)on added for easy
naviga;on to main menu

115. POWERPOINT PROTOTYPING
Benefits
• Used for User Testing
• Interactive
• Functionalities work
• Quick
• Easy arrangement of slides
User Testing
• Participants found
• 15 minute sessions screen captured
• ‘Talk Allowed’ technique used
• Notes taken
• Post-Interview

116. WIKITUDE
• Popular AR browser
• Mapping
• Point of Interest abilities
• Multiplatform
• Shows points of interest
• Markers can be selected in and
information pop-up appears

117. FINAL CONCEPT
DESIGN

118. FINAL DESIGN CONCEPT
Key Issues
• Fix issues found in previous sessions
• Design with guidelines in mind
• Appealing to the Mental Model
• Icon Design
• Aesthetic Design
• Colour/Font
• Buttons
• Look

119. VIDEO PROTOTYPE
" Flexible tool for capturing the use of
an interface
" Elaborate simula;on of how the
naviga;onal aid will work
" Does not need to be realis;c in
every detail
" Gives a good idea of how the
finished system will work

120. SUMMARY

121. Tom Chi’s Prototyping Rules
1. Find the quickest path to experience
2. Doing is the best kind of thinking
3. Use materials that move at the speed of
thought to maximize your rate of learning

122. Conclusions
• Prototyping is one of the most important skills you
can have as an ARinterface designer
• There are many tools available:
• Sketching, Paper Prototyping, Video, etc
• Always keep the user at the heart of the process
• User centered design
• User testing/feedback

123. Consider theWhole User

124. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au