Comp4010 lecture6 Prototyping

PROTOTYPING AND DESIGN
COMP 4010 Lecture Six
Mark Billinghurst
August 31st 2021
mark.billinghurst@unisa.edu.au

Different Types of AR Interaction
1. Browsing Interfaces
• simple (conceptually!), unobtrusive
2. 3D AR Interfaces
• expressive, creative, require attention
3. Tangible Interfaces
• embedded into conventional environments
4. Tangible AR
• combines TUI input + AR display
5. Natural AR Interfaces
• interact with gesture, speech and gaze

1. AR Interfaces as Data Browsers
• 2D/3D virtual objects are
registered in 3D
• “VR in Real World”
• Interaction
• 2D/3D virtual viewpoint control
• Applications
• Visualization, training

2. 3D AR Interfaces
• Virtual objects displayed in 3D
physical space and manipulated
• HMDs and 6DOF head-tracking
• 6DOF hand trackers for input
• Interaction
• Viewpoint control
• Traditional 3D user interface interaction:
manipulation, selection, etc.
Kiyokawa, et al. 2000

3. Augmented Surfaces and Tangible Interfaces
• Basic principles
• Virtual images are projected
on a surface
• Physical objects are used as
controls for virtual objects
• Support for collaboration
Wellner, P. (1993). Interacting with paper on the
DigitalDesk. Communications of the ACM, 36(7), 87-96.

Tangible User Interfaces (Ishii 97)
• Create digital shadows
for physical objects
• Foreground
• graspable UI
• Background
• ambient interfaces

Tangible Interface: ARgroove
• Collaborative Instrument
• Exploring Physically Based
Interaction
• Map physical actions to
Midi output
• Translation, rotation
• Tilt, shake

4. Tangible AR: Back to the Real World
• AR overcomes display limitation of TUIs
• enhance display possibilities
• merge task/display space
• provide public and private views
• TUI + AR = Tangible AR
• Apply TUI methods to AR interface design
Billinghurst, M., Kato, H., & Poupyrev, I. (2008). Tangible augmented reality. ACM Siggraph Asia, 7(2), 1-10.

Tangible AR: Tiles (Space Multiplexed)
• Tiles semantics
• data tiles
• operation tiles
• Operation on tiles
• proximity
• spatial arrangements
• space-multiplexed
Poupyrev, I., Tan, D. S., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2001,
July). Tiles: A Mixed Reality Authoring Interface. In Interact (Vol. 1, pp. 334-341).

Tangible AR: Time-multiplexed Interaction
• Use of natural physical object manipulations to control
virtual objects
• VOMAR Demo
• Catalog book:
• Turn over the page
• Paddle operation:
• Push, shake, incline, hit, scoop
Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000, October). Virtual object manipulation on a table-top AR
environment. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000) (pp. 111-119). IEEE.

5. Natural AR Interfaces
• Goal:
• Interact with AR content the same
way we interact in the real world
• Using natural user input
• Body motion
• Gesture
• Gaze
• Speech
• Input recognition
• Nature gestures, gaze
• Multimodal input
FingARtips (2004)
Tinmith (2001)

External Fixed Cameras
• Overhead depth sensing camera
• Capture real time hand model
• Create point cloud model
• Overlay graphics on AR view
• Perform gesture interaction
Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in space: Gesture interaction with
augmented-reality interfaces. IEEE computer graphics and applications, 34(1), 77-80.

Head Mounted Cameras
• Attach cameras/depth sensor to HMD
• Connect to high end PC
• Computer vision capture/processing on PC
• Perform tracking/gesture recognition on PC
• Use custom tracking hardware
• Leap Motion (Structured IR)
• Intel RealSense (Stereo depth)
Project NorthStar (2018)
Meta2 (2016)

Self Contained Systems
• Sensors and processors on device
• Fully mobile
• Customized hardware/software
• Example: Hololens 2 (2019)
• 3D hand tracking
• 21 points/hand tracked
• Gesture driven interface
• Constrained set of gestures
• Multimodal input (gesture, gaze, speech)

Speech Input
• Reliable speech recognition
• Windows speech, Watson, etc.
• Indirect input with AR content
• No need for gesture
• Match with gaze/head pointing
• Look to select target
• Good for Quantitative input
• Numbers, text, etc.
• Keyword trigger
• “select”, ”hey cortana”, etc https://www.youtube.com/watch?v=eHMkOpNUtR8

Eye Tracking Interfaces
• Use IR light to find gaze direction
• IR sources + cameras in HMD
• Support implicit input
• Always look before interact
• Natural pointing input
• Multimodal Input
• Combine with gesture/speech
Camera
IR light
IR view
Processed image
Hololens 2

Evolution of AR Interfaces
Tangible AR
Tangible input
AR overlay
Direct interaction
Natural AR
Freehand gesture
Speech, gaze
Tangible UI
Augmented surfaces
Object interaction
Familiar controllers
Indirect interaction
3D AR
3D UI
Dedicated
controllers
Custom devices
Browsing
Simple input
Viewpoint control
Expressiveness, Intuitiveness

Typical Interaction Design Cycle
Develop alternative prototypes/concepts and compare them, And iterate, iterate, iterate....

Prototyping in Interaction Design
Key Prototyping
Steps

Tom Chi’s Prototyping Rules
1. Find the quickest path to experience
2. Doing is the best kind of thinking
3. Use materials that move at the speed of
thought to maximize your rate of learning

How can we quickly prototype
XR experiences with little or no
coding?

XR Prototyping Techniques
Lo-
Fi
Hi-
Fi
Easy
Hard
Digital
Authoring
Immersive
Authoring
Web-Based
Development*
Cross-Platform
Development*
Native
Development*
* requires scripting and 3D programming skills
Sketching
Paper Prototyping
Video Prototyping
Wireframing
Bodystorming
Wizard of Oz

XR Prototyping Tools
Low Fidelity (Concept, visual design)
• Sketching
• Photoshop
• PowerPoint
• Video
High Fidelity (Interaction, experience design)
• Interactive sketching
• Desktop & on-device authoring
• Immersive authoring & visual scripting
• XR development toolkits

Storyboarding - Describing the Experience
http://monamishra.com/projects/Argo

Wireframes
It’s about
- Functional specs
- Navigation and interaction
- Functionality and layout
- How interface elements work together
- Defining the interaction flow/experience
Leaving room for the design to be created

Mockup
It’s about
- Look and feel
- Building on wireframe
- High fidelity visuals
- Putting together final assets
- Getting feedback on design

Interface Sketching in VR
Using VR applications for rapid prototyping
- Intuitive sketching in immersive space
- Creating/testing at 1:1 scale
- Rapid UI design/layout
Examples
- Quill - https://quill.fb.com/
- Tilt Brush - https://www.tiltbrush.com/

Microsoft Maquette
•Prototype AR/VR interfaces from inside VR
•3D UI for spatial prototyping
•Bring content into Unity with plug-in
•Javascript support

Scene Assembly In AR
• Many tools for creating AR scenes
• Drag and drop your assets
• Develop on web, publish to mobile
• Examples
• Catchoom - CraftAR
• Blippar - Blipbuilder
• ARloopa - Arloopa studio
• Wikitude - Wikitude studio
• Zappar - ZapWorks Designer

Video Sketching
• Process
• Capture elements of real world
• Use series of still photos/sketches in a movie format.
• Act out using the product
• Benefits
• Demonstrates the product experience
• Discover where concept needs fleshing out.
• Communicate experience and interface
• You can use whatever tools you want, e.g. iMovie.

Tvori - Animating AR/VR Interfaces
• Animation based AR/VR prototyping tool
• https://tvori.co/
• Key features
• Model input, animation, etc
• Export 360 images and video
• Simulate AR views
• Multi-user support
• Present in VR
• Create VR executable

MOBILE AUGMENTED
REALITY FOR SPATIAL
NAVIGATION
Sharon Brosnan
0651869
Bachelor of Science in Digital Media Design

Project Goals
Make a mobile AR Navigational Aid
○ Easy to use, clear and understandable
○ Useful to visitors
○ Creating interaction between the visitor and the aid through the user interface
○ Engage the visitor
To ensure this…
○ It is necessary to understand the visitor of a navigational aid in Bunratty Folk Park
○ Identify visitor motives and goals while going through the Folk Park.

NeedsAnalysis
Understanding Technology and Related Work
• Literature
• Similar Technologies
• Electronic Tours in Museum Settings
• Interactions design in Outdoor Museums
Understanding the User over time
• Observations
• Interviews

Findings
• Most visitors do not use the map
• Most visitors have mobile phones
• Visitors want more information
• View the Folk Park at their own pace
• Information should be straight to the point
• Large social interaction within groups
Needs Analysis

Desired Functionality
• View Options
• CameraView
• MapView
• ListView
• Sub-Options
• Places
• Events
• Restaurants
• Augmented Reality Features
for navigation
• Text Information
• 3D Objects
• 3D Tour Guide
• 3D Placement of Buildings

ITERATIVE DESIGN PROCESS
Prototyping and UserTesting
• Low Fidelity Prototyping
• Sketches
• Paper Prototyping
• Post-It Prototyping
• PowerPoint Prototyping
• High Fidelity Prototyping
• Wikitude

INITIAL SKETCHES (Photoshop)
Pros:
• Good for idea generation
• Cheap
• Concepts seem feasible
Cons:
• Not great feedback gained
• Photoshop not fast enough for
making changes

POWERPOINT PROTOTYPING
Benefits
• Used for User Testing
• Interactive
• Functionalities work when following the
story of Scenario 1
• Quick
• Easy arrangement of slides
User Testing
• Participants found
• 15 minute sessions screen captured
• ‘Talk Allowed’ technique used
• Notes taken
• Post-Interview

WIKITUDE
• Popular augmented reality browser
for mobile devices
• Mapping
• Point of Interest abilities
• Multiplatform
• Shows the points of interest of
Bunratty Folk Park
• Markers can be selected in and an
information pop-up appears

VIDEO PROTOTYPE
¢ Flexible tool for capturing the use
of an interface
¢ Elaborate simulation of how the
navigational aid will work
¢ Does not need to be realistic in
every detail
¢ Gives a good idea of how the
finished system will work

DemoVideo
• https://www.youtube.com/watch?v=8BqbEXUyDfo

Interactive Sketching
•Pop App
● Pop - https://marvelapp.com/pop
● Combining sketching and interactivity on mobiles
● Take pictures of sketches, link pictures together

Pop Demo
https://www.youtube.com/watch?v=Zdjg9yhBqTU

Proto.io
• Web based prototyping tool
• Visual drag and drop interface
• Rich transitions
• Scroll, swipe, buttons, etc
• Deploy on device
• mobile, PC, browser
• Ideal for mobile interfaces
• iOS, Android template
• For low and high fidelity prototypes

Proto.io Demo
https://www.youtube.com/watch?v=omw1o0IoSNw

CityViewAR Application
Goal: Create a mobile app that allowed
people to view immersive panoramas on
site, showing the 2011 damage
Key technology: Mobile phone, GPS, 360
panorama, map
User Experience: People walking the city
streets will be able to be immersed in the
earthquake damage from 10 years ago

CityViewAR Prototyping Process
1. Sketch Interaction Flow (Paper)
2. Create Wireframe (Powerpoint)
3. Sketch Screens (Paper)
4. Test Transitions (Pop)
5. Collect/Create Assets (Photoshop, various)
6. Create High Fidelity Prototype (Proto.io)
7. Code (Unity, Mapbox)
8. Make video demo (Various)
9. Publish (Android, iOS)

Sketch Interaction Flow/Wireframe

Pop Demo
https://www.youtube.com/watch?v=2XTgvjHDKlo

Test Transitions
https://youtu.be/eGxgpzXUiMQ

Demo
https://www.youtube.com/watch?v=4pL0sEB1884

Digital Prototyping
Lo-
Fi
Hi-
Fi
Easy
Hard
Digital
Authoring
Immersive
Authoring
Web-Based
Development*
Cross-Platform
Development*
Native
Development*
* requires scripting and 3D programming skills

Digital Authoring Tools for AR
Vuforia Studio
Lens Studio
• Support visual authoring of marker-
based and/or marker-less AR apps
• Provide default markers and support
for custom markers
• Typically enable AR previews
through emulator but need to deploy
to AR device for testing

AR Visual Programming
• Rapid prototype on desktop
• Deliver on mobile
• Simple interactivity
• Examples
• Zapworks Studio
• https://zap.works/studio/
• Snap Lens Studio
• https://lensstudio.snapchat.com/
• Facebook Spark AR Studio
• https://sparkar.facebook.com/ar-studio/

Zappar
• Zapworks Studio
• Code-free interactivity
• Desktop authoring for mobile AR
• Integrated computer vision (ARkit, ARCore)
• Scripting, visual programming
• Multiple publishing options
• Zappar App, WebAR, App enabled
• Zapbox
• Inexpensive mobile AR HMD solution
• Two handed input
ZapBox

Zapworks Studio
https://www.youtube.com/watch?v=dK6BmOsRxbw

Snap LensStudio - https://lensstudio.snapchat.com/
Author and preview AR prototypes
● Tool behind Snapchat Lenses, but also a powerful AR prototyping tool
● Can do face (using front camera) and world lenses (rear camera)
● Simulated previews using webcam
Deploy and use advanced AR features
● Can deploy to phone running Snapchat app via Snapcode
● Has advanced AR tracking and segmentation capabilities

Lens Studio – 4.0
https://www.youtube.com/watch?v=efu5pvjYtsw

Lens Studio Features
● SnapML: train and use ML models
● Face landmarks: track 93 points
● Face expressions: detect winks etc.
● Eye tracking: get eye pos/rotation
● Behaviour: set triggers & actions
● UI: add UI widgets without scripting
● Visual scripting: no code via graphs
● Templates: lots of templates and tutorials

Example: Museum AR Experience
Original art piece by Penelope
Umbrico on display at UMMA
One of the lenses On-site observation & Snapchat statistics
https://medium.com/@amcard/we-used-snapchats-lens-studio-to-build-ar-prototypes-and-this-is-what-we-learned-54e7a98a3d4c

Immersive Authoring Tools for AR
• Enable visual authoring of 3D
content in AR
• Make it possible to edit while
previewing AR experience in the
environment
• Provide basic support for interactive
behaviors
• Sometimes support export to
WebXR
Apple Reality Composer
Adobe Aero

Creating On Device
•Adobe Aero
•Create AR on mobile devices
•Touch based interaction and authoring
•Only iOS support for now
•https://www.adobe.com/nz/products/aero.html

Aero Demo
https://www.youtube.com/watch?v=-liG0I0aDdY

Apple Reality Composer
• Rapidly create 3D scenes and AR experiences
• Creation on device (iPhone, iPad)
• Drag and drop interface
• Loading 2D/3D content
• Simple interactivity – trigger/action
• Anchor content in real world (AR view)
• Planes (vertical, horizontal), faces, images

Behaviour Example
https://www.youtube.com/watch?v=tRcAGdM7878

XR Tools Landscape
Digital & Immersive Authoring
Proto.io, Tour Creator, ...
Tilt Brush, Blocks, Quill, …
Web-Based Development
THREE.js, Babylon.js, …
A-Frame, AR.js, …
Cross-Platform Development
Unity + SDKs
Unreal + SDKs
Native Development
Cardboard/Oculus/Vive/... SDK
Vuforia/ARCore/ARKit/… SDK

XR Tools Landscape
Digital & Immersive Authoring
Good for storyboarding but limited
support for interactions
Web-Based Development
Good for basic XR apps but often
interactions feel unfinished
Native Development
Good for full-fledged XR apps but
limited to a particular platform
Cross-Platform Development
Good for full-fledged XR apps but
usually high learning curve

XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

WebXR: A-Frame
• Based on Three.js and WebGL
• New HTML tags for 3D scenes
• A-Frame Inspector (not editor)
• Asset management (img, video,
audio, & 3D models)
• ECS architecture with many open-
source components
• Cross-platform XR

Hello World in A-Frame
https://aframe.io/examples/showcase/helloworld/
HTML code

AR.js – WebXR Tracking
• Web based AR tracking library
• Marker tracking: ARToolkit markers
• Image tracking: Natural feature tracking
• Location tracking: GPS and compass
• Key Features
• Very Fast : It runs efficiently even on phones
• Web-based : It is a pure web solution, so no installation required.
• Full javascript based on three.js + A-Frame + jsartoolkit5
• Open Source : It is completely open source and free of charge!
• Standards : It works on any phone with webgl and webrtc
• See https://ar-js-org.github.io/AR.js-Docs/

Source Code
<!doctype html>
<html lang=”en”>
<head>
<script src=”https://aframe.io/releases/0.7.0/aframe.min.js” ></script>
<script src=”https://jeromeetienne.github.io/AR.js/aframe/build/aframe-ar.js” ></script>
</head>
<body style =’margin : 0px; overflow: hidden;’ >
<a-scene embedded arjs=’sourceType: webcam;’ >
<a-box position=”0 0.5 -0.5″ color=”blue” scale=”1 0.5 1″ rotation= “0 45 0”
material=’opacity: 0.9;’ ></a-box>
<a-marker-camera preset=’hiro’ ></a-marker-camera>
</a-scene>
</body>
</html>
• For more info see https://aframe.io/blog/arjs/

AR.js Examples
• https://stemkoski.github.io/AR-Examples/

Unity – unity.com
• Started out as game engine
• Has integrated support for many
types of XR apps
• Powerful scene editor
• Asset management & store
• Basically all XR device vendors
provide Unity SDKs

Vuforia
• Highly optimized computer vision tracking
• Multiple types of tracking
• Image tracking, object tracking, model tracking, area tracking, etc.
• Interaction features
• Virtual buttons, occlusion, visual effects,
• Multi-platform
• Mobile AR, AR headsets See https://www.vuforia.com/

AR Foundation
• A unified Framework for AR
• Multi-platform API
• Includes core features from ARKit, ARCore, Magic Leap, and HoloLens
• Set of behaviours and API with following features
• Tracking, light estimation, occlusion, meshing , video pass-through, etc.
• Integrates with Unity MARS
• See https://unity.com/unity/features/arfoundation

Unity XR Interation ToolKit (preview package)
• Easy way to add interactivity to AR/VR experience
• Object interactions
• UI interactions
• Locomotion
• Enabling common interactions without writing code
• AR gesture, object placement, annotations
• https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@1.0/

https://www.youtube.com/watch?v=hi-zVWJOfxY
Unity XR Interaction ToolKit AR Demo

Unity MARS
• Features
• Visually author AR apps (WYSIWYG)
• Test apps in Unity editor
• Develop apps that can interact with real world
• Intelligent real-world recognition
• Multi-platform development
• Based on ARFoundation
• ARKit, ARCore, Magic Leap and Hololens
• See unity.com/mars

https://www.youtube.com/watch?v=825WgLYUqvg

Mixed Reality ToolKit (MRTK)
• Open-Source Mixed Reality ToolKit
• Set of Unity modules/Unreal plugin
• Interaction Models
• Controllers, gesture, gaze, voice, etc.
• UX elements
• Foundational elements
• Material, text, light, etc.
• Controls and behaviours
• button, menu, slider, etc.
• Tutorials, documentation, guidelines
• See https://github.com/microsoft/MixedRealityToolkit-Unity

Unity vs. A-Frame
Unity is a game engine and XR
dev platform
●De facto standard for XR apps
●Increasingly built-in support
●Most “XR people” will ask you
about your Unity skills :-)
Support for all XR devices
●Basically all AR and VR device
vendors provide Unity SDKs
A-Frame is a declarative WebXR
framework
●Emerging XR app development
framework on top of THREE.js
●Good for novice XR designers with
web dev background
Support for most XR devices
●Full WebXR support in Firefox,
Chrome, & Oculus Browser

Conclusions
• Prototyping vital part of AR Interface Design
• Focus on user experience
• Rapidly design/test/iterate
• Wide variety of Prototyping Tools
• Low fidelity
• ”Design at the speed of thought”
• Sketching, wireframes, mock-ups, layout tools, video prototypes
• High fidelity
• “Transitions and Final form”
• Interactive prototypes, visual authoring, development tools

XR Prototyping Web Site
XR Prototyping resources (http://xr-prototyping.org/)

www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au

Comp4010 lecture6 Prototyping

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Comp4010 lecture6 Prototyping

Ähnlich wie Comp4010 lecture6 Prototyping (20)

Mehr von Mark Billinghurst

Mehr von Mark Billinghurst (11)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Comp4010 lecture6 Prototyping