1.
Designing Mobile AR Applications
Mark Billinghurst
mark.billinghurst@hitlabnz.org
The HIT Lab NZ, University of Canterbury
December 2nd 2013

2.
Introduction

3.
1983 – Star Wars

4.
1999 - HIT Lab US

5.
1998: SGI O2
CPU: 300 Mhz
HDD; 9GB
RAM: 512 mb
Camera: VGA 30fps
Graphics: 500K poly/sec
2008: Nokia N95
CPU: 332 Mhz
HDD; 8GB
RAM: 128 mb
Camera: VGA 30 fps
Graphics: 2m poly/sec

6.
Mobile Phone AR
 Mobile Phones
 camera
 processor
 display
 AR on Mobile Phones
 Simple graphics
 Optimized computer vision
 Collaborative Interaction

7.
2005: Collaborative AR
 AR Tennis




Shared AR content
Two user game
Audio + haptic feedback
Bluetooth networking

8.
Mobile AR History

9.
Evolution of Mobile AR
Camera phone
Wearable
Computers
Camera phone
- Thin client AR
Wearable AR
Handheld
AR Displays
Camera phone
- Self contained AR
PDAs
-Thin client AR
PDAs
-Self contained AR
1995
1997
2001
2003
2004

10.
Handheld Displays
Tethered Applications
 Fitzmaurice Chameleon (1994)
 Rekimoto’s Transvision (1995)
 Tethered LCD
 PC Processing and Tracking

11.
Handheld AR Display - Tethered
1995, 1996 Handheld AR
 ARPad, Cameleon
 Rekimoto’s NaviCam, Transvision
 Tethered LCD
 PC Processing and Tracking

12.
Mobile AR: Touring Machine (1997)
 University of Columbia
 Feiner, MacIntyre, Höllerer, Webster
 Combines





See through head mounted display
GPS tracking
Orientation sensor
Backpack PC (custom)
Tablet input

13.
MARS View
 Virtual tags overlaid on the real world
 “Information in place”

14.
Backpack/Wearable AR
1997 Backpack AR




Feiner’s Touring Machine
AR Quake (Thomas)
Tinmith (Piekarski)
MCAR (Reitmayr)
 Bulky, HMD based

15.
Mobile AR - Hardware
RTK correction Antenna
GPS
Antenna
HMD
Controller
Example self-built working
solution with PCI-based 3D graphics
PCI 3D Graphics Board
Tracker
Controller
PC104 Sound Card
DC to DC
Converter
Wearable
Computer
CPU
PC104 PCMCIA
Battery
GPS
RTK
correction
Radio
Hard Drive
Serial
Ports
Columbia Touring Machine

16.
HIT Lab NZ Wearable AR (2004)
 Highly accurate outdoor AR
tracking system
 GPS, Inertial, RTK system
 HMD
 First prototype
 Laptop based
 Video see-through HMD
 2-3 cm tracking accuracy

17.
Image Registration
AR Stakeout Application

18.
Wearable AR Video

19.
Sharp J-SH04
 1997 Philip Kahn invents camera phone
 1999 First commercial camera phone

20.
Millions of Camera Phones
1200
1000
800
DSC
600
Phone
400
200
0
2002 2003 2004 2005 2006 2007 2008 2009 2010

21.
Handheld AR – Thin Client
2001 BatPortal (AT&T Cambridge)
 PDA used as I/O device
 Wireless connection to workstation
 Room-scale ultrasonic tracking (Bat)
2001 AR-PDA (C Lab)
 PDA thin graphics client
 Remote image processing
 www.ar-pda.com

22.
Mobile Phone AR – Thin Client
2003 ARphone (Univ. of Sydney)
 Transfer images via Bluetooth (slow – 30 sec/image)
 Remote processing – AR Server



23.
Early Phone Computer Vision Apps
2003 – Mozzies Game - Best mobile game
Optical motion flow detecting phone orientation
Siemens SX1 – Symbian, 120Mhz, VGA Camera
2005 – Marble Revolution (Bit-Side GmbH)
Winner of Nokia's Series 60 Challenge 2005
2005 – SymBall (VTT)

24.
Handheld AR – Self Contained
2003 PDA-based AR
 ARToolKit port to PDA
 Studierstube ported to PDA
 AR Kanji Educational App.
 Mr Virtuoso AR character
 Wagner’s Invisible Train
- Collaborative AR

25.
Mobile Phone AR – Self Contained
2004 Mobile Phone AR
 Moehring, Bimber
 Henrysson (ARToolKit)
 Camera, processor, display together

26.
AR Advertising
 Txt message to download AR application (200K)
 See virtual content popping out of real paper advert
 Tested May 2007 by Saatchi and Saatchi

27.
2008 - Location Aware Phones
Motorola Droid
Nokia Navigator

28.
Real World Information Overlay
 Tag real world locations
 GPS + Compass input
 Overlay graphics data on live video
 Applications
 Travel guide, Advertising, etc
 Eg: Mobilizy Wikitude (www.mobilizy.com)
 Android/iOS based, Public API released
 Other companies
 Layar, AcrossAir, Tochnidot, RobotVision, etc

29.
Layar – www.layar.com

30.
2013 State of the Art
Handheld Hardware available
PDA, mobile phones, external cameras
Sensors: GPS, accelerometer, compass
Software Tools are Available
Tracking: ARToolKitPlus, stbTracker, Vuforia
Graphics: OpenGL ES
Authoring: Layar, Wikitude, Metaio Creator
What is needed:
High level authoring tools
Content development tools
Novel interaction techniques
User evaluation and usability

31.
Mobile AR Companies
 Mobile AR
 GPS + compass
 Many Companies







Layar
Wikitude
Acrossair
PressLite
Yelp
Robot vision
Etc..

32.
$2 million USD in 2010
$732 million USD in 2014

33.
Qualcomm





Acquired Imagination
October 2010 - Releases free Android AR SDK
Computer vision tracking - marker, markerless
Integrated with Unity 3D renderer
http://developer.qualcomm.com/ar

34.
Rock-em Sock-em
 Shared AR Demo
 Markerless tracking

35.
Mobile AR Browsing

36.
Nokia City Lens
 More recent AR Browser

37.
AR Browsers
 AR equivalent of web browser
 Request and serve up content
 Commercial outdoor AR applications
 Nokia, Junaio, Layar, Wikitude, etc
 All have their own language specifications
 Wikitude – ARML
 Junaio – XML, AREL

38.
Architecture

39.
Junaio - www.junaio.com

40.
Key Features
 Content provided in information channels
 Over 2,000 channels available
 Two types of AR channels
 GLUE channels – visual tracking
 Location based channels – GPS, compass tracking
 Simple to use interface with multiple views
 List, map, AR (live) view
 Point of Interest (POI) based
 POIs are geo-located content

41.
Try it Yourself
 Download Junaio (app store, android market)
 Search for Junaio
 Run Junaio
 To try Glue channels
 Download the Junaio Demo Book from
www.junaio.com
 Hit scan button, point at QR codes
 To try Location channels
 Hit search icon in Junaio
 Click popular tab, pick channel (eg “Flickr”)

42.
Junaio Interface (Location Based)
Search
Icon
AR Tag
Current
Channel
View
Mode
Scan
Mode
Radar
Display

43.
Selecting an AR Tag
 Selecting a POI shows more information

44.
Multiple Views
Map View
List View
AR (Live) View
 Select View Mode to see different views

45.
QR Code Launch
 Hit scan button on interface, point at QR
code

46.
Glue Tracking - Markerless
 Search for “instant tracker”

47.
How Junaio Works

48.
Back-end Servers

49.
Data Flow

50.
Creating Your First Glue Channel
(Using Junaio Creator)

51.
Key Steps
1. Download and install Junaio application
 From iOS and Android app stores
2. Create a developer account on Junaio.com
3. Download and install Metaio Creator
 From http://www.metaio.com/creator/
4. Build sample AR scene
5. Test scene on desktop
6. Publish mobile AR channel

52.
Junaio.com
Click here to download
mobile application
Click here to create
developer account

53.
 http://www.junaio.com/develop/

54.
 http://www.metaio.com/creator/

55.
Creator Features





Drag and drop AR scene creation
Multiple types of content (2D, 3D, text, video, etc)
Multiple tracking techniques (image, model, point cloud)
Upload to Metaio server space
Offline preview mode

56.
Creator Interface

57.
Creator Interface
Resources
UI Tools
AR view
Chanel Info
Tracking Images

58.
Demo
1.
2.
3.
4.
5.
Upload tracking image
Add 3D content
Manipulate content
Test scene
Publish scene

59.
Desktop Test
 Click “Quick Preview” button

60.
Creating the AR Channel
 Click “New Chanel”, Fill out Channel Details

61.
Testing the Channel
 Launch Junaio, scan QR code

62.
QR Code Launch
 Hit scan button on interface, point at QR
code

63.
Test Result

64.
Other Demos
 Adding other trackable images
 Adding other content (video, 3D models, etc)
 Adding buttons and interactions
 Using the User Interface designer

65.
More Trackables
 Add more images
 Set properties

66.
Other Types of Trackables
 Image, object, environment tracking
 Use Metaio Toolbox for data capture

67.
Good Tracking Patterns
 pattern that is highly structured
 lot of visual hints with different colors
 high contrasts and sharp edges
 pattern in a "common" format,
 Square or rectangle format in 3:2 or 4:3 or similar
 not too dark and no reflection points
 shortest side of the image 150 – 200 pixels

68.
Good Examples

69.
Bad Tracking Patterns
 Reference Image not flat and blurry
 Shadows create false contrasts
 Angled reference images create false
reference orientation
 Pattern too bright or dark
 Angled with surrounding information

70.
Bad Examples

71.
Adding More Content
 3D models
 Position
 Adjust properties
- Shadow, occlusion, etc
- Trigger animation
 Video content
 In-page video, full-screen video
 Websites, audio
 Page triggered

72.
Adding Buttons
 Drag button icon into scene
 Scale and position
 Right click to add behaviours
 Open website, play video, etc

73.
Using the User Interface Designer
 Drag and drop 2D UI elements
 Buttons, images, etc
 Runs in screen aligned mode

74.
Making the Chanel Public
Click to change
channel status
 Becomes available for anyone in the world
 Takes 1-2 days for review by Metaio
 Need a paid Creator License ($530 USD)

75.
More Information
 http://dev.metaio.com/creator/
 Online tutorials, getting started, helpdesk, etc

76.
Creating Your First POI Channel
(Creating by Hand)

77.
You will Need
 A Junaio developer account
 Create at dev.junaio.com
 A web server where content can be uploaded
 Eg free server from http://www.000webhost.com/
 GPS Location of POI
 POI content
 Text, 2D image, etc

78.
Free Webserver
 Other options also available

79.
Finding the POI location
 Use Google maps, right click the POI location
 Copy Lat/Long information

80.
Server Content
 Download “Hello World” template content
 http://www.junaio.com/develop/quickstart/
 Edit on local machine
 Edit index.php to add POI information
 Use own POI icon
 Upload to web server

81.
Edit index.php
POI icon
POI location
POI pop-up info
 Edit index.php to add your own POI information

82.
Create a New Channel
 Click ‘My channels’ then ‘Create a New Channel’

83.
Fill Out the New Channel Form

84.
Channel Creation
 Use any name and channel description
 Channel Type: Location Based Channel
 Callback URL is most important
 Path to the index.php file on your server
 http://www.junaiotest.comze.com/JunaioTest/1HelloWorld/?path=
 Note ?path= at end of URL, you may not need this
 Once channel is saved then it is added to My Channels list
 Next Validate the Channel

85.
Channel Validation
 Runs a number of tests to see if server path is
correct, if content is there, returned code correct

86.
Validation Tests

87.
Expand Results – shows XML feedback

88.
Loading the Channel
 If the Channel is validated you can run it
 Either login into Junaio and enter developer mode
 Channel appears under ‘My Favourites’
 Or use QR code from My Channel page

89.
AR Outcome

90.
XML Parameters
 Many XML Parameters can be set
 See http://www.junaio.com/develop/docs/arel-xml-schemaxml-parameters/

91.
Adding More POI – edit index.php
<results>
<object id="1”>
//Define POI One
<location>
<lat>-43.536743</lat>
<lon>172.587705</lon>
<alt>0</alt>
</location>
</object>
<object id="2">
//Define POI Two
<location>
<lat>-43.536743</lat>
<lon>172.587000</lon>
<alt>0</alt>
</location>
</object>
</results>";
POI One Location
POI Two Location

92.
Junaio Results

93.
Limitations of Plain XML
 No interactivity
 Only simple pop-ups
 No user interface Customizations
 Can only use Junaio GUI elements
 No local interactivity
 Always needs remote server connection

94.
Junaio AREL

95.
AREL
 Augmented Reality Environment Language
 Overcomes limitations of XML by itself
 Based on web technologies; XML, HTML5, JavaScript
 Core Components
1. AREL XML: Static file, specifies scene content
2. AREL JavaScript: Handles all interactions and animation.
Any user interaction send an event to AREL JS
3. AREL HTML5: GUI Elements. Buttons, icons, etc
 Advantages
 Scripting on device, more functionality, GUI customization

96.
Example 2: Customizing your POI
 Using AREL HTML5 to develop custom interface
 Download Tutorial 2
 http://www.junaio.com/develop/quickstart/customizingyour-pois-images-sounds-videos-and-more/
 In Example 1 edited index.php, now use search.php

97.
index.php
 Just loads search.php

98.
search.php
 Loads the AREL helper classes
 php code providing valid information to Junaio
 Contains Channel Definition
1. Start it
startArelXMLHelper::start(NULL, WWW_ROOT . "/arel/index.php");
2. Return objects
ArelXMLHelper::outputObject($oObject);
3. End it
ArelXMLHelper::end();

99.
Defining a POI
//1. Sound POI
$oObject = ArelXMLHelper::createLocationBasedPOI(
"1", //id
"Hello Sound POI", //title
array(48.12310, 11.218648, 0), //location
WWW_ROOT . "/resources/thumb_sound.png", //thumb
WWW_ROOT . "/resources/icon_sound.png", //icon
"This is our Sound POI", //description
array(array("Start Audio", "soundButton", "http://www.junaio.com/publisherDownload/
tutorial/test.mp3")) //buttons
);
//output the object
ArelXMLHelper::outputObject($oObject);

100.
Location Based POI

101.
AR Application Running

102.
Loading a 3D model

103.
Example 3: Loading a 3D Model
 Position a model relative to the user position
if(!empty($_GET['l']))
$position = explode(",", $_GET['l']);
//calculate the position of T-Rex based on the position of the request. An offset is added to
the latitude value.
$tRexLocation = $position;
$tRexLocation[0] += 0.00004;
Use createLocationBasedModel3D to load 3D
model

104.
createLocationBasedModel3D

105.
Loading T-Rex
//T-Rex as static obj
$oObject = ArelXMLHelper::createLocationBasedModel3D(
"trex", //ID
"The T-Rex", //name
"http://dev.junaio.com/publisherDownload/junaio_model_obj.zip", //model
NULL, //texture
$tRexLocation, //position
array(5,5,5), //scale
new ArelRotation(ArelRotation::ROTATION_EULERDEG, array(0,-90,0)) //
rotation
);

106.
Result

107.
3D Models
 Junaio supports two model formats:
 MD2: Animated models, simple textures
 OBJ: Static models, high quality textures
 Use OBJ for high quality static models, MD2 for animated
 Making Models
 Make Models using Blender or similar tools
 May need file conversion tools
 Limit size to 500 – 1000 polygons/model
 See http://www.junaio.com/develop/docs/3d-models/

108.
Adding Interactivity

109.
Basic Interactivity
 Add a button on screen to move virtual character
 Use the following
 HTML: button specification
 Javascript: Interaction
 PHP/XML: 3D model
 Junaio Tutorial 5
 http://www.junaio.com/develop/quickstart/advancedinteractions-and-location-based-model-3ds/

110.
Server File Structure
HTML – GUI
JavaScript - interactivity
Main Index
PHP - content

111.
search.php – specify Lego Man
if(!empty($_GET['l']))
$position = explode(",", $_GET['l']);
Use local position
…
//return the lego man
$oLegoMan = ArelXMLHelper::createLocationBasedModel3D(
"1", // id
Lego model and texture
"lego man", //title
WWW_ROOT . "/resources/walking_model3_7fps.md2", // mainresource
WWW_ROOT . "/resources/walking_model.png", // resource
$position, // location
array(0.2, 0.2, 0.2), // scale
new ArelRotation(ArelRotation::ROTATION_EULERRAD, array(1.57,0,1.57)) // rotation
);
…

112.
styles.css – HTML GUI
#buttons {
position: absolute;
bottom: 44px;
right: 44px;
}
Button location
.ipad div {
width: 104px;
height: 106px;
}
#buttons div {
background-image: url("../images/button.png");
background-repeat: no-repeat;
background-size: 100%;
float:
left;
}
Button style

113.
Logic_LBS5.js - JavaScript
 Create an event listener
 setEventListener();
 Add functionality to model object
 Load model from scene
 Adding model behaviours
 Add functionality to GUI objects
 Define the event listener
 Bind model behaviours to GUI objects

114.
Result

115.
Looking to the Future
What’s Next?

116.
MIT Wearable Computing (1996)

117.
Google Glass

118.
What's Inside Google Glass?

119.
 Hardware
 CPU TI OMAP 4430 – 1.2 Ghz
 16 GB SanDisk Flash,1 GB Ram
 570mAh Battery
 Input
 5 mp camera, 720p recording, microphone
 GPS, InvenSense MPU-9150 inertial sensor
 Output
 Bone conducting speaker
 640x360 micro-projector display

120.
View Through Google Glass
Always available peripheral information display
Combining computing, communications and content capture

121.
User Interface
 dfasdf

122.
User Experience
 Truly Wearable Computing
 Less than 46 ounces
 Hands-free Information Access
 Voice interaction, Ego-vision camera
 Intuitive User Interface
 Touch, Gesture, Speech, Head Motion
 Access to all Google Services
 Map, Search, Location, Messaging, Email, etc

123.
Junaio on Google Glass
 Junaio Mirage - Junaio Browser on Glass
 Adapt UI to Glass (see through, touch gesture)
 Backend server unchanged

124.
 asdfa

125.
Living Heads Up vs. Heads Down

126.
Competitors
 Vuzix M100
 $999, profession
 Recon Jet
 $600, more sensors, sports
 Opinvent
 500 Euro, multi-view mode
 Motorola Golden-i
 Rugged, remote assistance

127.
Projected Market
 > 10 million displays by 2016

128.
Resources

129.
Book Demos

130.
Resources
• Developer documentation
– http://www.junaio.com/develop/docs/
• Google Group
– https://groups.google.com/forum/?fromgroups#!forum/junaiodeveloper

131.
Conclusions

132.
Conclusion
 Junaio provides easy way to create mobile AR
 Cross platform, client/server based, customisable
 Metaio Creator allows non-programmers to
create mobile AR applications
 Drag and drop visual interface
 Junaio supports HTML, XML, Javascript
 AREL format
 Support for wearable devices/computers
 Google Glass, etc

133.
More Information
 Mark Billinghurst
 mark.billinghurst@hitlabnz.org
 HIT Lab NZ
 http://www.hitlabnz.org/