What’s new at Wikitude

Who is talking
2
Philipp Nagele
● CTO of Wikitude (Product &
Technology)
● MSc in Computer System
Engineering from Halmstad
University in Sweden
● Previous companies:
● 3united/VeriSign Inc.
● T-Mobile Austria

Agenda for today
● A look back
● A look ahead
● Q&A

v v v
Wikitude AR at a glance.
First AR app
in the world
in 2008
20,000 apps
powered by
Wikitude SDK
3D tracking
technology
v
REFERENCES

5
HQ office in Salzburg, Austria

Wikitude’s path to success
20142013 2015
Business Model: B2C
Focus: Wikitude AR App
2016 2017201220112009
Founder stage in
Salzburg garage:
First AR Browser
launched.
2010
VCs brought in.,
Management hired:
Martin Herdina,
Andy Gstoll
Established
Wikitude Inc.
in USA
Business Model: B2B
Focus: AR technology and tools
Achieved 25
million
downloads and
app installs
globally
Launch of
Wikitude SDK
for 2D image
recognition
Launch of
Wikitude Studio,
the AR CMS
Launch of
Wikitude Cloud
Recognition
service
Launch of 3D
tracking
SLAM

AR authoring and CMSCloud Recognition
Wikitude AR Products & Tools

12
in 150+
countries globally
Largest independent AR platform

Platform agnostic technology
13
AR/VR wearables TabletsSmartphones
multi platform technology core based on C++

Tailored SDKs for Smart Glasses
Partnership since 2014,
optimized SDK for BT-200, full
documentation
→BT-300 in pipeline
Partnership since 2016
optimized SDK for R-7
Partnership since 2014, SDK for
M100, full documentation
→M300 in pipeline

Native API JavaScript API
Easy access to AR mobile development
Xamarin
PhoneGap
Cordova
titaniumunity
Android
iOS

16

Wikitude SDK Architecture
Unity3D
Unity
Plugin
Plugins API
Computer
Vision
plugin
Camera
plugins
Your
App
Your
App
Cordova
Plugin
Titanium
Module
Xamarin
Component
Your
App
Cordova Titanium Xamarin
Your
App
Your
App
Java, Obj-C APIs
Your
App
3D Rendering
Engine
Augmentations LBS
Hardware Layer
Core
Components
ARMv7 NEON
optimizations
ARMv8
optimizations
Camera
GPGPU
optimizations
IMU
Intel x86 SSE
optimizations
Image
Recognition &
Tracking
Cloud
Recognition
SLAM Engine
Plugin
Manager
Metal
Rendering
OpenGL ES
Rendering
JavaScript API

Wikitude JS SDK Java API
Wikitude JavaScript SDK has a very slim Java API
• ArchitectView
• Lifecycle events (onCreate, onPause, onResume, … methods)
• Load Architect Worlds
Everything else is optional
• ArchitectUrlListener
• ArchitectWorldLoadedListener
• CaptureScreenCallback

Architect view composition
19
OpenGL ES 2/ES 3.x view
* Renders camera stream
* Renders augmentations
Web view
* Loads main .html file
* Transparent background

JavaScript Augmented Reality
Experience
20
Architect Worlds consist of:
• .html file(s)
• .js file(s)
• .css file(s)
• Defines your augmented reality experience
• Use the Wikitude JS API provided within the Architect view context
<script src=“https://wikitude.com/libs/architect.js"></script>
• Loaded by the Wikitude SDKs Architect view

Architect world
21
Javascript execution triggers actions in the underlying C++ layer

Types of augmentations
22
Augmentations are represented by AR.Drawable subclasses
• Images - AR.ImageResource / AR.ImageDrawable
• Videos (also with alpha channel) - AR.VideoDrawable
• 3D models - AR.Model
• Web views - AR.HTMLDrawable
• Labels - AR.Label

Wikitude JavaScript API
23
Model
• AR.GeoLocation
• AR.RelativeLocation
• AR.ImageTracker
• AR.InstantTracker
• AR.ImageResource
View
• AR.ImageDrawable
• AR.VideoDrawable
• AR.Model
• AR.Label
Controller
• AR.GeoObject
• AR.ImageTrackable
• AR.InstantTrackable
• AR.PropertyAnimation
• AR.AnimationGroup

What happened after AWE 2016
● 6 releases of the Wikitude SDK
● 4 feature releases
● 2 maintenance stability releases
● Entire new Wikitude Studio Editor
● Updated Unity plugin and Editor
support
● Re-vamped support platform
● New ticketing system and forum
● New documentation

Computer Vision changes
● SLAM 3D engine
● New tracker available called InstantTracker
● Improved performance for Extended Tracking
● Revamped 2D engine
● Boosted recognition accuracy
● Improved tracking stability and accuracy
● Reduced tracking jitter

Computer Vision changes – Image Improvements

Computer Vision changes – New SLAM engine

API changes and Improvements
● Updates to Plugins API
● Re-designed API to CV Tracker
● New CV Tracker
● Rendering and Camera Improvements
● Multi-touch gesture Support in JavaScript API
● 1:1 calibration for Smartglasses
● Support for ODG R-7 glasses

Updates for the Plugins API
● New type of plugin: Input Plugins
● YV12 color space for input plugins
● New type for rendering: Positionables
● Own implemented trackers can re-use rendering
of augmentations in JavaScript API
Custom
Frame
Provider
Image
Frame
Standard
Frame
Provider
Wikitude
CV EngineRendering
Custom CV
Plugin
Plugins API

Tracker changes
● Old naming scheme for trackers didn’t
scale
● More descriptive names
● ImageTracker now can have options
● Local/Offline
● Cloud
● New Trackers added (and to come)

Rendering and Camera
Improvements
● Update to work with Android Camera2 API
● Enables capturing with higher fps
● Fine-grained control over focus behaviour
● New Graphics APIs supported
● OpenGL ES 3.x
● Metal
● HD Camera Frame rendering
● Choose between SD to FullHD
● Drawables in billboard mode

Multi-Touch Gestures
● Needed for any interaction with augmentations
on screen
● Only available in the JavaScript API
● Main gestures covered
● Rotate
● Pinch-to-zoom
● Move/Drag/Pan
● React on callbacks for gestures

Additions for Smart Glasses
● Official support for ODG R-7 smart glass
● Personal calibration and calibration profile for
all supported smart glasses with see-through
glasses
● Profile can be stored for each user
● Calibration process can be whitelabeled

New Studio Editor
● Entirely reworked from former Wikitude Studio
● Based on MEAN stack together with WebGL
rendering
● Fully web-based
● Integrated with Target Manager solution
● Experiences can be exported and are
converted to match JavaScript API
● Keyboard controls

What happens after AWE 2017
● Next release of Wikitude SDK is around the corner
● Object Recognition based on SLAM engine with pre-
generated maps
● Update for the 2D Image Computer Vision engine
● Enhancements for SLAM 3D engine

Object Recognition & Tracking
● Small-Scaled objects can be recognized and tracked like
any other Image Tracker
● Maps are pre-generated based on video provided by the
developer
● Define occlusion models (not only for Object
Recognition)
● New tracker for all APIs

Object Recognition & Tracking

Update for 2D Image CV engine
● Support for multiple image tracker at same time
● Unlimited – processing power is the limit
● Can detect duplicate targets (more of the same)
● API gives you distance and orientation between
targets
● Faster and more accurate initial recognition
● Image Recognition Extended Range (IR ER)
● Boost recognition distance by factor 3 – US letter
sized target can be recognized from 8 ft

Update for 2D Image CV engine

Enhancements for SLAM 3D Engine
● Better performance for 32-bit devices
● Higher accuracy during mapping and tracking
● Depth API to query depth information in SLAM for any
screen-coordinate
● Additional initialization plane for Instant Tracking
(vertical, horizontal)

Thx and Visit us at
Booth 634 for live
demos

Geo AR
42
How to define e.g. a point
of interest (POI)
• Define where the
augmentation is placed by
creating AR.GeoLocations
(latitude/longitude coordinates)
• Define the augmentation by
creating AR.Drawables
• Combine location and
drawables using
AR.GeoObjects
= The same geo object is displayed at different locations
POI C
POI A
POI A
Poi B

Pokémon Go like Geo AR
43
How to build a Pokémon Go like AR experience:

Pokémon Go like Geo AR
44
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
//a relative location being 12 meters south, 20 meters west and 13 meters lower than current position of the user
var relativeLocation = new AR.RelativeLocation(null, -12, -20, -13);

Pokémon Go like Geo AR
45
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
//latitude, longitude, altitude
var location = new AR.GeoLocation(47.77317, 13.069929, 320.);

Pokémon Go like Geo AR
46
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon
//create a new Model
var model = new AR.Model(“models/pokemon.wt3”);

Pokémon Go like Geo AR
47
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon
• Can be animated through .fbx defined animations
//We want to start an animation inside a model
var model = new AR.Model(“models/pokemon.wt3");
var modelAnim = new AR.ModelAnimation(model, "hit"); //start hit-animation defined inside model

Pokémon Go like Geo AR
48
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• …
• AR.Model object to render the 3D model of a Pokémon
• …
• AR.GeoObject to actually bring the model onto the screen
// a GeoObject with drawables set on creation time
var geoObject = new AR.GeoObject(location, {
drawables : {
cam: [model] //the drawable representing the GeoObject in the camera view
}
});

Pokémon Go like Geo AR
49
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
geoObject.onEnterFieldOfVision = function() {
/* your custom behaviour e.g. start a certain model animation */
};
geoObject.onExitFieldOfVision = function() {
/* your custom behaviour */
};

Pokémon Go like Geo AR
50
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
var imageResource = new AR.ImageResource(“assets/pokemonIndicator.png”);
var pokemonIndicator = new AR.ImageDrawable(imageResource, 0.2);
// a GeoObject with drawables set on creation time
var geoObject = new AR.GeoObject(location, {
drawables : {
cam: [model], //the drawable representing the GeoObject in the camera view
indicator: pokemonIndicator
}
});

Pokémon Go like Geo AR
51

GeoObjects & HTML content
Menu bar
jQuery support
Geo-located POIs
Multiple Drawables
- 3 images
- 3 text labels
Web View
Augmented Reality View

Pokémon Go like Geo AR
53
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
• Use the AR.Radar component to permanently show the position of
a Pokémon in the AR scene

Pokémon Go like Geo AR
54

Pokémon Go like Geo AR
55
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
• Use the AR.Radar to permanently show the position of a Pokémon in
the AR scene
• Capture a screenshot after you caught a Pokémon using the
corresponding ArchitectView Java API.

Image Recognition
Augment target images using
JavaScript API capabilities
(Drawables, Sounds,
Animations,Videos, 3D Models)
var kitchenTracker = new AR.Tracker("Kitchen.wtc");
var overlay = new AR.VideoDrawable("howdens.mp4", 0.65);
var trackable2DObject = new AR.Trackable2DObject(
kitchenTracker: "Kitchen",
{ drawables:
{ cam: overlay
}});

Mock-ups only
… understand what you are looking at
Mock-ups only

… make the World your playground

….augment books
Mock-ups only

… augment print, ads, catalogues, packaging

… virtually mount products

Maintenance