A talk from the Develop Track at AWE USA 2017 - the largest conference for AR+VR in Santa Clara, California May 31- June 2, 2017.
Philipp Nagele (Wikitude): What's Next with Wikitude
An in-depth look into the recent developments at Wikitude and what the next version of the Wikitude SDK will offer augmented reality developers.
http://AugmentedWorldExpo.com
2. Who is talking
2
Philipp Nagele
● CTO of Wikitude (Product &
Technology)
● MSc in Computer System
Engineering from Halmstad
University in Sweden
● Previous companies:
● 3united/VeriSign Inc.
● T-Mobile Austria
10. Wikitude’s path to success
20142013 2015
Business Model: B2C
Focus: Wikitude AR App
2016 2017201220112009
Founder stage in
Salzburg garage:
First AR Browser
launched.
2010
VCs brought in.,
Management hired:
Martin Herdina,
Andy Gstoll
Established
Wikitude Inc.
in USA
Business Model: B2B
Focus: AR technology and tools
Achieved 25
million
downloads and
app installs
globally
Launch of
Wikitude SDK
for 2D image
recognition
Launch of
Wikitude Studio,
the AR CMS
Launch of
Wikitude Cloud
Recognition
service
Launch of 3D
tracking
SLAM
11. AR authoring and CMSCloud Recognition
Wikitude AR Products & Tools
14. Tailored SDKs for Smart Glasses
Partnership since 2014,
optimized SDK for BT-200, full
documentation
→BT-300 in pipeline
Partnership since 2016
optimized SDK for R-7
Partnership since 2014, SDK for
M100, full documentation
→M300 in pipeline
15. Native API JavaScript API
Easy access to AR mobile development
Xamarin
PhoneGap
Cordova
titaniumunity
Android
iOS
24. What happened after AWE 2016
● 6 releases of the Wikitude SDK
● 4 feature releases
● 2 maintenance stability releases
● Entire new Wikitude Studio Editor
● Updated Unity plugin and Editor
support
● Re-vamped support platform
● New ticketing system and forum
● New documentation
25. Computer Vision changes
● SLAM 3D engine
● New tracker available called InstantTracker
● Improved performance for Extended Tracking
● Revamped 2D engine
● Boosted recognition accuracy
● Improved tracking stability and accuracy
● Reduced tracking jitter
28. API changes and Improvements
● Updates to Plugins API
● Re-designed API to CV Tracker
● New CV Tracker
● Rendering and Camera Improvements
● Multi-touch gesture Support in JavaScript API
● 1:1 calibration for Smartglasses
● Support for ODG R-7 glasses
29. Updates for the Plugins API
● New type of plugin: Input Plugins
● YV12 color space for input plugins
● New type for rendering: Positionables
● Own implemented trackers can re-use rendering
of augmentations in JavaScript API
Custom
Frame
Provider
Image
Frame
Standard
Frame
Provider
Wikitude
CV EngineRendering
Custom CV
Plugin
Plugins API
30. Tracker changes
● Old naming scheme for trackers didn’t
scale
● More descriptive names
● ImageTracker now can have options
● Local/Offline
● Cloud
● New Trackers added (and to come)
31. Rendering and Camera
Improvements
● Update to work with Android Camera2 API
● Enables capturing with higher fps
● Fine-grained control over focus behaviour
● New Graphics APIs supported
● OpenGL ES 3.x
● Metal
● HD Camera Frame rendering
● Choose between SD to FullHD
● Drawables in billboard mode
32. Multi-Touch Gestures
● Needed for any interaction with augmentations
on screen
● Only available in the JavaScript API
● Main gestures covered
● Rotate
● Pinch-to-zoom
● Move/Drag/Pan
● React on callbacks for gestures
33. Additions for Smart Glasses
● Official support for ODG R-7 smart glass
● Personal calibration and calibration profile for
all supported smart glasses with see-through
glasses
● Profile can be stored for each user
● Calibration process can be whitelabeled
34. New Studio Editor
● Entirely reworked from former Wikitude Studio
● Based on MEAN stack together with WebGL
rendering
● Fully web-based
● Integrated with Target Manager solution
● Experiences can be exported and are
converted to match JavaScript API
● Keyboard controls
35. What happens after AWE 2017
● Next release of Wikitude SDK is around the corner
● Object Recognition based on SLAM engine with pre-
generated maps
● Update for the 2D Image Computer Vision engine
● Enhancements for SLAM 3D engine
36. Object Recognition & Tracking
● Small-Scaled objects can be recognized and tracked like
any other Image Tracker
● Maps are pre-generated based on video provided by the
developer
● Define occlusion models (not only for Object
Recognition)
● New tracker for all APIs
38. Update for 2D Image CV engine
● Support for multiple image tracker at same time
● Unlimited – processing power is the limit
● Can detect duplicate targets (more of the same)
● API gives you distance and orientation between
targets
● Faster and more accurate initial recognition
● Image Recognition Extended Range (IR ER)
● Boost recognition distance by factor 3 – US letter
sized target can be recognized from 8 ft
40. Enhancements for SLAM 3D Engine
● Better performance for 32-bit devices
● Higher accuracy during mapping and tracking
● Depth API to query depth information in SLAM for any
screen-coordinate
● Additional initialization plane for Instant Tracking
(vertical, horizontal)
42. Geo AR
42
How to define e.g. a point
of interest (POI)
• Define where the
augmentation is placed by
creating AR.GeoLocations
(latitude/longitude coordinates)
• Define the augmentation by
creating AR.Drawables
• Combine location and
drawables using
AR.GeoObjects
= The same geo object is displayed at different locations
POI C
POI A
POI A
Poi B
43. Pokémon Go like Geo AR
43
How to build a Pokémon Go like AR experience:
44. Pokémon Go like Geo AR
44
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
//a relative location being 12 meters south, 20 meters west and 13 meters lower than current position of the user
var relativeLocation = new AR.RelativeLocation(null, -12, -20, -13);
45. Pokémon Go like Geo AR
45
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
//latitude, longitude, altitude
var location = new AR.GeoLocation(47.77317, 13.069929, 320.);
46. Pokémon Go like Geo AR
46
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon
//create a new Model
var model = new AR.Model(“models/pokemon.wt3”);
47. Pokémon Go like Geo AR
47
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• Relative to the current user position using AR.RelativeLocation
• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon
• Can be animated through .fbx defined animations
//We want to start an animation inside a model
var model = new AR.Model(“models/pokemon.wt3");
var modelAnim = new AR.ModelAnimation(model, "hit"); //start hit-animation defined inside model
48. Pokémon Go like Geo AR
48
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon
• …
• AR.Model object to render the 3D model of a Pokémon
• …
• AR.GeoObject to actually bring the model onto the screen
// a GeoObject with drawables set on creation time
var geoObject = new AR.GeoObject(location, {
drawables : {
cam: [model] //the drawable representing the GeoObject in the camera view
}
});
49. Pokémon Go like Geo AR
49
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
geoObject.onEnterFieldOfVision = function() {
/* your custom behaviour e.g. start a certain model animation */
};
geoObject.onExitFieldOfVision = function() {
/* your custom behaviour */
};
50. Pokémon Go like Geo AR
50
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
var imageResource = new AR.ImageResource(“assets/pokemonIndicator.png”);
var pokemonIndicator = new AR.ImageDrawable(imageResource, 0.2);
// a GeoObject with drawables set on creation time
var geoObject = new AR.GeoObject(location, {
drawables : {
cam: [model], //the drawable representing the GeoObject in the camera view
indicator: pokemonIndicator
}
});
52. GeoObjects & HTML content
Menu bar
jQuery support
Geo-located POIs
Multiple Drawables
- 3 images
- 3 text labels
Web View
Augmented Reality View
53. Pokémon Go like Geo AR
53
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
• Use the AR.Radar component to permanently show the position of
a Pokémon in the AR scene
55. Pokémon Go like Geo AR
55
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger
• trigger custom events when the Pokémon becomes visible in the
camera
• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
• Use the AR.Radar to permanently show the position of a Pokémon in
the AR scene
• Capture a screenshot after you caught a Pokémon using the
corresponding ArchitectView Java API.
56. Image Recognition
Augment target images using
JavaScript API capabilities
(Drawables, Sounds,
Animations,Videos, 3D Models)
var kitchenTracker = new AR.Tracker("Kitchen.wtc");
var overlay = new AR.VideoDrawable("howdens.mp4", 0.65);
var trackable2DObject = new AR.Trackable2DObject(
kitchenTracker: "Kitchen",
{ drawables:
{ cam: overlay
}});
