artoolkitX is a new open-source framework based on the long-running ARToolKit software development kit for augmented reality (AR) application development. In this presentation given at an AR event in January 2018, Philip Lamb discusses opportunities and challenges facing facing open-source AR in 2018.
Vector Search -An Introduction in Oracle Database 23ai.pptx
Â
Opportunities and Challenges in Open Source AR in 2018
1. Opportunities and Challenges in
Open Source AR in 2018
Philip Lamb - Principal Engineer and Project Lead, artoolkitX
2018-01-31
phil@artoolkitx.org
2. Who am I?
⢠Lead engineer for artoolkitX project, supported by Realmax artoolkitX
⢠19 years experience in AR & VR
⢠Former Chief Technical Officer of ARToolworks
⢠Former Chief Open Source Architect at DAQRI
3. 2015
ARToolworks becomes part of DAQRI
(Lamb, Vaughan, Furness, Billinghurst, Kato)
2009
⢠1999 - first demonstrated publicly
⢠2001- v1 released open source (Washington)
⢠2001 - ARToolworks incorporated with dual licensing model
⢠2004 - v2 released open source (Sourceforge)
⢠2008/2012 - open source innovation
⢠2012/2014 - extending platform support
⢠2015 - acquired by DAQRI - pro versions open sourced
⢠2017 - Realmax announces support for artoolkitX
1999
SIGGRAPH 1999 ARToolKit Shared Space demo VR2009: Kato award
4. ARToolKit
userbase
Self-reported affiliation data from 2016-06 survey of ~1000 users on ARToolKit mailing list.
Response rate 51%, of whom 78% were ARToolKit users.
Comm
ercial
Devel
oper
51%
Resea
rcher
21%
Indepe
ndent
Develo
per
21%
Educa
tor
7%
5. ⢠Existing ARToolKit 5 open source
community shows strong ongoing
growth
⢠2015-05-13 to 2017-04-01 (698 days):
⢠182k SDK downloads
⢠31k page views per month on
artoolkit.org
⢠ARToolKit5 in top 0.1% of C++
projects on GitHub with 479 forks*
and 1015 stars
⢠ARUnity5 in top 0.5% of C# projects
on GitHub with 56 forks and 315
stars
⢠3.1k** members and 5.6k unique
posts on ARToolKit Forum
*# of active forks. **1000+ active users.
Number of GitHub âStarsâ (users
recording ARToolKit v5 as
favorite)
ARToolKit usage by the numbers
6. Motivation and Goals
ARToolKit is a collection of software tools to help solve
some of the fundamental problems in augmented
reality, including geometric and photometric registration
⢠geometric registration: aligning the position of the
virtual environment with the actual
⢠photometric registration: matching the appearance of
objects in the virtual environment to the actual
7. Problems!
1.Vendor-provided visual-inertial SLAM is closed source
(ARKit/ARCore)
2.Our identity, emotional state, or body actions, are not part of AR
sensing
3.âAR anywhereâ is still a dream. There is no AR content âright here
right nowâ.
4.AR visuals are cartoon-like and not realistic.
5.Multi-user AR is virtually non-existent.
8. artoolkitX: Future AR platform
Computer
vision
Tangible
user
interaction
High-
fidelity
AR
Multi-user
virtual
environments
Tool
s
Frameworks
Community
IOT
artoolkitX project goals
Q1-Q2 2018
⢠One open-source framework
from each area integrated
⢠One new project started in
each area with collaboration
between artoolkitX and a 3rd
party
9. Problem statement (2):
Problem: The identity, emotional state, bodily position and non-hand
actions are extremely rich sources of information for AR systems, but
are routinely ignored.
Approach: We need to apply active sensing to collect information from
the user, and make it easier to integrate active sensing into AR
applications, to fuse data from different sources, and to process time-
based data for informative structure.
Outcome: AR systems will be better user experiences with lower
barriers to use.
10. Tangible user interaction
⢠Object and environment mapping - Kinect, RealSense, StructureIO
integration to allow use of physical proxies / totems
⢠Integrate support for haptic data into virtual environment
⢠Library support for force-feedback hardware
⢠Integration of face tracking, gaze tracking, hand and gesture tracking
11.
12. Problem statement (3):
Problem: When we build interactive AR applications that are deployed in
a fixed location, we face the challenge of sufficiently bringing computer-
mediated content into the fixed space around the user. This is an
outwards-in transferral of information.
In contrast, in deploying mobile AR applications we face the opposite
problem, that of guiding the user to the content.
Approach: Provide a simple system for reliable tagging of physical
objects.
Outcome: Users will know where to find content.
13. BIDBOT
Big Internet DataBase Of Things
Virtual environments (i.e. âAR contentâ) need to be attached to specific
real-world things.
Currently, any given system only knows the appearance of a tiny number
of things, and its reach is therefore also tiny.
We need to enable client applications to learn the appearance of any
object in a global database of things
Outcome: we enable the fine-grained tracking of a large (and expandable)
set of things both in online and offline modes, and consequent sharing of
state across time and space.
15. IoT
⢠Support for ambient sensing from Zigbee (à la HomeKit) sensor
networks, Bluetooth-LE beacons and sensors
16. Problem statement (4)ďź
Problem: AR experiences are cartoon-ish.
The visual presentation of the virtual environment is never matched to the
optical properties of the environment in which it is being embedded.
Approach: We need to sense those optical properties so that we can make
virtual content with correct lighting, shadowing, and surface texture.
Non-visual dimensions of AR experiences are also neglected and are likely to
be significant drivers of future AR experiences and interaction.
Outcome: enhancement of the aesthetic properties of AR experiences, and
increased user engagement, acceptance, and satisfaction.
17. High-fidelity AR
⢠Radiometric scanning methods using consumer RGBD cameras
⢠HDR spherical panoramas for specular lighting
⢠Environment mapping (Kinect/Realsense) for dynamic object
occlusion in video see-through AR
⢠Positional audio integration into virtual environment
⢠Dynamic stereo vision and rendering
21. Open-source benefits: for enterprise and institutions
FAST
GOOD
CHEAP
⢠Contributions from
other commercially-
motivated developers
⢠Side benefits for
recruiting/retention,
marketing
⢠Incentivised
development
⢠Choose problems
that benefit from the
âlong leverâ
⢠Parallelise projects
⢠Thousands of eyeballs
beats hurried in-house
code review
⢠Fixes can go live
immediately
⢠Use selects desired
feature set
AWE-
SOME
22. License terms
BSD-like Apache LGPLv3 GPLv3
Use of code in closed-
source
All All As linked library None
Acknowledgement
required
No No Yes, plus link Yes, plus link
Retention of
copyright/license
headers
Disclaimer only
If redistributing complete
file(s)
Yes Yes
Requires publication
of users source
No No
Modifications to the
library
Yes
License for
example code
License for
libraries & utilities
23. artoolkitX - Project Operation
⢠Existing open-source tools
and techniques
⢠Newly developed IP
⢠IP licensed from /
codeveloped with partners
Funded
artoolkitX
engineers
e.g.
sponsored
by
Realmax
University
& Technical
Institution
Collaborato
rs
Existing
Open-
Source
Projects
e.g.
OpenCV
artoolkitX
Project
Leaders
24. artoolkitX - Upcoming
⢠Today: supporting existing ARToolKit open-source community.
⢠Q1 2018: Update to ARToolKit v5 and JSARToolKit (web-based).
⢠Low-cost AR platform for education â Raspberry Pi + artoolkitX.
⢠Integration of commercial platforms ARKit/ARCore
25. ⢠Open-source new developments including high-quality tracking
⢠Support embedded commercial tracking (ARKit/ARCore)
⢠Modernise target platforms and reduce need for legacy support.
⢠Create SDK suitable for full spectrum of expertise
⢠high-level low-complexity API for novices
⢠expose underlying CV APIs for experts while simplifying ancillary
tasks
User experience goals
26. libARTextureTracker libAR libARVI
ARTrackerTexture ARTrackerSquare ARTrackerVI
ARTrackableTex
ure
ARTrackableSquare
ARTrackableMultiSquar
e
ARTrackableVI
libARVideo
ARVideoSource
Camera
calibration
database
sqlite3
curl
CoreMotion OpenGL/GLES
libARG
ARTracker
ARTrackable
AVFoundation
Video4Linux2
C API
Java API C# APIObjective-C API
C++ class External library Internal library Hybrid system BindingKey:
ARTrackableAppearance
OpenCVAndroid
Video
Push
JNI P/Invoke
ARController
Architectural overview
ARVideoView
I am very pleased to be here through the support of Realmax. I am lucky to be able to continue my work on augmented and mixed reality as lead engineer for a new project named artoolkitX. My background is first as a developer of experimental virtual reality systems 19 years ago. Then I was fortunate to work with Tom Furness at the HIT Lab in Seattle, and later becoming Chief Technical Officer of ARToolworks where I worked for many years helping bring AR software to developers. More recently I have worked for DAQRI using open-source to drive innovation inside and outside that company. With Realmax, we have some great ideas which Iâll tell you some more about.
ARToolKit has a long history and is arguably one of the most widely used pieces of software in the history of AR. It was originally developed by Kato and Billinghurst at the HIT Lab in Seattle, and over many years we took it from research to a commercial product, and then back to open-source community-oriented project again.
Who uses ARToolKit? Mostly developers who are building commercial applications and platforms. With artoolkitX, our goal is to grow all pieces of this pie.
To put some figures on the usage, we see that ARToolKit version 5 is not only one of the leading open source augmented reality software development kits, it is one of the leading overall open-source projects.
These figures were collected in 2017, and we expect to continue this growth as we move from ARToolKit version 5 to artoolkitX.
The goal of the ARToolKit project is to solve fundamental problems that AR software developers face. Up until about 18 months ago, that was still two particular problems: geometric registration and photometric registration. Matching the layout and appearance of a virtual environment to the actual environment surrounding the user. But if we look 18 months into the future, these are not the problems that developers are still going to be facing. What problems are there?
Iâm going to list 5 problems, and for the remainder of my time, Iâm going to talk about how weâre tackling these problems with open-source development, and I hope I will interest some of you in joining the project and participating in the solution.
So here they are [read off slide]. I am going to discuss some approaches to problems 2, 3 and 4.
This slide really sums up the approach weâre trying to take to solving these problems. Each of the circles represents an area of active research relevant to the problems I just posed. And in each area, weâre choosing a small area relevant to augmented reality. Because each of these areas is very large on its own! And the idea illustrated here is that if we carefully combine solutions from each area, we will generate a cohesive whole that is very useful and relevant to AR.
This video shows some of the possibilities for far more engaging interaction when we allow the user to directly influence and alter the represented digital content. Although this particular example is not recent, we are just now at the point where we can achieve some of the same techniques using a monocular optical pathway.
We are all familiar now with the use of QR codes, but use of QR codes in AR is problematic, and not only because theyâre visually ugly, but because they have poor utility as sources of tracking information. augmented.info markers simultaneously address both appearance and trackability aspects in a robust way. The system supports both online and offline modes for ID lookup.
There is a lot that could be said about the integration of IoT with AR, but one easy integration weâre planning to target in artoolkitX is to utilise the rich data source provided by your in-homeIoT networks and to map that sensor space into a visual space.
Itâs unfortunate that the problem of visual realism in AR is still widely treated as an output problem on the visualisation pipeline. However, in order to effectively increase the visual match between virtual and real objects, we need to actively sense the environment; this is the photometric registration problem. There are a number of useful techniques we plan to make available in the AR pipeline, independent of the final rendering engine.
This recent research is one of the first low-cost techniques Iâve seen that doesnât depend on placing probes into the real scene.
Iâd like to conclude this overview with a brief exposition of how the artoolkitX project operates and how commercial users, educational and government funded partners, and independent developers can benefit from participation.
The artoolkitX project has a long open-source history. The benefits of open source are self-evident to anyone who has ever picked up an open-source library or application as a user.
Enterprise users often mistakenly believe that open-source is incompatible with commercial goals. To that I would counter that all of the largest tech companies are significant users and contributors to open-source, and in the case of both Apple and Google, have core commercial operating systems built on open-source foundations. So when applied at the appropriate part of the value chain, open source can help you get closer to the sweet spot between cheap, fast and good quality.
