ABSTRACT: Creating and maintaining scenes for mobile Augmented Reality browsers can be a challenging and time consuming task. The timeliness of digital information artifacts connected to changing urban environments require authors to constantly update the structural representations of augmented scenes or to accept that the information will soon be outdated. We investigated approaches for retrieving multimedia content and relevant web services for mobile Augmented Reality applications at runtime. Using semantic web technologies we are able to postpone the retrieval of actual media items to the moment a user actually perceives an augmented scene. This allows content creators to augment a scene only once and avoid continous manual updates.

1. Jens Grubert+, Lyndon Nixon†, Gerhard Reitmayr+
+Institute for Computer Graphics and Vision
†STI International
Augmenting the World using Semantic
Web Technologies

2. Do I have time?
Are my friends attending?
What kind of music they play?
How much are the tickets?

3. Do I have time?
Are my friends attending?
What kind of music they play?
How much are the tickets?
As time goes by …

4. How to create information?
How to update information?

5. Standard approach:
direct links to content
Creating
augmented media
…
<mainresource>
http://www.klimmstein.com/before.jpg
</mainresource>
…

6. Standard approach:
manual update
…
<mainresource>
http://www.klimmstein.com/before.jpg
</mainresource>
…
outdated information
after.jpg
Updating
augmented media

7. Link to concepts, not to content
Retrieve the content at runtime
Our approach
Less effort for updating augmented scenes
Potential added value due to context-sensitive
retrieval

8. Link to concepts, not to content
Retrieve the content at runtime
Our approach

9. Concept
Link to structured machine-readable data in
ontologies (Linked Data)
Link to concepts, not to content

10. • Events:
o Clubbings, festivals
o Small to midscale
• Authors:
o Event organizers
o public relations
Link to concepts, not to content

11. Link to concepts, not to content
concept
(URI or tag)
display
region

12. Link to concepts, not to content
TOI
reposit
ory
TOI
Thing of Interest
structured description of
an augmented object

13. <toi:segment_annotation …>
<toi:representation>
<toi:hasDisplayLocation>0.59, 0.03</toi:hasDisplayLocation>
<toi:hasDisplayDimension>0.44,
0.31</toi:hasDisplayDimension>
<toi:hasLocatingInformation>0.11,
0.03</toi:hasLocatingInformation>
<toi:hasDimension>0.44, 0.34</toi:hasDimension>
</toi:representation>
<toi:represents>
<toi:identifier="http://data.play.fm/artist/chrisharvey">
<toi:classification
rdf:resource="http://www.play.fm/ontology#Artist"/>
</toi:represents>
</toi:segment_annotation>

14. Link to concepts, not to content
Retrieve the content at runtime
Our approach

16. TOI
Reposit
ory
SmartReality
Server
Web of
Data
Semantic Web
Services
Augmented Reality Client
Object
Recognition
Thing-of-Interest
Initialization
Content
Retrieval
Tracking and Rendering
Image
Recognition
DB
Retrieve the content at runtime

17. Image
Recognition
DB
ID
Millions of Images
Augmented Reality Client
Object
Recognition

18. TOI
Reposit
ory
SmartReality
Server
Augmented Reality Client
Thing-of-Interest
Initialization
Image ID
Sesame
Triple Store
http://www.smartreality.at/rdf/toi
How is it
represented?
What does it
represent?
(classification)
Thing-of-Interest (TOI) TOI
TOI
TOI
…
Linking Open Data cloud diagram, Sept. 2011.
http://lod-cloud.net/
Artist,
Event,
Club,
Show,
Recording,
Track
Timeline, Event,
Dublin Core
Uses:

19. SmartReality
Server
Augmented Reality Client
Content
Retrieval
Linked Open Data Collector
TOI
classification
Crawling Rules
RDF/N3
e.g., for each Music
Group crawl also each
member
+
BBC Music
MusicBrainz

20. SmartReality
Server
Semantic Web
Services
Augmented Reality Client
Content
Retrieval
RDF Service
Repository
(WSMO-Lite) Discover relevant web services
Rank and execute web
services

21. How are the Posters
annotated?

22. who | what?
when? where?

23. • Authors:
o event organizers
o PR personal
• Events:
o clubbings, festivals
o small to mid scale

27. <toi:segment_annotation …>
<toi:representation>
<toi:hasDisplayLocation>0.59, 0.03</toi:hasDisplayLocation>
<toi:hasDisplayDimension>0.44,
0.31</toi:hasDisplayDimension>
<toi:hasLocatingInformation>0.11,
0.03</toi:hasLocatingInformation>
<toi:hasDimension>0.44, 0.34</toi:hasDimension>
</toi:representation>
<toi:represents>
<toi:identifier="http://data.play.fm/artist/chrisharvey">
<toi:classification
rdf:resource="http://www.play.fm/ontology#Artist"/>
</toi:represents>
</toi:segment_annotation>

28. Challenges
• Effort to extend it beyond existing use case?
o Suitable Ontologies
o Crawling Rules
• Semantic Descriptions of 3D Models?
• Performance (roundtrip currently ~ 10 seconds)
• How to capture the context?
o User Model, Context Model (e.g., FOAF, Activity Streams)

29. Summary
• Goals
o Context-aware information access ion real objects in-situ
o Minimize creation effort for authors (non-programmers)
• How?
o Semantic Web Technologies for content retrieval
o Augmented Reality for content presentation
• Challenges
o Not yet able to model visually compelling (3D) scenes

30. This work is made possible by the
Austrian National Research Funding Agency FFG in the
SmartReality project.
Thank you for your attention.
www.smartreality.at