1. A Real-time Collaboration-enabled Mobile
Augmented Reality System with
Semantic Multimedia
Dejan Kovachev, Gökhan Aksakali & Ralf Klamma
RWTH Aachen University, Germany
Advanced Community Information Systems (ACIS)
kovachev@dbis.rwth-aachen.de
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-1 This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

2. Advanced Community
Information Systems (ACIS)
Responsive
Web Engineering Community
Web Analytics
Open
Visualization
Community
and
Information
Simulation
Systems
Community Community
Support Analytics
Lehrstuhl Informatik 5
Requirements
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-2
Engineering

3. Agenda
 Motivation and scenarios
 Conceptual approach
 System design and implementation
 Evaluation
 Related Work
 Conclusions and Future Work
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-3

4. Motivating Scenario
Documentation of Ghazni Wall
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-4

5. Digital Documentation in Cultural
Heritage
 Ghazni, Afghanistan, capital of Islamic Culture 2013
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-5 http://arch.rwth-aachen.de/cms/Architektur/Forschung/Forschungsprojekte/Cultural_Heritage/~cqdn/Ghazni_2013/lidx/1/

6. Workflow of Documentation
1- images and videos
2- tag basic metadata
On-site
Documentation
Expert
On site
Architecture Remote expert (Historian)
Expert
Collaborative 8-view repository
Multimedia Cloud
Metadata
Local Workforce Repository
Lehrstuhl Informatik 5
MPEG-7,
(Information Systems) CIDOC CRM
Prof. Dr. M. Jarke
I5-KAKl-1012-6

7. Increase Awareness
of Cultural Heritage
 Multimedia
overlays on
video stream
Metadata
 Interaction via
position and 3D Captured time
movement Taken location
Creation time
Provenance
….
 Still missing Ancient Helmet
– real-time collab.
– semantic
Lehrstuhl Informatik 5
metadata
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-7

8. XMPP-base Mobile Multimedia
Collaboration (XMMC)
Goals: How to enable
 Collaborative digital documentation of historical
sites with rich multimedia and metadata using
inexpensive hardware
 Increase of cultural heritage awareness
Prototype: XMMC
 Mobile real-time multimedia collaboration system with
- integrated AR multimedia browser
- standardized metadata for semantic multimedia and
annotations (MPEG-7)
Lehrstuhl Informatik 5
(Information Systems) - Coordination, field note taking, instant geo-tagged multimedia
Prof. Dr. M. Jarke
I5-KAKl-1012-9 acquisition and sharing

9. XMMC Use Cases
Share
Acquire multimedia
multimedia Content
Producer <<extend>> Add POI
Annotate
multimedia Synchronize with
other clients
Collaborator Chat
Call LAS MPEG-7
Browse services
AR
<<include>>
<<include>> Display the
Consumer Select a POI
details
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-10

10. Concurrent Editing of XML
 Most metadata standards use XML format
– Used by standards like MPEG-7, CIDOC CRM, SVG
 XML suited for concurrent editing of metadata
– Hierarchal structure more efficient then linear
 Concurrency Problems
– Divergence, Causality Violation, Intention Violation
Approach Architecture Algorithms Examples
Pessimistic centralized locking, turn taking SVN,CVS
Optimistic P2P OT, treeOPT, CMAX Google Operational
[Gerlicher 2007] Wave Transformation,
Lehrstuhl Informatik 5
(Information Systems)
CEFX+
Prof. Dr. M. Jarke
I5-KAKl-1012-11

11. Consistency Maintenance Algorithm
for XML (CMAX) [Gerlicher 2007]
 Out-of-the-box lightweight concurrent editing of XML files
– CEFX+ [Voigt 2009] based on CMAX
 Optimistic approach mainly inspired from OT
– First execute locally then propagate
 Direct addressing with <cefx-uid>
 Concurrent editing of the same node
– Delegates conflict resolution to user
 State Vector
– Operation count for every participant
 History Buffer
– Previous operations
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-12

12. eXtensible Messaging and Presence
Protocol (XMPP)
 Classic SOAs like HTTP, SOAP are inadequate for
real-time purposes
 Good fit for real-time communication
 Easily customizable & extendable
– Open XML-based standard
 Existing extensions
– File transfer, multi-user chat, PubSub
 Library and server support
– Smack(Java), Openfire
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-13

13. XMMC Architecture
Client side Server Side
Client Side
XmmC Client XmmC Server
Media Acquisition Collab.
Modules
XMPP
AR Multimedia Relational
Annotation
Module Module DB
AR Browser Module
Metadata LAS-
Annotation Media Media Collab.
AR MPEG7 MPEG-7
Catalog Store Editing
Service Integration Services
Service Service Service
CEFX+ Service
XMPP Connection XMPP Client (Smack) implemented
Layer (XCL) modified
previously available
XMPP
XMPP Client
(aSmack)
XMPP XMPP Server (Openfire)
Camera WLAN GPS Compass ACCEL
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-14

14. Technical Contributions
 Adaptation of CEFX+ for multimedia annotation
– New operations: editing and deleting of nodes
– Later join, leave collaboration session
 XMPP-based AR browser by extending Mixare
[http://www.mixare.org/]
– POI awareness, ARML-based, mobile content provider
 Custom XMPP IQ stanzas for multimedia sharing and AR
 Integration with MPEG-7 metadata services
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-15

15. Collaborative Annotation
Mobile Client-2 Mobile Client-1
Collaborative Collaborative MPEG-7
Mpeg-7
Annotation Editing Integration
<message id="8Dpbh-26" Service Service Services
Module
to="xmmc@merian.informatik.rwth-aachen.de/Smack"
from="3b90aa319d44@conference.merian...de/test_1"
<jabber:iq:rpc> joinSession() joinSession()
type="groupchat">
<jabber:iq:rpc> SessionData SessionData
<body>
<basetype type="time" cefx:uid="101"> loadDocument()
<jabber:iq:rpc> loadDocument()
<name cefx:uid="201">
<jabber:iq:rpc> DocumentData DocumentData
Second World War</name>
<date cefx:uid="301">
<jabber:iq:rpc> joinSession()
…
1997-09-24T00:00:00:000F1000+01:00
<jabber:iq:rpc> DocumentData
</date> updateSemantic
<groupchat>Mess </basetype> Message (Operation)
<groupchat> executeOperation() Semanaticbasetype
age (Operation) BaseTypes()
</body> Service()
…
<x xmlns="jabber:x:cefx#ins" Ack Ack
p="100" ci="1" leaveSession()
<jabber:iq:rpc> sv="0,1"
ba="1" fn="null"/>
<jabber:iq:rpc> leaveSession()
</message>
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-16

16. XMPP-based Augmented
Reality Browser
Contemporary mobile AR Browsers
3rd Party
GPS data Content
Provider
Provider
Server
Get POIs
Wikipedia,
SOAP/RESTful Data Flicker,
Facebook,
etc.
XmmC
GPS data
XMPP XmmC
Create POI Server Create POI Server
Edit POI Metadata Edit POI Metadata
Event Message PubSub Publish Event
Node Data
Get POIs Get POIs
XMPP XMPP
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-17

17. Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-18

18. Technical Evaluation
Collaborative Annotation
 Remote operation execution
– Average = 412ms
– Stan. Derivation = 209ms
 Conflict resolution
– Resolved the possible conflicts
– with 900ms latency and1,2 KB/s bandwidth network
– Still has open issues regarding undo and redo operations
AR Browser
 The positions of closed POIs are not very accurate due to
Lehrstuhl Informatik 5
- Inaccuracy in GPS signal, compass, accelerometer
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-20
- Presence of nearby metal objects causes inaccurate results

19. Technical Evaluation
Energy consumption
 XMPP
– Generates verbose XML streams
– Requires constant open sockets
– Community works for
Distribution of Stanza Types
improvement for mobile networks
 AR browser consumes 4
times more battery than
collaborative annotation
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-21

20. User Evaluation
 Field study with 7 participants
– various mobile devices, Android versions and screen sizes
 Tasks
– acquisition of multimedia artifacts in a historical site
– editing basic metadata
– collaborative annotation
– browsing with AR
 Questionnaire
– collect subjective qualitative measures and user experience
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-22

21. Evaluation Session Results
 Majority of the participants observed real-time updates on
the annotations (85%)
 Participants mostly satisfied with the user interface and
found it user-friendly and responsive
 MPEG-7metadata service integration was successful
 The prototype increased
cultural heritage
awareness of
the participants
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-23

22. Related Work
Real-time
Bi-directional Mobile AR Multimedia
System Mobility Light-weight Collaborative
Comm. Browser Metadata
Editing
Mobilis
++ ++ - - ++ +
(TU Dresden, 2009)
Mobile Campfire
++ - - + - ++
(RWTH Aachen, 2010)
Junction
++ + - ++ - -
(Stanford, 2010)
Google Wave
- ++ - - ++ -
(Google Inc, 2009)
Novell Vibe Cloud
- ++ - - ++ +
(Novell, 2011)
CEFX+
+ ++ - + ++ -
(London Arts U., 2007)
Layar
++ - ++ - - -
(Layar, 2009)
Mixare
++ - + ++ - -
(Mixare.org, 2010 )
Wikitude Architect
++ - + + - -
(Wikitude, 2009)
XmmC
(This work)
++ ++ + ++ ++ ++
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-24

23. Conclusions
 XmmC is capable of storing and sharing various
multimedia types and sizes
 Concurrent XML file editing with mobile clients
 Collaborative annotation in digital documentation of
historical sites and increasing cultural heritage
awareness
 XMPP fits well for mobile real-time multimedia
collaboration context
 However, AR and XMPP consumes battery
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-25

24. Future Work
 Further multimedia types
– Videos, 3D objects, visual formats
 Improved AR browser with 3D rendering and
visual location mapping
 Integration with real-time video streaming
services, mobile cloud computing
 Community support
– User centric features, e.g. trust and security
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-26

25. Thanks for your attention!
Q&A
Lehrstuhl Informatik 5
(Information Systems)
Prof. Dr. M. Jarke
I5-KAKl-1012-27