Nowadays multimedia data is produced and consumed at an ever increasing rate. Similarly to this trend, diverse storage approaches for multimedia data have been introduced. These observations lead to the fact that distributed and heterogeneous multimedia repositories exist whereas an unified and easy access to the stored multimedia data is not given. This paper presents an architecture, named AIR, that offers the aformentioned retrieval possibilites. To ensure interoperability, AIR makes use of recently issued standards, namely the MPEG Query Format (MPQF) (multimedia query language) and the JPSearch transformation rules (metadata interoperability).

1. Motivation
A.I.R. Framework
Conclusion
AIR: Architecture for Interoperable Retrieval on
Distributed and Heterogeneous Multimedia
Repositories
Florian Stegmaier 1 ,
2
Mario D¨ller 1 , Harald Kosch 1 ,
o
Andreas Hutter and Thomas Riegel 2
1 Chair of Distributed Information Systems, University of Passau
2 Corporate Technology, Siemens AG
April 13th, 2010
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

2. Motivation
A.I.R. Framework
Conclusion
Outline of this talk
1 Motivation
Multimedia retrieval today and its issues
Distributed and heterogeneous retrieval use cases
2 A.I.R. Framework
System requirements
Integrated standards
Search concepts
3 Conclusion
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

3. Motivation
Multimedia retrieval today and its issues
A.I.R. Framework
Distributed and heterogeneous retrieval use cases
Conclusion
The role of multimedia retrieval in today’s world
tremendous interest of users in multimedia (MM) retrieval
terrabytes of multimedia data and metadata offered in popular
systems
efficient analysis of MM data could improve workflows of...
...video surveillance
...medical examination
...tourism (e.g., computer based tourist guides)
...
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

4. Motivation
Multimedia retrieval today and its issues
A.I.R. Framework
Distributed and heterogeneous retrieval use cases
Conclusion
Multimedia retrieval issues
different kinds of MM data (e.g., video vs. audio) with
different compression techniques (e.g., MPEG-2 vs. MP3)
diversity between available metadata formats lead to
interoperability issues (complexity, coverage,
comprehensiveness, serialization)
beside syntactic, semantic informations need to be processed
→ “bridging the Semantic Gap”
Current situation:
various scientific disciplines (e.g., signal processing, machine
learning and information retrieval) and data formats (e.g., XML
and RDF) in use to overcome these issues
→ soon ending up in distributed, heterogeneous environments
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

5. Motivation
Multimedia retrieval today and its issues
A.I.R. Framework
Distributed and heterogeneous retrieval use cases
Conclusion
Use cases for distributed and heterogeneous retrieval
We have been confronted in recent projects with two use cases:
Video surveillance Medical examination
Medical examination system
Surveillance system 1 Surveillance system 2
query query query
Surveillance system n
query query query query
Dublin Image
MM data MPEG-7 MM data ... RDF triple
Core Retrieval DICOM
store(diseases
System (patient data)
and anatomy)
(CT Scans)
“Select CT Scan, where disease
“Find video segments where an is lesion, anatomy near liver and
identified person is running the CT scan is similar to
through the scene!” aGivenCTScan.jpg!”
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

6. Motivation System requirements
A.I.R. Framework Integrated standards
Conclusion Search concepts
AIR: General requirements
Overall goal:
Providing an unified access to distributed and heterogeneous
environment
middleware based architecture (for connecting an arbitrary
amount of backends)
loose coupled, modular architecture (easy extensibility)
support of a broad scope of multimedia queries (e.g.,
Query–By–Example or relevance feedback)
cross system multimedia retrieval (cross language as well as
cross metadata formats)
use of international standards for a wide support
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

7. Motivation System requirements
A.I.R. Framework Integrated standards
Conclusion Search concepts
AIR: Integrated standards
MPEG Query Format (MPQF) – part 12 of MPEG-7
(XML based) multimedia query language supporting f.e.
QueryByMedia, QueryByDescription and QueryByFreeText
beside query structure, MPQF offers management features
like (de-)register and service discovery
JPSearch transformation rules – part 2 of JPSearch
targeting (XML based) metadata interoperability using a pivot
metadata format and syntactical defined mappings
Ontology and API for Media Resource 1.0 (soon W3C Rec.)
tackling (ontology based) metadata interoperability using core
properties as a pivot metadata format and syntactic as well as
semantic mappings
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

8. Motivation System requirements
A.I.R. Framework Integrated standards
Conclusion Search concepts
AIR: Supported search concepts
Local processing Distributed processing
MPQF query MPQF query
AIR AIR
Image Image Image Image Other Ontology
... ... search ... search ... search
search search search
engine 1 engine i engine n engine engines engine
Data set 1 Data set i Data set n Global data set
→ e.g., video surveillance → e.g., medical examination
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

9. Motivation
Results & future work
A.I.R. Framework
Discussion
Conclusion
Conclusion & future work
Conclusion
current implementation targets the local processing
functionalities take a fractional amount of processing speed
first prototype already in use in several projects
Future work (excerpt of projects in queue)
“intelligent” result aggregation (e.g., by query classification or
quality measurement / calibration of backends)
start implementation of the W3C interoperability approach
multimodal result presentation
extending AIR to the distributed processing
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

10. Motivation
Results & future work
A.I.R. Framework
Discussion
Conclusion
Discussion
Thank you for your attention...
...any questions?
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

11. Motivation
Results & future work
A.I.R. Framework
Discussion
Conclusion
AIR architecture
MPQF Factory Layer
metadata
extraction
API
calls Client MPQF
client Communication API file
validator plug-ins
Layer conversion
MPQF
...
MPQF
Backend Response Layer MPQF Backend
Benchmarking Management Management
Layer Layer Layer
result MPQF query service
aggregation management discovery
evaluation &
benchmarking
fetch result MPQF query backend
(async mode) distributor management
Backend
Communication
Layer
MPQF MPQF MPQF MPQF MPQF
interpreter 1 interpreter 2 interpreter 3 interpreter 4 interpreter i
relational XML
triple store filesystem ...
database database
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

12. Motivation
Results & future work
A.I.R. Framework
Discussion
Conclusion
MPQF overview
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

13. Motivation
Results & future work
A.I.R. Framework
Discussion
Conclusion
JPSearch overview
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

14. Motivation
Results & future work
A.I.R. Framework
Discussion
Conclusion
JPSearch transformation rules
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval

15. Motivation
Results & future work
A.I.R. Framework
Discussion
Conclusion
W3C Media Annotations Working Group
F. Stegmaier, M. D¨ller, H. Kosch, A. Hutter and T. Riegel
o WIAMIS2010 – AIR: Architecture for Interoperable Retrieval