With the tremendous increase in multimedia content on the Web and in corporate intranets, discovering hidden meaning in raw multimedia is becoming one of the biggest challenges. Analysing multimedia content is still in its infancy, requires expert knowledge, and the few available products are associated with excessive price tags, while still not delivering sufficient quality for many tasks. This makes it hard, especially for small and medium-size enterprises, to make use of this technology. Ina addition analysis components typically operate in isolation and do not consider the context (e.g. embedding text) of a media resource. This paper presents how MICO tries to address these problems by providing an Open Source service platform, that allows to analyse media in context and includes various analysis engines for video, images, audio, text, link structure and metadata.

1. 3rd International Workshop on Linked Media, WWW 2015, Florence
Sergio Fernández, Sebastian Schaffert, Thomas Kurz
MICO — Towards Contextual Media Analysis

2. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
The MICO project

3. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
The MICO project idea

4. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
The MICO project idea

5. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
The MICO project idea
● provide an architecture to analyze media in context by
orchestrating various analysis components for several
media types
● provide all necessary technologies for a distributed
analysis workflow: extractors, orchestration, schema,
querying
● contribute the core software results as components
under business-friendly OSS licenses (building on
projects such as Apache Marmotta), but allowing
inclusion of both OSS and closed-source extractors to
simplify the use of the technology in industrial products.

6. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Requirements in short
■ large content items (video, audio), located on different
storage systems
■ composite content items, consisting of a collection of
related media objects (e.g. website with video and images)
■ independent analysis components, each contributing a
small part to the overall analysis result
o Java and C/C++ components, as analysis services are
typically implemented in the language that best serves
the job
o user feedback, where users might be involved in the
analysis process as “crowd workers” still to be defined
■ metadata stored in RDF following a cross-media model
o external applications can further process the metadata
o querying of metadata, content and context in a unified
way using SPARQL

7. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Architecture (High Level)

8. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Components overview

9. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Platform Prototype
● collection of UNIX services (mostly Java and
C++ analysers, FTP server, RabbitMQ
Marmotta, PostgreSQL, etc)
● analysis services can be either started on
the same machine or connect remotely
● distribution as Maven artifacts, VirtualBox
image and Debian Linux packages
● available as open source
http://mico-project.bitbucket.org

10. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Get MICO running
1. Setup Debian
Jessie
1. Add MICO
Repository
1. Install MICO
Platform
$ sudo add-apt-repository 'deb
http://apt.mico-project.eu/ mico main
contrib non-free'
$ wget -O- http://apt.mico-project.eu/apt-
repo.key | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install mico-platform
https://bitbucket.org/mico-project/platform#markdown-header-debian-repository

11. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
DEMO

12. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Using the API: Java
final EventManager eventManager = new EventManagerImpl("192.168.56.101");
eventManager.init();
final PersistenceService persistenceService =
eventManager.getPersistenceService();
final ContentItem ci = persistenceService.createContentItem();
final Content cp = ci.createContentPart();
cp.setType("text/plain");
IOUtils.copyLarge(new FileInputStream("example.txt"),
cp.getOutputStream());
eventManager.injectContentItem(ci);

13. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Using the API: C++
EventManager eventManager(mico_host, mico_user, mico_pass);
ContentItem* item =
eventManager.getPersistenceService().createContentItem();
Content* c = item->createContentPart();
c->setType("text/plain");
int fd = open("example.txt", O_RDONLY);
size_t len = st.st_size;
char* buffer = (char*)mmap(NULL, len, PROT_READ, MAP_SHARED, fd, 0);
std::ostream* os = c->getOutputStream();
os->write(buffer, len);
eventManager.injectContentItem(*item);
https://bitbucket.org/mico-project/platform/src/master/api/c++/tools/mico_inject.cpp

14. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Future Plans
● Refine messaging API
● Model API based on Open Annotation
Ontology
● SPARQL-MM extensions
● Specialized Data APIs
● and Stabilize
Next Release: May 22 2015

15. © MICO Project. Funded by the European Commission FP7 (grant agreement no: 610480). No reproduction without written permission.
Thanks for your attention!
Sergio Fernández, Sebastian Schaffert, Thomas Kurz
thomas.kurz@salzburgresearch.at
Knowledge and Media Technologies
Salzburg Research
http://www.salzburgresearch.at