A number of novel application ideas will be introduced based on the media fragment creation, specification and rights management technologies. Semantic search and retrieval allows us to organize sets of fragments by topical or conceptual relevance. These fragment sets can then be played out in a non-linear fashion to create a new media re-mix. We look at a server-client implementation supporting Media Fragments, before allowing the participants to take the sets of media they have selected and create their own re-mix.

1. 1
Media fragment
re-mixing and playout
Lyndon Nixon
MODUL University Vienna
lyndon.nixon@modul.ac.at
RE-USING MEDIA ON THE WEB
WWW2014 Tutorial, Seoul, S Korea, April 8 2014

2. 06.01.14 Slide 2 of 32
3. Media fragment re-mixing and
playout
• Storing the data: repositories
• Querying the data: retrieval
• Re-mixing the media: semantics
• Play out the media: clients
2

3. 06.01.14 Slide 3 of 32
Repositories

4. 06.01.14 Slide 4 of 32
Storing your RDF data
Like any data, RDF needs to be stored. It can be serialised to different formats
like XML and JSON, but this is not the same as the underlying data model.
• RDF is a GRAPH, XML and JSON are TREEs -> the same RDF data can be
serialised in different XML and JSON syntax
• RDF datatypes have different semantics than XML and JSON datatypes
In other words: if you want to persist and manipulate your data as RDF, you
should use a RDF specific solution.

5. 06.01.14 Slide 5 of 32
RDF Stores
A RDF Store (or RDF repository, or triple store) is a database specialized for
RDF triples.
•Can ingest RDF in different serialisations
•Supports a RDF specific query language such as SPARQL
•May support RDF insert/delete (a.k.a. SPARQL UPDATE)
•Can support inferencing over RDF data
Different implementations, can be broadly categorized into:
•In-memory: Triples are stored in main memory (usually fastest)
•Native: Persistence is provided by using a custom DB, where the database
structure is optimised to the RDF model (most commonly used)
•Non-native: Persistence is provided by using a third party relational DBMS
(e.g. mySQL or postgres) (may be better if there are many updates or a need
for good concurrency control)

6. 06.01.14 Slide 6 of 32
RDF Store Overview
Good overview at
http://www.w3.org/
2001/sw/wiki/Categ
ory:Triple_Store
(47 listed stores)
Scalability:
http://www.w3.org/
wiki/LargeTripleSto
res lists quoted
large scale
deployments of
stores (up to 1+
trillion RDF triples)
Comparison from 9/2012 at http://www.garshol.priv.no/blog/231.html

7. 06.01.14 Slide 7 of 32
OpenRDF Sesame
Sesame is an open source RDF framework
for RDF storage, query and inference.
•http://www.openrdf.org
•Implemented in Java
•Supports in-memory, native and non-native
stores.
Features
•Lightweight yet powerful Java API
•SPARQL query language support (including
SPARQL UPDATE)
•Highly scalable
•Can support inference via a RDF Schema
reasoner
•Transactionality
•Context (a.k.a. Named Graphs)
RDF Model
Rio
SAIL API
SAIL Query Model
SeRQL SPARQL
Repository Access API
HTTP Serverapplication
application
HTTP / SPARQL protocol

8. 06.01.14 Slide 8 of 32
Insert RDF into Sesame
The central concept in Sesame is the repository. You add and query data by
repository.

9. 06.01.14 Slide 9 of 32
Using context
You can distinguish data sets in
one repository by using context
This is an additional URI attached to
each triple you insert. (thus also
referred to as quads)
You can now query or update only on
triples within one named context (in
RDF, this is called a named graph)
Context is useful for provenance
tracking and for versioning
If a query does not specify a context,
the default context is used. It can be
exclusive (only triples without a
context) or inclusive (all triples in all
contexts or none)

10. 06.01.14 Slide 10 of 32
Browse RDF in Sesame

11. 06.01.14 Slide 11 of 32
Retrieval

12. 06.01.14 Slide 12 of 32
SPARQL RDF query language
RDF data is a „labeled, directed graph“
without any fixed vocabulary (this is defined
in RDF Schema). A query language for RDF
needs to support this data model:
•Tranversing paths in a RDF graph
•Being independent of any defined
vocabulary
•Able to query over the data or the schema
SPARQL („sparkle“) is a W3C standard
supported in most RDF tools
•SELECT... WHERE.... constructions like in
SQL
•Path expressions
•Additional keywords for more query
expressiveness

13. 06.01.14 Slide 13 of 32
Examples: triple patterns
The basic idea in SPARQL is to match graph patterns against RDF graphs.To
understand graph patterns we must first define triple patterns:
• A triple pattern is similar to a triple (in RDF) but with one or more variables in
the place of a RDF resource (URI or literal)
Triple: dbpedia:Lou_Reed foaf:givenName „Lewis Allen Reed“ .
Triple pattern: dbpedia:Lou_Reed foaf:givenName ?name .
?name is the variable. If a RDF graph with the above triple is queried with the
below triple pattern, then in the SPARQL results the variable ?name would be
‚bound‘ to the value „Lewis Allen Reed“.
• A SPARQL query result is a set of bindings for the variables appearing in the
SELECT clause, based on matching RDF resources to variables in triple
patterns in the WHERE clause. .

14. 06.01.14 Slide 14 of 32
Examples: conjunctive and
disjunctive query
In graph patterns, several triple patterns are listed within braces { ... } and
these are interpreted conjunctively:
Ex: { ?what tech:noOfWheels „4“ .
?what tech:minSpeed „180“ . }
The variable ?what will only be bound to resources which BOTH have 4 wheels
AND a minimum speed of 180.
You can also join results from distinct graph patterns using the UNION
keyword. Note that result sets from graph patterns and from UNIONs are
different, since UNION works disjunctively:
Ex: { ?what tech:noOfWheels „4“ . }
UNION { ?what tech:minSpeed „180“ . }

15. 06.01.14 Slide 15 of 32
Examples: FILTER, ORDER BY
SPARQL has many other keywords. FILTER restricts variable bindings returned
in query results to those for which the filter expression evaluates to true. A filter
applies to solutions over the entire graph pattern it is contained in.
Ex: { ?what tech:noOfWheels „4“ .
?what tech:minSpeed ?speed .
FILTER ( ?speed > 170 ) }
The ORDER BY keyword determines the sequence of query results returned
according to a sort on the referenced variable‘s bindings, ascending by default:
Ex: { ?what tech:minSpeed ?speed . }
ORDER BY ?speed
Or ORDER BY DESC(?speed)

16. 06.01.14 Slide 16 of 32
SPARQL in Sesame

17. 06.01.14 Slide 17 of 32
Query for media fragments:
SPARQL-MM
Research work in progress: extending SPARQL to Media Fragments by adding
spatio-temporal filter and aggregration functions.
Relation Function Aggregation Function
Spatial mm:rightBeside mm:spatialIntersection
mm:spatialOverlaps mm:spatialBoundingBox
… …
Temporal mm:after mm:temporalIntersection
mm:temoralOverlaps mm:temporalIntermediate
… …
Combined mm:overlaps mm:boundingBox
mm:contains mm:intersection
Courtesy Thomas Kurz (Salzburg Research) & MICO EU project
http://demos.mico-project.eu/sparql-mm/sparql-mm/demo/index.html

18. 06.01.14 Slide 18 of 32
Semantics

19. 06.01.14 Slide 19 of 32
Semantic search?
Broadly, any search paradigm which makes use of machine understanding of
the data being queried.
Search term „Presidents of the United States“
CLASSICAL SEARCH
Return all media with the
string „Presidents of the
United States“ in their title or
description
SEMANTIC SEARCH
Return all media with an occurance
of an instance of the type „Presidents
of the United States“ in their
metadata, e.g. George W Bush, Bill
Clinton, Barack Obama ...

20. 06.01.14 Slide 20 of 32
Search by types or categories
The set of „Presidents of the United States“
The category of „Presidents of the United States“
World Leaders
 Presidents
 Presidents of the United States
 Bill Clinton
 Barack Obama
 George W Bush
Barack Obama
George W Bush
Bill Clinton

21. 06.01.14 Slide 21 of 32
Search by paths in the graph
RDF can be more expressive through defining properties with specific
semantics; semantic search can be more powerful by using these semantics
However, independent of a RDF vocabulary, shared graph properties can be an
indicator of resource similarity & used in domain-independent semantic search
?
? < 50
England
Age
BornIn
?v
?
?
?p
?p

22. 06.01.14 Slide 22 of 32
Search with Linked Data
Linked Data: RDF metadata published on the Web, with information about each
resource discoverable via its unique URI. Machines can transverse the RDF
graph by looking up each URI in turn.
DBPedia: an extraction of the knowledge in Wikipedia (esp. the info boxes) as
RDF and published in its own namespace (http://dbpedia.org)
Each Wikipedia article of the form http://en.wikipedia.org/wiki/X has a DBPedia
resource at the URI http://dbpedia.org/resource/X
Has its own ontology which defines resource properties and classes, also
maps the Wikipedia categories using the dcterms:subject property.
Advantages: Linked Data URIs allow an unambiguous reference to information
about a concept, that information carries the authority of the data provider, all
applications have the same understanding of the concept
Disadvantages: information at the URI is not under your control, sometimes
messy/dirty (the case with DBPedia)  use local copies, extend with own
metadata

23. 06.01.14 Slide 23 of 32
SPARQL endpoints and the
SERVICE keyword
Linked Data sources usually provide a SPARQL endpoint for their datasets
• A SPARQL endpoint is a SPARQL query processing service that supports the
SPARQL protocol (http://w3.org/TR/rdf-sparql-protocol): a definition of how to
send a SPARQL query to a Web Service and receive a query result.
e.g. DBPedia.org has a SPARQL endpoint at http://dbpedia.org/sparql
See a fuller list at http://esw.w3.org/topic/SparqlEndpoints
SPARQL 1.1 includes the SERVICE keyword to include queries to SPARQL
endpoints within a SPARQL query addressing a different repository.
• With this in Sesame, you can SPARQL query a local repository BUT include
bindings from external datasets within the query
Ex: SERVICE <http://dbpedia.org/sparql>
{ ?resource dcterms:subject . ?subject .
?related dcterms:subject ?subject . }

24. 06.01.14 Slide 24 of 32
SPARQL for semantic search

25. 06.01.14 Slide 25 of 32
Clients

26. 06.01.14 Slide 26 of 32
What is a Media Fragment
Player?
Regular media players playback
full media assets, only the
consumer may seek within the
media.
A media fragment player supports
playback of (spatial and/or
temporal) media fragments in
place of the full media asset, the
customer doesn‘t have to do
anything.
We mean a player which can
correctly interpret a media
fragment as per the W3C Media
Fragment URI specification.

27. 06.01.14 Slide 27 of 32
Media Fragments – Web
browser support
• Mozilla Firefox, since version 9
• Apple Safari, since Jan 2012
• Google Chrome, since Jan 2012
Focus on support for the temporal dimension in the HTML5
video player.

28. 06.01.14 Slide 28 of 32
Media Fragments –
Javascript libraries
• mediafragment.js:
https://github.com/tomayac/Media-Fragments-URI
• xywh.js: https://github.com/tomayac/xywh.js
Focus on implementing the spatial and clock dimensions
within browsers.

29. 06.01.14 Slide 29 of 32
Media Fragments –
SyNote player
http://smfplayer.synote.org/smfplayer/
• Cross-browser/device solution for Media Fragment playback via JavaScript
• Supports Media Fragment playback from online video platforms such as
YouTube and DailyMotion
• Wider video format support than native HTML5 (e.g. also Flash or Windows
Media)

30. 06.01.14 Slide 30 of 32
What about Media
Fragment servers?
The advantage of media fragment playback should be also that ONLY the
content of the media fragment needs to be provided by the server to the client.
(in reality, Web servers are still serving the whole media asset, the fragment is
determined on the client)
Implementing Media Fragment support on the server side requires new
implementations of Web (media) servers... Ninsuma Media Server is a
prototypical demonstration of how a server should function with Media
Fragments.
http://ninsuna.elis.ugent.be/MediaFragmentsServer

31. 06.01.14 Slide 31 of 32
Our Media Fragment Jukebox
http://www.mediamixer.eu/jukebox/ox/

32. 32
Wrap up