The second session looks at how using Linked Data principles for media fragment annotation publication and retrieval (Linked Media) can enable online media fragment re-use: Introducing the Linked Media principles Publishing Linked Media using dedicated multimedia RDF repositories Retrieval of media resources that illustrate linked data concepts Using the Linked Data graph to find relevant links between distinct media assets (examples with SPARQL) Retrieval of links between annotated media to enable topical browsing (using the TVEnricher service) Examples of Linked Media at scale: VideoLyzard and HyperTED

1. 1
LinkedMedia:
An approach to online media re-use
Lyndon Nixon
MODUL University Vienna
lyndon.nixon@modul.ac.at
RE-USING MEDIA ON THE (SEMANTIC) WEB
ISWC2014 Tutorial, Riva de Garda, Italy, October 20 2014

2. 20.10.14 Slide 2 of 50
Agenda
• Session 1: Media fragment specification and semantics
• Summary: Introduce the W3C Media Fragment URI specification and the Open
Annotation model. Highlight how media fragments can be annotated using NER
tools.
• Session 2: Linked Media principles
• Summary: Introduce the Linked Media principles. How to
publish Linked Media in RDF and how to retrieve media
enrichments. Illustration with Linked Media applications.
• Session 3: User experience driven design of Linked Media applications
• Summary: Present the Web and TV convergence. Describe LinkedTV experience
via two innovative applications.
2

3. 2/3: LinkedMedia
• Introducing the Linked Media principles
• Publishing Linked Media
• LinkedTV Platform (Virtuoso)
• Retrieval of Linked Media
• Enrichment of media (LinkedTV)
• Browsing and linking media
(VideoLyzard, HyperTED)
3
20.10.14 Slide 3 of 50

4. Linked Media
20.10.14 Slide 4 of 50

5. Why is Online Media important?
„It is growing at more than 20% per annum, fuelled by increased
demands for new programming and the huge saving it represents
compared with shooting new footage. Interactive technology and the
Internet will further contribute to the growth of the market as it makes
stock footage cheaper and easier to locate and license.“
- http://moneyam.uk-wire.com/cgi-bin/articles/200201020827103514P.html
20.10.14 Slide 5 of 50

6. Why Semantic Media?
20.10.14 Slide 6 of 50

7. 20.10.14 Slide 7 of 50
Why Linked Media?

8. 20.10.14 Slide 8 of 50
Why Linked Media?

9. 20.10.14 Slide 9 of 50
Why Linked Media?

10. 20.10.14 Slide 10 of 50
Why Linked Media?
From Lyndon Nixon, „The importance of Linked Media to the Future Web“
slideshare.net/linkedtv/www-linked-media-keynote

11. Linked Media Principles
1. Web media descriptions
need a common
representation of media
structure
22.. WWeebb mmeeddiiaa ddeessccrriippttiioonnss
nneeeedd aa ccoommmmoonn
rreepprreesseennttaattiioonn ooff mmeeddiiaa
ccoonntteenntt
33.. WWeebb mmeeddiiaa ddeessccrriippttiioonnss nneeeedd ttoo uussee aa
mmeeddiiaa oonnttoollooggyy wwhhiicchh ssuuppppoorrttss ddeessccrriippttiioonn
ooff bbootthh tthhee ssttrruuccttuurree aanndd ccoonntteenntt ooff mmeeddiiaa
20.10.14 Slide 11 of 50

12. Linked Media Principles
44.. TThhee ddeessccrriippttiioonnss ooff
mmeeddiiaa iinn tteerrmmss ooff
ccoommmmoonn
rreepprreesseennttaattiioonnss ooff
ssttrruuccttuurree aanndd ccoonntteenntt
aarree tthhee bbaassiiss ffoorr ddeerriivviinngg
lliinnkkss aaccrroossss mmeeddiiaa oonn
tthhee WWeebb ((LLiinnkkeedd MMeeddiiaa))
20.10.14 Slide 12 of 50

13. 20.10.14 Slide 13 of 50
Linking Media
Prime Minister
South Africa President
Mozambique

14. Publishing
Linked Media
20.10.14 Slide 14 of 50

15. Linked Media is Linked Data
Good news: Linked Media can be published as Linked
Data!
•The media resource has a globally (Web wide) unique
identifier
•Metadata about the media resource can be accessed via
its identifier
•Media identifier can‘t be = media locator!
20.10.14 Slide 15 of 50

16. 20.10.14 Slide 16 of 50
LinkedTV Platform
Web administration interface & REST API at
http://api.linkedtv.eu

17. Media Resource view
20.10.14 Slide 17 of 50

18. Media Metadata view
http://data.linkedtv.eu is the base of the RDF graph of
LinkedTV content, and via Virtuoso, every LinkedTV
instance URI can return HTML or RDF, e.g.
http://data.linkedtv.eu/mediaresource/8a8187f2-3fc8-cb54-
0140-7dccd76f0001
20.10.14 Slide 18 of 50

19. Media Metadata view
Media fragments of a media resource:
http://data.linkedtv.eu/mediaresource/8a8187f2-3fc8-cb54-
0140-7dccd76f0001/mediafragment
20.10.14 Slide 19 of 50

20. Media Metadata view
Annotations of a media fragment:
http://data.linkedtv.eu/mediafragment/3d5b11e0-f6df-11e3-
b0fe-005056a7235c%23t=1,299/annotation
20.10.14 Slide 20 of 50

21. Linked Media
retrieval
20.10.14 Slide 21 of 50

22. 20.10.14 Slide 22 of 50
SPARQL endpoint
SPARQL queries allow us to connect metadata across the
Linked Media store AND to connect it to other metadata
outside in the Linked Data cloud.
Linked Media queries courtesy LinkedTV deliverable 2.4
„Annotation and retrieval module of media fragments“ (Jose
Luis Redondo Garcia & Raphael Troncy) available from:
http://de.slideshare.net/linkedtv/annotation-and-retrieval-module-
of-media-fragments

23. 20.10.14 Slide 23 of 50
SPARQL queries
Get all Shots or Chapters for a media resource.

24. 20.10.14 Slide 24 of 50
SPARQL queries
Return all entities of type nerd:Person in the annotation of a media resource.

25. 20.10.14 Slide 25 of 50
SPARQL queries
Return all entities occurring within a temporal boundary of a media resource.

26. 20.10.14 Slide 26 of 50
LinkedTV

27. Why Linked Television?
http://www.linkedtv.eu
20.10.14 Slide 27 of 50
40% of TV viewers are using a
companion device alongside the TV
program.*
* J. Abreu, P. Almeida, B. Teles, and M. Reis. Viewer behaviors and practices
in the (new) television environment. In Proceedings of the 11th European
Conference on Interactive TV and Video, EuroITV '13.
Ever saw
something on TV
and wanted to know
more about it, but
didn‘t even know
how to search for
it?

28. LinkedTV Technology
Paintings by Jan Sluijters
Selection of
related concepts
Selection of
related content
20.10.14 Slide 28 of 50
Video object and
word detection
Connection to
concepts
“...schilderij van Jan Sluijters....”
dbpedia.org/resource/Jan_Sluyters
Presentation
engine

29. The use of Linked Data in the identification
of concepts in LinkedTV means we can
expand concepts along different facets, i.e.
allow users to explore in terms of their
different interests in a given concept.
For example, for an artist like Jan Sluijters,
has art style
Leo Gestel
20.10.14 Slide 29 of 50
Related concepts
 Additional information, e.g.
 biography of artist
 style of painting
 Related information, e.g.
 artists from same
period
 paintings in similar style
 related styles
Paintings by Jan Sluijters
Expansion of
related concepts
Jan Sluijters
luminism
has art style
Piet Mondriaan
has art style
LinkedTV can link into:

30. 20.10.14 Slide 30 of 50
Related content
LinkedTV provides enrichment services
which provide recommendations for
related online content (Web pages,
images, audio, video) pertaining to the
concepts in the TV program:
– Fresh Social Web content coming
from e.g. Twitter and Facebook
– User Generated content coming
from e.g. Flickr and YouTube
– Whitelist content coming from
partner Websites like public
broadcasters in Germany or cultural
heritage archives in the Netherlands
– Extendable and configurable by
source and media type
… and their digital images
Linking to
related content

31. LinkedTV „enrichment“
Base enrichments: title, thumbnail (poster), description (abstract)
Information cards: set of properties and values according to the entity type
Linksets: links to online content determined by queries
over Web content sources using a group of entities as
search term
•Linksets can be split along enrichment dimensions
•Dimensions are distinct aspects of interest to viewers
•They map to different queries & services in LinkedTV
20.10.14 Slide 31 of 50

32. LinkedCulture enrichment
TKK Video
Chapter segmentation + tagging as „Art Object“
TV2RDF incl. Art Object annotation
Entity Proxy
20.10.14 Slide 32 of 50
Related
TKK
chapters
(Solr)
Related
art objects
(Europeana)
Related
White List
media
(IRAPI)
TVEnricher (TKK configuration)
Editor Tool LinkedTV Player

33. Art objects in TKK episodes
20.10.14 Slide 33 of 50

34. Art object: semantic model
20.10.14 Slide 34 of 50
http://data.linke
dtv.eu/object/a
vro/8a8187f2-
3fc8-cb54-
0140-
7dccd76f0001/
2138
A silver tea jar
RDF Is-a Container http://vocab.getty.ed
u/aat/300045611
CRM Consists-of Silver http://vocab.getty.ed
u/aat/300010975
VRA
locationCreationSite
Friesland http://www.geoname
s.org/2755812
DCT temporal Start: 1690, End: 1742

35. Mapping to Europeana API
what:
(container+OR+houder+OR+bak+OR+ta
nk+OR+blik)
proxy_dc_format:zilver
where:Friesland
20.10.14 Slide 35 of 50
http://data.linke
dtv.eu/object/a
vro/8a8187f2-
3fc8-cb54-
0140-
7dccd76f0001/
2138
A silver tea jar
RDF Is-a Container
CRM Consists-of Silver
VRA
Friesland
locationCreationSite
DCT temporal Start: 1690, End: 1742
YEAR:[1690+TO+1742]

36. 20.10.14 Slide 36 of 50
Enrichment results
Friesian silver from
1690 to 1742 to 1742

37. LinkedNews enrichment
RBB Video
Chapter segmentation + tagging as „News Item“
Named Entity Expansion over news items
Entity Proxy
20.10.14 Slide 37 of 50
Related
news
items
(Solr)
Related
news articles
(TVNews-Enricher)
Related
White List
media
(IRAPI)
TVEnricher (RBB configuration)
Editor Tool LinkedTV Player

38. 20.10.14 Slide 38 of 50
Entity Proxy
Fill information cards for entities giving values
for their most relevant properties.

39. 20.10.14 Slide 39 of 50
TVNewsEnricher
Returns news media sources related to given
entity sets, powered by Google CSE

40. IRAPI: Web media crawler
Extract descriptions of media on websites so that
related media can be found
Video
Title
Description
Entities, e.g.
http://dbpedia.org/
resource/Edward_
Snowden
20.10.14 Slide 40 of 50

41. 20.10.14 Slide 41 of 50
VideoLyzard

42. Climate Change
Portal
20.10.14 Slide 42 of 50
42

43. http://link.weblyzard.com/video-showcase
Video
fragments
in search
results Video
20.10.14 Slide 43 of 50
VideoLyzard
43
fragment
playback

44. 20.10.14 Slide 44 of 50
VideoLyzard

45. 20.10.14 Slide 45 of 50
VideoLyzard
More Negative More Positive

46. 20.10.14 Slide 46 of 50
HyperTED
Content courtesy EURECOM (José Luis Redondo García,
Raphael Troncy, Mariella Sabatino & Pasquale Lisena)

47. 20.10.14 Slide 47 of 50
198
4
.com
200
6
CHAPTERS
201
4
HOT SPOTS
ENTITIES
RELATED TED’S
CHAPTERS
THE
MYSTERIOUS
FIELD OF
ENGINEERING
SYSTEMS
UNDERSTANDI
NG
ENVIRONMEN
T:
A SYSTEM
APPROACH
SYSTEMS
PRACTICE:
MANAGING
SUSTAINABIL
ITY
COURSES

48. Hotspots
Cluster video chapters which share similar topics
20.10.14 Slide 48 of 50

49. 20.10.14 Slide 49 of 50
Architecture
49

50. Demo
http://linkedtv.eurecom.fr/HyperTED
20.10.14 Slide 50 of 50

51. MediaMixer
community portal
Introduction to all technologies at
community.mediamixer.eu/technology
Updated with latest materials on all
Media Mixer topics:
Technology use cases
Demonstrators
Tutorials
Presentations
Software
Specifications
http://community.mediamixer.eu
20.10.14 Slide 51 of 50

52. VideoLecturesMashup
20.10.14 Slide 52 of 50

53. MediaMixer use case:
VideoLecturesMashup
20.10.14 Slide 53 of 50

54. Video fragment creation
Fragments were created
based on the slide
synchronisation timeline.
Transcripts (auto-generated
by speech-to-text
technology where
necessary) were parsed and
… there are three split across fragments.
Kingdoms of Life,
Bacteria, Archaea
and Eukaryota...
20.10.14 Slide 54 of 50

55. Video fragment annotation
Fragments were then annotated by
extracting topics from their textual
metadata (slide OCR or speaker
transcription).
Topics are connected to a global
knowledge model (DBPedia).
20.10.14 Slide 55 of 50
Video Fragment
(4:41-5:12)
Archaea

56. 20.10.14 Slide 56 of 50
Video fragment
management
Annotations are managed in
a separate metadata store.
The store provides a
semantic query endpoint
returning lists of video
fragments matching a query
topic (including semantically
related topics)
Acidiplasma „type“ relation
Archaea
Video
Fragment
(4:41-5:12)

57. Video fragment playback
The front end uses HTML5
or Flash. Both codebases
are extended to support
video fragment playout.
Individual playback can be
modified to linear or non-linear
channels (for e.g. a
TV or mobile video
experience)
20.10.14 Slide 57 of 50

58. VideoLecturesMashup - demo
20.10.14 Slide 58 of 50
http://mediamixer.videolectures.net

59. 20.10.14 Slide 59 of 50

60. 20.10.14 Slide 60 of 50

61. 20.10.14 Slide 61 of 50

62. 20.10.14 Slide 62 of 50

63. 20.10.14 Slide 63 of 50
Reference

64. SPARQL RDF query
language
RDF data is a „labeled, directed graph“
without any fixed vocabulary (this is defined
in RDF Schema). A query language for RDF
needs to support this data model:
•Tranversing paths in a RDF graph
•Being independent of any defined
vocabulary
•Able to query over the data or the schema
SPARQL („sparkle“) is a W3C standard
supported in most RDF tools
•SELECT... WHERE.... constructions like in
SQL
•Path expressions
•Additional keywords for more query
expressiveness
20.10.14 Slide 64 of 50

65. Examples: triple patterns
The basic idea in SPARQL is to match graph patterns against RDF graphs.To
understand graph patterns we must first define triple patterns:
• A triple pattern is similar to a triple (in RDF) but with one or more variables in
the place of a RDF resource (URI or literal)
Triple: dbpedia:Lou_Reed foaf:givenName „Lewis Allen Reed“ .
Triple pattern: dbpedia:Lou_Reed foaf:givenName ?name .
?name is the variable. If a RDF graph with the above triple is queried with the
below triple pattern, then in the SPARQL results the variable ?name would be
‚bound‘ to the value „Lewis Allen Reed“.
• A SPARQL query result is a set of bindings for the variables appearing in the
SELECT clause, based on matching RDF resources to variables in triple
patterns in the WHERE clause. .
20.10.14 Slide 65 of 50

66. Examples: conjunctive and
disjunctive query
In graph patterns, several triple patterns are listed within braces { ... } and
these are interpreted conjunctively:
Ex: { ?what tech:noOfWheels „4“ .
?what tech:minSpeed „180“ . }
The variable ?what will only be bound to resources which BOTH have 4 wheels
AND a minimum speed of 180.
You can also join results from distinct graph patterns using the UNION
keyword. Note that result sets from graph patterns and from UNIONs are
different, since UNION works disjunctively:
Ex: { ?what tech:noOfWheels „4“ . }
UNION { ?what tech:minSpeed „180“ . }
20.10.14 Slide 66 of 50

67. Examples: FILTER, ORDER
BY
SPARQL has many other keywords. FILTER restricts variable bindings
returned in query results to those for which the filter expression evaluates to
true. A filter applies to solutions over the entire graph pattern it is contained in.
Ex: { ?what tech:noOfWheels „4“ .
?what tech:minSpeed ?speed .
FILTER ( ?speed > 170 ) }
The ORDER BY keyword determines the sequence of query results returned
according to a sort on the referenced variable‘s bindings, ascending by default:
Ex: { ?what tech:minSpeed ?speed . }
20.10.14 Slide 67 of 50
ORDER BY ?speed
Or ORDER BY DESC(?speed)

68. Query for media fragments:
SPARQL-MM
Research work in progress: extending SPARQL to Media Fragments by adding
spatio-temporal filter and aggregration functions.
Relation
Function
Aggregation
Function
Spatial mm:rightBeside mm:spatialIntersection
mm:spatialOverlaps mm:spatialBoundingBo
x
… …
Temporal mm:after mm:temporalIntersecti
on
mm:temoralOverlaps mm:temporalIntermedi
ate
… …
Courtesy Thomas Kurz (Salzburg Research) & MICO EU project
http://demos.mico-project.eu/sparql-mm/sparql-mm/demo/index.html
Combined mm:overlaps mm:boundingBox
mm:contains mm:intersection
20.10.14 Slide 68 of 50