The talk was delivered by Dorothea Tsatsou at the workshop at the UMAP 2012 conference, 16 - 20 July 2012, Montreal, Canada. More info: http://bit.ly/Qvnbz4

1. Television Linked To The Web
Dorothea Tsatsou1, Lyndon Nixon2, Matei Mancas3, Miroslav Vacura4, Rüdiger Klein5, Julien
Leroy3, Jaroslav Kuchař4, Tomáš Kliegr4, Manuel Kober5, Maria Loli1, Vasileios Mezaris1
Contextualised user profiling in
networked media environments
1
CERTH-ITI, Thessaloniki, Greece 4
University of Economics Prague, Prague, Czech Republic
2
STI International, Vienna, Austria 5
Fraunhofer IAIS, Sankt Augustin, Bonn, Germany
3
University of Mons, Mons, Belgium
2nd Augmented User Modeling workshop, UMAP 2012, Montreal, July 2012
www.linkedtv.eu

2. A new era to TV viewing
www.linkedtv.eu
60% of Americans engage in couch
potato multitasking
86% of TV viewers with broadband surf
while they watch
Social TV
38% of networked media users
discussing what they're watching on
social media (53% in the 16 to 23-
year-old demographic)
-- Nielsen (2009). Three screen report. Technical report, Nielsen Company
-- Yahoo! and Nielsen (2010). Mobile shopping framework - the role of mobile devices in
the shopping process. Technical report, Yahoo! and The Nielsen Company
http://www.designbynotion.com/metamirror-next-generation-tv
-- Ovum survey, published in TheRegister, 6 October 2011
Information Technologies Institute
2 Centre for Research and Technology
Hellas

3. Towards networked media…
www.linkedtv.eu
Second Screen content push –
breakthrough with HTML5 mobile
Second screen apps show related
content without disturbing the TV view
The rise of Smart TVs
LG SmartTV, pic courtesy
http://www.wired.com/gadgetlab/2011/01/lg-smart-tv/
Information Technologies Institute
3 Centre for Research and Technology
Hellas

4. …to LinkedTV
www.linkedtv.eu
TV and Web access are
unified on the TV device
but not the experience
Interweaving TV and Web
content into a single
experience a focus of the
LinkedTV project
Information Technologies Institute
4 Centre for Research and Technology
Hellas

5. LinkedTV ― Television Linked To the Web
www.linkedtv.eu
Vision: 12 Excellent Partners
ubiquitously online cloud of Fraunhofer Eurecom
Networked Audio-Visual Content STI GMBH Condat
decoupled from place, device or CERTH BEELD EN GELUID
source UEP Noterik
UMONS U. ST GALLEN
 Aim: CWI RBB
provide interactive multimedia service
for non-professional end-users
focus television broadcast content as
seed videos
 Web: http://www.linkedtv.eu
Information Technologies Institute
5 Centre for Research and Technology
Hellas

6. LinkedTV Workflow
www.linkedtv.eu
Overall Architecture
Use Case Scenarios
Intelligent Video Analysis
Linking Hypervideo to Web Content
Contextualization
and Personalization
Interface and Presentation Engine
Information Technologies Institute
6 Centre for Research and Technology
Hellas

7. LinkedTV Workflow
www.linkedtv.eu
Interactive News Show
Overall Architecture
Hyperlinked Documentary Use Case Scenarios
Media Arts
Intelligent Video Analysis
Linking Hypervideo to Web Content
Contextualization
and Personalization
Interface and Presentation Engine
Information Technologies Institute
7 Centre for Research and Technology
Hellas

8. LinkedTV Workflow
www.linkedtv.eu
Overall Architecture
Use Case Scenarios
Intelligent Video Analysis
Cubism
Linking Hypervideo to Web Content
Contextualization
Fauvism and Personalization
Interface and Presentation Engine
Expressionism
Information Technologies Institute
8 Centre for Research and Technology
Hellas

9. LinkedTV Workflow
www.linkedtv.eu
Overall Architecture
Cubism
Use Case Scenarios
Fauvism
Expressionism
Intelligent Video Analysis
Linking Hypervideo to Web Content
Contextualization
and Personalization
Interface and Presentation Engine
CONTENT
ENRICHMENT
Information Technologies Institute
9 Centre for Research and Technology
Hellas

10. LinkedTV Workflow
www.linkedtv.eu
Overall Architecture
Use Case Scenarios
Intelligent Video Analysis
Linking Hypervideo to Web Content
Contextualization
and Personalization
Interface and Presentation Engine
Information Technologies Institute
10 Centre for Research and Technology
Hellas

11. Challenge
www.linkedtv.eu
Digital information overload
Digital information heterogeneity
Information Technologies Institute
11 Centre for Research and Technology
Hellas

12. Transactional user modeling – Information sources
www.linkedtv.eu
Explicit user preferences
Demographics  Strereotypes
Direct definition of pre-determined concepts/categories/keywords
Content (media, external web resources) consumption history
Data/text mining
Ratings
Actions on player/browser
Social interaction
Comments, likes
Peer-to-peer similarities
Semantic knowledge bases (ontologies)
Information Technologies Institute
12 Centre for Research and Technology
Hellas

13. Information sources pros & cons
www.linkedtv.eu
Explicit user preferences Social interaction
+ Accurate + Intuitive (unforeseen preferences)
- Outdated, intrusive - Cold start, scalability, data sparcity
Content consumption history Semantic knowledge bases (ontologies)
+ Straightforward, indicative + Information from the get-go
- Video, audio, articles, wikis, tags, rss + Uniform, finite vocabulary
feeds…: too diverse and non-
- Manual creation, lack of mappings
uniformly characterized
- Cold start, scalability, data sparcity
Solution: Hybrid
Information Technologies Institute
13 Centre for Research and Technology
Hellas

14. Context indicators based on transactions
www.linkedtv.eu
Time
Time of day, season etc
Location
At home, at work, out of the country etc
Actions on the player/browser
Play, stop, rewind, skip, bookmark, scroll
Recent content consumptions
Goal:
Recognize persistent preference patterns for certain contexts
Information Technologies Institute
14 Centre for Research and Technology
Hellas

15. Learning from transactions
www.linkedtv.eu
Nature of transaction
Positive, negative
Preference impact
Weighted preferences
User behaviour pattern discovery
Association rules
Clustering
Utility functions
Information Technologies Institute
15 Centre for Research and Technology
Hellas

16. View the viewers
www.linkedtv.eu
 TV with integrated cameras are on the way (Samsung, Phillips, ...)
 Being viewed is well accepted at homes
 Kinect  Xbox
 SoftKinetic Orange TV
Information Technologies Institute
16 Centre for Research and Technology
Hellas

17. Features that can be extracted
www.linkedtv.eu
Location: determine the environment close to the TV.
Identity and (physical) context: a person can be recognized.
Orientation: gaze-based focus of attention
Distance and static features: the distance of the users relative to the
TV
Motion and dynamic features: behavioural changes
Information Technologies Institute
17 Centre for Research and Technology
Hellas

18. Feature extraction
www.linkedtv.eu
Location-based features depending on the viewer position, different
interfaces are displayed
Greenberg, S., Marquardt, N., et al. Proxemic interactions:the new ubicomp? interactions
18, ACM (2011), 42–50.
Information Technologies Institute
18 Centre for Research and Technology
Hellas

19. Feature extraction
www.linkedtv.eu
Orientation (left), identity and context (right)
Information Technologies Institute
19 Centre for Research and Technology
Hellas

20. Feature extraction
www.linkedtv.eu
Distance features (with TV and other people)
Information Technologies Institute
20 Centre for Research and Technology
Hellas

21. Feature extraction
www.linkedtv.eu
Dynamic features (sudden changes in position, orientation ...)
Information Technologies Institute
21 Centre for Research and Technology
Hellas

22. Holistic user modeling
www.linkedtv.eu
What information do we care about?
All: transactional, behavioural, social
The more you know about the user, the better: implicit data
How to unify diverse implicit information?
One uniform vocabulary and conceptualization about the world
Lightweight, finite, expressive knowledge
Solution:
Semantic user modeling
Information Technologies Institute
22 Centre for Research and Technology
Hellas

23. Semantic User Profiling
www.linkedtv.eu
Explicit User Information
User-defined preferences
Stereotypes
Demographics
User User Profile:
Requirements Update Structure
Implicit User Information &
Formalize
Weighted concepts
Extract Learn Understand (preferences), quantified
by ontology properties
and relations between
them (conjunction,
disjunction, negation)
Tracking Ontologies
Information Technologies Institute
23 Centre for Research and Technology
Hellas

24. Requirements
www.linkedtv.eu
Ontology: formal
Knowledge base (not only the vocabulary)
Describe the “world”
Reflect the user
E.g. actorX playsIn movieY, pollution isRelatedTo environment, anchorman is-a person,
attention(low) → disinterest etc
Compromise between coverage and expressivity
Semantic content and actions classification
Map raw data to ontology
Expressive representation schema
Suitable for logical inferencing
Information Technologies Institute
24 Centre for Research and Technology
Hellas

25. Understanding content
www.linkedtv.eu
Multimedia content External content
Direct mappings Textual analysis for mappings discovery (classification)
(GATE, SProUT, OpenCalais)
Linked
Social Web Activities
Media
Layer
Ontology
Common vocabulary
Classified consumption behaviour
{c∙w, a:c∙w, …..}
Information Technologies Institute
25 Centre for Research and Technology
Hellas

26. Understanding the relation of the content to the user
www.linkedtv.eu
Interests vs Disinterests
Click behaviour (actions on player, dwell time)
Physical reaction
Weight: impact of the concept to the user
Based just on the importance of a concept in the annotated/classified content (scene)
User engagement
Click behaviour (actions on player, dwell time)
Physical state recognition (attention, mood)
Social interaction (likes, shares etc)
Profile concept weights updated:
Upon every transaction (content consumption), based on frequency
Over time (instantly every minute): Time decay
Information Technologies Institute
26 Centre for Research and Technology
Hellas

27. Structure and advantages of the profile
www.linkedtv.eu
Profile ← (¬ ConceptA(X) ∙ w1 ∧ ConceptB(a) ∙ w2)
∨ ∃relationA.ConceptC(b) ∙ w3
Lightweight & uniform user model
Storage even in limited resource devices, scalability
Able to represent and take advantage of more complex knowledge
Relations of interests/knowledge concepts
Constructors and rules (conjunction, disjunction, disjointness)
Richer semantics (inverse, transitive properties, complements)
Especially useful to represent/discern disinterests
Can easily be used with reduced semantics for simple inferencing methods (spreading activation,
clustering etc)
Easily breakable to contextual instances
Information Technologies Institute
27 Centre for Research and Technology
Hellas

28. Contextualization Workflow
www.linkedtv.eu
Information Technologies Institute
28 Centre for Research and Technology
Hellas

29. The LinkedTV personalisation & contextualisation
approach www.linkedtv.eu
Extract, understand & structure in a machine-understandable way
user preferences in regard to context
Understand what the data means (to the world & to the user)
Augment preferences with additional related information
User information manifested in a machine-understandable format
Learn impact and relative patterns of preferences
Harvest mass intelligence
Determine the reactional and physical state of the user
Understand and determine contextual variations of the user profile
Provide a conceptual user profile able to be used for semantic
inferencing
Information Technologies Institute
29 Centre for Research and Technology
Hellas

30. One step further
www.linkedtv.eu
Knowledge pulling
Alleviate information overload in the inferencing stage by reducing the knowledge
based on user context
Ontology learning
Determine new or group-specific knowledge
Privacy preservation
Minimize client-server communication
Anonymization & encryption techniques
Information Technologies Institute
30 Centre for Research and Technology
Hellas

31. www.linkedtv.eu
Questions ?
More information:
dorothea@iti.gr
www.linkedtv.eu
@linkedtv
Information Technologies Institute
31 Centre for Research and Technology
Hellas