1. Affect in recommender systems
Marko Tkalčič
University of Ljubljana
Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
2. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Presentation overview
I: LDOS presentation & Motivation
II: What are emotions?
III: Emotion in recsys – related work
IV: Role of emotions in the MM consumption chain
V: Affect in the decision-making stage
Conclusions
Note: some material is not ours ... Fair use ...
3. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Part I: LDOS group at UL FE and underlying assumption
4. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LDOS group at UL FE
University of Ljubljana
– Faculty of electrical engineering
• LDOS (Digital signal processing laboratory)
– Approx 15 members
Relevant people
Head: prof. Jurij Tasič
Andrej Košir
Marko Tkalčič
Ante Odić
Matevž Kunaver Tomaž Požrl
5. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LDOS work on recommender systems
2002-2009: public movie datasets
– CBR Basic RecSys
– CF
2009-2012
– Emotions Affective
– Context Computing
2012 – Affective
– Decision making (affective + cognitive attributes) RecSys
• Ajzen model
• Kahneman/Tversky model
• ...
Decision making
Modeling in RecSys
6. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Underlying presumption of our work
Recommender System = predictor of users‘ decision making
Decision making: EMOTIONS DO INFLUENCE
(c) Dilbert.com
7. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART II : What is affect/emotions/mood/personality
8. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART II : What are affect/emotions/mood/personality
Not well defined (wikipedia):
Psychophysiological
expressions
– Emotion = subjective, conscious experience
Biological
reactions
– Affect = experience of emotion (interchangable)
Mental
states
– Emotion vs. Mood:
• Emotion = high arousal, short term
• Mood = low arousal, long term
– Personality = accounts for the individual differences in the users’
emotional, interpersonal, experiential, attitudinal and motivational styles [John and Srivastava, 1999]
CHANGES FIXED
emotion mood personality
Time (duration)
9. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Overview of emotions/moods
Several definitions
We take simple models, easy to incorporate in computers:
– Basic emotions
– Dimensional model
– Circumplex model
10. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Basic emotions
Discrete classes model
Different sets
Charles Darwin: Expression of emotions in man and animal
Paul Ekman definition (6 + neutral):
– Happiness
– Anger
– Fear
– Sadness
– Disgust
– Surprise
(c) Paul Ekman
11. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Basic emotions
Discrete classes model
Different sets
Charles Darwin: Expression of emotions in man and animal
Paul Ekman definition (6 + neutral):
– Happiness
– Anger
– Fear
– Sadness
– Disgust
– Surprise
(c) Paul Ekman
12. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Dimensional model
Three dimensions
– Valence (positive vs. Negative)
– Arousal (high vs. Low)
– Dominance (power(less) over emotions)
13. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Dimensional model
Three dimensions
– Valence
– Arousal
– Dominance
– (c) Lang, P. J. (1980)
Each emotive state is a point in the VAD space
14. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Circumplex model
Maps basic emotions dimensional model (Posner et al.)
Arousal
high
joy
anger
surprise
disgust
fear
Valence
neutral
negative positive
sadness
low
15. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
How to detect emotions
Emotions are characterized:
– psychophysiological expressions,
– biological reactions
– mental states
SENSORS !!!
16. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
How to detect emotions?
Explicit vs. Implicit
Explicit
– Questionnaires (SAM)
17. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
How to detect emotions?
Explicit vs. Implicit
Explicit
– Questionnaires (SAM)
Implicit:
– Work done in the affective computing community
– Different modalities (sources):
• Facial actions (video)
• Physiological signals ( GSR, EEG)
• Voice
• Posture
• ...
– ML techniques
• Classification (basic emotions)
• Regression (dimensional model)
18. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Lie To Me
Part1.avi
(c) 20th Century Fox
Main Character Cal Lightman = Paul Ekman
Defined the FACS
(Facial Action Coding System)
19. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LDOS Experiment
2 datasets:
– Posed (Cohn-Kanade dataset)
– Spontaneous (LDOS-PerAff-1 dataset)
Input: Video streams of facial expressions as responses to visual stimuli
Output: emotive states as distinct classes
Gabor features kNN
Emotive
state
20. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Results and conclusions
Posed dataset: accuracy = 92 %
Spontaneous dataset: accuracy = 62%
Reasons for bad results:
– Weak learning supervision
– Non optimal video acquisition (face rotation, occlusions, changing lightning ...)
– Non extreme facial expressions
Upcoming paper: IEEE Transactions on Multimedia
21. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Personality
Definition:
User personality accounts for the individual differences in the users’
emotional, interpersonal, experiential, attitudinal and motivational styles [John and
Srivastava, 1999]
Ever-lasting
Several models
Five Factor Model (FFM or Big5):
Openness (inventive/curious vs. consistent/cautious)
Conscientiousness (efficient/organized vs. easy-going/careless)
Extraversion (outgoing/energetic vs. solitary/reserved)
Agreeableness (friendly/compassionate vs. cold/unkind)
Neuroticism (sensitive/nervous vs. secure/confident)
How to measure?
Questionnaires:
International Personality Item Pool ( http://ipip.ori.org/ )
22. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LDOS PerAff-1 dataset
Emotive responses
Ratings
Personality data
Videos of facial expressions
50 users, 70 items, sparsity=0
http://slavnik.fe.uni-lj.si/markot/Main/LDOS-PerAff-1
23. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART III : Related work on emotions in recsys
24. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART III : Related work on emotions in recsys
Emotions and personality
Scattered work
25. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Emotions in recsys
Gonzalez, 2007 ? Emotions as context in recsys?
User affective feedback from
Arapakis et al., 2009
automatic facial expression analysis
26. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Emotions in recsys
Gonzalez, 2007 ? Emotions as context in recsys?
User affective feedback from
Arapakis, 2009
automatic facial expression analysis
Tkalčič et al., 2010 Affective user model
27. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Emotions in recsys
Gonzalez, 2007 ? Emotions as context in recsys?
User affective feedback from
Arapakis, 2009
automatic facial expression analysis
Tkalčič et al., 2010 Affective user model
find an appropriate musical score that
Kaminskas, Ricci
would reinforce the affective state induced
2011
by the touristic attraction.
28. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Emotions in recsys
Gonzalez, 2007 ? Emotions as context in recsys?
User affective feedback from
Arapakis, 2009
automatic facial expression analysis
Tkalčič et al., 2010 Affective user model
find an appropriate musical score that
Kaminskas, Ricci
would reinforce the affective state induced
2011
by the touristic attraction.
Lops et al, 2012 Ongoing work: emotion detection in the phase of
presentation of the recommendations for generating
unexpected and seredipitous recommendations
29. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Personality in recsys
Nunes et al., 2007 Personality as ???
Personality-based user similarity measure
Tkalčič et al., 2009
For the cold start problem
Rong Hu and Personality-based user similarity measure
Pearl Pu, For the cold start problem
Dennis and Masthoff, Adapting persuasive (learning)
2012 Technologies to personality traits
30. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART IV: Emotions in the MM consumption chain
31. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART IV: Emotions in the MM consumption chain
Scattered work on emotions in RecSys
Unifying framework (too ambitious?)
32. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 1
time
choice
Give Give
recommendations content
Content application
Entry stage Consumption stage Exit stage
33. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 2
time
Entry mood Exit mood
choice
Detect
Give Give
entry
recommendations content
mood
Content application
• Context
• Decision making
• Influence
• Diversification
Entry stage Consumption stage Exit stage
34. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 3
time
Entry mood Content-induced affective state
choice
Detect
Give Give
entry Observe user
recommendations content
mood
Content application
• Affective tagging
• Affective user profiles
Entry stage Consumption stage Exit stage
35. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 3
time
Entry mood Content-induced affective state Exit mood
choice
Detect Detect
Give Give
entry Observe user exit
recommendations content
mood mood
Content application
• Implicit feedback
• Evaluation metrics
Entry stage Consumption stage Exit stage
36. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
The proposed framework - 3
time
Entry mood Content-induced affective state Exit mood
choice
Detect Detect
Give Give
entry Observe user exit
recommendations content
mood mood
Content application
• Context
• Decision making • Affective tagging
• Affective user profiles • Implicit feedback
• Influence • Evaluation metrics
• Diversification
Entry stage Consumption stage Exit stage
37. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART V: Affect in the decision making step
38. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
PART V: Affect in the decision making step
Stage 2 and 3 are straightforward
Stage 1 is interesting = new research avenues
time
Entry mood Content-induced affective state Exit mood
choice
Detect Detect
Give Give
entry Observe user exit
recommendations content
mood mood
Content application
• Context
• Decision making • Affective tagging
• Affective user profiles • Implicit feedback
• Influence • Evaluation metrics
• Diversification
Entry stage Consumption stage Exit stage
39. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
From data-centric to user-centric
The community is problem-solving oriented
– „The existing datasets are real, why building synthetic ones?“ (??, RecSys
2011)
The data-centric approach is still rooted in the research community:
– „It‘s about music, not about recommenders“ (?? at RecSys 2011)
Solving existing problems is only a part of research ...
... the other part is generating new knowledge (on how the world works) ...
... which in turn generates new problems ...
... which in turn opens new publishing/funding/citing possibilities
40. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
General user modeling framework
Data-centric = uses data that
– Is available (genres, actors, directors ...)
– Easy to acquire (rating, „liking“ ...)
But NOT necessarily data that carry information
Controlled variables
USER MODEL
Prediction accuracy
?
Uncontrolled variables
41. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
LET‘S MOVE FORWARD
Try new models!
Generate new kind of data!
Find out how the world really works!
Model DECISION MAKING:
– Ajzen model (Andrej‘s talk)
– Kahneman model
42. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
System 1 / System 2
(c) Kahneman, 2003
43. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Decision Making Modeling in RecSys
Personality Content
detection Affective stimuli metadata
detection
Emotion
detection
System 1 model System 2 model
Aggregation
Decision prediction
44. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
SoA Modeling in RecSys
Personality Content
detection Affective stimuli metadata
detection
Emotion
detection
System 1 model System 2 model
Aggregation
Decision prediction
45. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Conclusions
RecSys = decision making predictor
Assumption = emotions do influence
Scattered work Unifying framework
Our wish = Focus on stage 1: decision making:
– System 1 / System 2 modeling
46. Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..
Future work
Improve models
Generate dataset
Validate