5. Introduction
Bart Knijnenburg
- First user-centric
evaluation framework
- UMUAI, 2012
- Founder ofonUCERSTI
- workshop user-centric
evaluation of recommender
systems and their interfaces
- Statistics expert reviewer
- Conference + journal
- Research methods teacher
- SEM advisor
INFORMATION AND COMPUTER SCIENCES
7. Introduction
“What is a user experiment?”
“A user experiment is a scientific method to
investigate how and why system aspects
influence the users’ experience and behavior.”
INFORMATION AND COMPUTER SCIENCES
8. Introduction
My goal:
Teach how to scientifically evaluate recommender systems
using a user-centric approach
How? User experiments!
My approach:
- I will provide a broad theoretical framework
- I will cover every step in conducting a user experiment
- I will teach the “statistics of the 21st century”
INFORMATION AND COMPUTER SCIENCES
9. Introduction
Welcome everyone!
Evaluation framework
A theoretical foundation for user-centric evaluation
Hypotheses
What do I want to find out?
Participants
Population and sampling
Testing A vs. B
Experimental manipulations
Measurement
Measuring subjective valuations
Analysis
Statistical evaluation of the results
13. Framework
Offline evaluations may not give the same outcome as
online evaluations
Cosley et al., 2002; McNee et al., 2002
Solution: Test with real users
INFORMATION AND COMPUTER SCIENCES
14. Framework
System Interaction
algorithm rating
INFORMATION AND COMPUTER SCIENCES
15. Framework
Higher accuracy does not
always mean higher
satisfaction
McNee et al., 2006
Solution: Consider other
behaviors
INFORMATION AND COMPUTER SCIENCES
16. Framework
System Interaction
algorithm rating
consumption
retention
INFORMATION AND COMPUTER SCIENCES
17. Framework
The algorithm counts for only 5% of the relevance of a
recommender system
Francisco Martin - RecSys 2009 keynote
Solution: test those other aspects
INFORMATION AND COMPUTER SCIENCES
18. Framework
System Interaction
algorithm rating
interaction consumption
presentation retention
INFORMATION AND COMPUTER SCIENCES
19. Framework
“Testing a recommender against a random
videoclip system, the number of clicked clips
and total viewing time went down!”
INFORMATION AND COMPUTER SCIENCES
20. Framework
number of
clips watched
from beginning
to end total number of
+ viewing time clips clicked
personalized − −
recommendations
OSA
perceived system
effectiveness
+ EXP
+
perceived recommendation
quality
SSA
+
choice
satisfaction
EXP
Knijnenburg et al.: “Receiving Recommendations and Providing Feedback”, EC-Web 2010
INFORMATION AND COMPUTER SCIENCES
21. Framework
Behavior is hard to interpret
Relationship between behavior and satisfaction is not
always trivial
User experience is a better predictor of long-term retention
With behavior only, you will need to run for a long time
Questionnaire data is more robust
Fewer participants needed
INFORMATION AND COMPUTER SCIENCES
22. Framework
Measure subjective valuations with questionnaires
Perception and experience
Triangulate these data with behavior
Ground subjective valuations in observable actions
Explain observable actions with subjective valuations
Measure every step in your theory
Create a chain of mediating variables
INFORMATION AND COMPUTER SCIENCES
23. Framework
System Perception Experience Interaction
algorithm usability system rating
interaction quality process consumption
presentation appeal outcome retention
INFORMATION AND COMPUTER SCIENCES
24. Framework
Personal and situational characteristics may have an
important impact
Adomavicius et al., 2005; Knijnenburg et al., 2012
Solution: measure those as well
INFORMATION AND COMPUTER SCIENCES
25. Framework
Situational Characteristics
routine system trust choice goal
System Perception Experience Interaction
algorithm usability system rating
interaction quality process consumption
presentation appeal outcome retention
Personal Characteristics
gender privacy expertise
INFORMATION AND COMPUTER SCIENCES
26. Framework
Situational Characteristics
routine system trust choice goal
Objective System Aspects (OSA)
System Perception Experience Interaction
These are manipulations
algorithm usability system rating
interaction
- visual / interaction design consumption
quality process
presentation
- recommenderoutcome
appeal
algorithm
retention
- presentation of recommendations
- additional features
Personal Characteristics
gender privacy expertise
INFORMATION AND COMPUTER SCIENCES
27. Framework
Situational Characteristics
routine
User Experience (EXP) system trust choice goal
Different aspects may
System different things
influence Perception Experience Interaction
algorithm usability system rating
- interface -> system
interaction
evaluations quality process consumption
-
presentation appeal
preference elicitation outcome retention
method -> choice process
- algorithm -> quality of Personal Characteristics
the
final choice gender privacy expertise
INFORMATION AND COMPUTER SCIENCES
28. Framework
Situational Characteristics
routine system trust choice goal
Subjective System Aspects
(SSA)
System Perception Experienceto EXP
Link OSA Interaction
algorithm usability system
(mediation) rating
interaction quality process consumption
Increase the robustness
presentation appeal of the effects of OSAs on
outcome retention
EXP
How and why OSAs
Personal Characteristics
affect EXP
gender privacy expertise
INFORMATION AND COMPUTER SCIENCES
29. Framework
Situational Characteristics
routine system trust choice goal
System Perception Experience Interaction
Personal and Situational characteristics (PC and SC)
algorithm usability system rating
Effect of specific user and task
interaction quality process consumption
Beyond the influence of the system
presentation appeal outcome retention
Here used for evaluation, not for augmenting algorithms
Personal Characteristics
gender privacy expertise
INFORMATION AND COMPUTER SCIENCES
30. Framework
Situational Characteristics
routine system trust choice goal
Interaction (INT)
System Perception Experience Interaction
Observable behavior
algorithm usability system rating
- browsing, viewing, log-ins
interaction quality process consumption
presentation of evaluation
Final step appeal outcome retention
Grounds EXP in “real” data
Personal Characteristics
gender privacy expertise
INFORMATION AND COMPUTER SCIENCES
31. Framework
Situational Characteristics
routine system trust choice goal
System Perception Experience Interaction
algorithm usability system rating
interaction quality process consumption
presentation appeal outcome retention
Personal Characteristics
gender privacy expertise
INFORMATION AND COMPUTER SCIENCES
32. Framework
Has suggestions for measurement scales
e.g. How can we measure something like “satisfaction”?
Provides a good starting point for causal relations
e.g. How and why do certain system aspects influence the
user experience?
Useful for integrating existing work
e.g. How do recommendation list length, diversity and
presentation each have an influence on user experience?
INFORMATION AND COMPUTER SCIENCES
34. test your recommender system with real users
measure behaviors other test aspects other than
than ratings the algorithm
measure subjective take situational and
valuations with personal characteristics
questionnaires into account
Evaluation framework
Help in measurement and causal relations
use our framework as a starting point for user-centric research
39. Hypotheses
What does good mean?
- Recommendation accuracy?
- Recommendation quality?
- System usability?
- System satisfaction?
We need to define measures
INFORMATION AND COMPUTER SCIENCES
40. Hypotheses
“Can you test if the user interface of my
recommender system scores high on this
usability scale?”
INFORMATION AND COMPUTER SCIENCES
41. Hypotheses
What does high mean?
Is 3.6 out of 5 on a 5-point scale “high”?
What are 1 and 5?
What is the difference between 3.6 and 3.7?
We need to compare the UI against something
INFORMATION AND COMPUTER SCIENCES
42. Hypotheses
“Can you test if the UI of my recommender
system scores high on this usability scale
compared to this other system?”
INFORMATION AND COMPUTER SCIENCES
44. Hypotheses
Say we find that it scores higher on usability... why does it?
- different date-picker method
- different layout
- different number of options available
Apply the concept of ceteris paribus to get rid of
confounding variables
Keep everything the same, except for the thing you want
to test (the manipulation)
Any difference can be attributed to the manipulation
INFORMATION AND COMPUTER SCIENCES
45. Hypotheses
My new travel recommender Previous version
(too many options)
INFORMATION AND COMPUTER SCIENCES
46. Hypotheses
To learn something from the study, we need a theory behind
the effect
For industry, this may suggest further improvements to the
system
For research, this makes the work generalizable
Measure mediating variables
Measure understandability (and a number of other
concepts) as well
Find out how they mediate the effect on usability
INFORMATION AND COMPUTER SCIENCES
47. Hypotheses
An example:
We compared three
recommender systems
Three different algorithms
Ceteris paribus!
INFORMATION AND COMPUTER SCIENCES
48. Hypotheses
Knijnenburg et al.: “Explaining the user experience of recommender systems”, UMUAI 2012
The mediating variables show the entire story
INFORMATION AND COMPUTER SCIENCES
49. Hypotheses
Matrix Factorization recommender with Matrix Factorization recommender with
explicit feedback (MF-E) implicit feedback (MF-I)
(versus generally most popular; GMP) (versus most popular; GMP)
OSA OSA
+ +
perceived recommendation perceived recommendation perceived system
variety + quality + effectiveness
SSA SSA EXP
Knijnenburg et al.: “Explaining the user experience of recommender systems”, UMUAI 2012
INFORMATION AND COMPUTER SCIENCES
50. Hypotheses
“An approximate answer to the right problem
is worth a good deal more than an exact
answer to an approximate problem.”
John Tukey
INFORMATION AND COMPUTER SCIENCES
51. define measures
compare system get rid of
aspects against each confounding
other variables
apply the concept of look for a theory
ceteris paribus behind the found effects
Hypotheses
What do I want to find out?
measure mediating variables to explain the effects
55. Participants
“We are testing our recommender system
on our colleagues/students.”
-or-
“We posted the study link
on Facebook/Twitter.”
INFORMATION AND COMPUTER SCIENCES
56. Participants
Are your connections, colleagues, or students typical users
of your system?
- They may have more knowledge of the field of study
- They may feel more excited about the system
- They may know what the experiment is about
- They probably want to please you
You should sample from your target population
An unbiased sample of users of your system
INFORMATION AND COMPUTER SCIENCES
58. Participants
What are the consequences of limiting your scope?
You run the risk of catering to that subset of users only
You cannot make generalizable claims about users
For scientific experiments, the target population may be
unrestricted
Especially when your study is more about human nature
than about a specific system
INFORMATION AND COMPUTER SCIENCES
60. Participants
Is this a decent sample size?
Anticipated Needed
Can you attain statistically e!ect size sample size
significant results?
Does it provide a wide small 385
enough inductive base?
medium 54
Make sure your sample is
large enough
large 25
40 is typically the bare
minimum
INFORMATION AND COMPUTER SCIENCES
61. sample from your target population
the target population make sure your sample
may be unrestricted is large enough
Participants
Population and sampling
65. Manipulations
“Are our users more satisfied if our news
recommender shows only recent items?”
INFORMATION AND COMPUTER SCIENCES
66. Manipulations
Proposed system or treatment:
Filter out any items > 1 month old
What should be my baseline?
- Filter out items < 1 month old?
- Unfiltered recommendations?
- Filter out items > 3 months old?
You should test against a reasonable alternative
“Absence of evidence is not evidence of absence”
INFORMATION AND COMPUTER SCIENCES
67. Manipulations
“The first 40 participants will get the baseline,
the next 40 will get the treatment.”
INFORMATION AND COMPUTER SCIENCES
68. Manipulations
These two groups cannot be expected to be similar!
Some news item may affect one group but not the other
Randomize the assignment of conditions to participants
Randomization neutralizes (but doesn’t eliminate)
participant variation
INFORMATION AND COMPUTER SCIENCES
69. Manipulations
Between-subjects design: 100 participants
Randomly assign half the
participants to A, half to B
Realistic interaction 50 50
Manipulation hidden from
user
Many participants needed
INFORMATION AND COMPUTER SCIENCES
70. Manipulations
Within-subjects design: 50 participants
Give participants A first,
then B
- Remove subject variability
- Participant may see the
manipulation
- Spill-over effect
INFORMATION AND COMPUTER SCIENCES
71. Manipulations
Within-subjects design: 50 participants
Show participants A and B
simultaneously
- Remove subject variability
- Participants can compare
conditions
- Not a realistic interaction
INFORMATION AND COMPUTER SCIENCES
72. Manipulations
Should I do within-subjects or between-subjects?
Use between-subjects designs for user experience
Closer to a real-world usage situation
No unwanted spill-over effects
Use within-subjects designs for psychological research
Effects are typically smaller
Nice to control between-subjects variability
INFORMATION AND COMPUTER SCIENCES
73. Manipulations
Low High
You can test multiple diversity diversity
manipulations in a factorial 5
design 5+low 5+high
items
The more conditions, the 10
10+low 10+high
more participants you will items
need! 20
20+low 20+high
items
INFORMATION AND COMPUTER SCIENCES
74. Manipulations
Let’s test an algorithm against random recommendations
What should we tell the participant?
Beware of the Placebo effect!
Remember: ceteris paribus!
Other option: manipulate the message (factorial design)
INFORMATION AND COMPUTER SCIENCES
75. Manipulations
“We were demonstrating our new
recommender to a client. They were amazed
by how well it predicted their preferences!”
“Later we found out that we forgot to activate
the algorithm: the system was giving
completely random recommendations.”
(anonymized)
INFORMATION AND COMPUTER SCIENCES
76. test against a reasonable alternative
randomize assignment use between-subjects
of conditions for user experience
use within-subjects for you can test more
psychological research than two conditions
Testing A vs. B
Experimental manipulations
you can test multiple manipulations in a factorial design
81. Measurement
Does the question mean the same to everyone?
- John likes the system because it is convenient
- Mary likes the system because it is easy to use
- Dave likes it because the recommendations are good
A single question is not enough to establish content validity
We need a multi-item measurement scale
INFORMATION AND COMPUTER SCIENCES
82. Measurement
Perceived system effectiveness:
- Using the system is annoying
- The system is useful
- Using the system makes me happy
- Overall, I am satisfied with the system
- I would recommend the system to others
- I would quickly abandon using this system
INFORMATION AND COMPUTER SCIENCES
83. Measurement
Use both positively and negatively phrased items
- They make the questionnaire less “leading”
- They help filtering out bad participants
- They explore the “flip-side” of the scale
The word “not” is easily overlooked
Bad: “The recommendations were not very novel”
Good: “The recommendations felt outdated”
INFORMATION AND COMPUTER SCIENCES
84. Measurement
Choose simple over specialized words
Participants may have no idea they are using a
“recommender system”
Avoid double-barreled questions
Bad: “The recommendations were relevant and fun”
INFORMATION AND COMPUTER SCIENCES
85. Measurement
“We asked users ten 5-point scale questions
and summed the answers.”
INFORMATION AND COMPUTER SCIENCES
86. Measurement
Is the scale really measuring a single thing?
- 5 items measure satisfaction, the other 5 convenience
- The items are not related enough to make a reliable scale
Are two scales really measuring different things?
- They are so closely related that they actually measure the
same thing
We need to establish convergent and discriminant validity
This makes sure the scales are unidimensional
INFORMATION AND COMPUTER SCIENCES
87. Measurement
Solution: factor analysis
- Define latent factors, specify how items “load” on them
- Factor analysis will determine how well the items “fit”
- It will give you suggestions for improvement
Benefits of factor analysis:
- Establishes convergent and discriminant validity
- Outcome is a normally distributed measurement scale
- The scale captures the “shared essence” of the items
INFORMATION AND COMPUTER SCIENCES
91. Measurement
“Great! Can I learn how to do this myself?”
Check the video tutorials at
www.statmodel.com
INFORMATION AND COMPUTER SCIENCES
92. establish content validity with multi-item scales
follow the general establish convergent
principles for good and discriminant
questionnaire items validity
Measurement
Measuring subjective valuations
use factor analysis
96. Analysis
Perceived quality
Manipulation -> perception: 1
Do these two algorithms 0.5
lead to a different level of 0
perceived quality?
-0.5
T-test -1
A B
INFORMATION AND COMPUTER SCIENCES
97. Analysis
System effectiveness
2
Perception -> experience: 1
Does perceived quality
0
influence system
effectiveness? -1
Linear regression -2
-3 0 3
Recommendation quality
INFORMATION AND COMPUTER SCIENCES
98. Analysis
Perceived quality
0.6
Two manipulations ->
low diversification
perception: 0.5
high diversification
What is the combined 0.4
effect of list diversity and 0.3
list length on perceived 0.2
recommendation quality?
0.1
Factorial ANOVA 0
5 items 10 items 20 items
Willemsen et al.: “Not just more of the same”, submitted to TiiS
INFORMATION AND COMPUTER SCIENCES
99. Analysis
var1 var2 var3 var4 var5 var6 qual1 qual2 qual3 qual4 diff1 diff2 diff3 diff4 diff5
Only one method: structural 1
perceived
recommendation
1
perceived
recommendation
choice
1
difficulty
equation modeling
variety quality
The statistical method of
movie choice
1 1
expertise satisfaction
the 21st century exp1 exp2 exp3 sat1 sat2 sat3 sat4 sat5 sat6 sat7
Combines factor analysis
and path models
Top-20 Lin-20
vs Top-5 recommendations vs Top-5 recommendations
- Turn items into factors
.455 (.211) .612 (.220) .879 (.265)
-.804 (.230)
p < .05 p < .01 p < .001
+ + p < .001 − +
perceived + perceived + choice
recommendation recommendation
variety .503 (.090) quality .336 (.089)
difficulty
- Test causal relations
p < .001 p < .001
+ + -.417 (.125)
1.151 (.161)
.181 (.075) .205 (.083) p < .001 p < .005
p < .05 p < .05 + −
.894 (.287)
movie choice p < .005
+
expertise satisfaction
INFORMATION AND COMPUTER SCIENCES
100. Analysis
var1 var2 var3 var4 var5 var6 qual1 qual2 qual3 qual4 diff1 diff2 diff3 diff4 diff5
perceived perceived
choice
1 recommendation recommendation 1
1
variety quality difficulty
Very simple reporting 1
movie
expertise
choice
satisfaction
1
- Report overall fit + effect exp1 exp2 exp3 sat1 sat2 sat3 sat4 sat5 sat6 sat7
of each causal relation
- A path that explains the Top-20
vs Top-5 recommendations
Lin-20
vs Top-5 recommendations
effects .455 (.211)
p < .05
+ +
.612 (.220)
p < .01
-.804 (.230)
p < .001 −
.879 (.265)
p < .001
+
perceived + perceived + choice
recommendation recommendation
variety .503 (.090) quality .336 (.089)
difficulty
p < .001 p < .001
+ + -.417 (.125)
1.151 (.161)
.181 (.075) .205 (.083) p < .001 p < .005
p < .05 p < .05 + −
.894 (.287)
movie choice p < .005
+
expertise satisfaction
INFORMATION AND COMPUTER SCIENCES
101. Analysis
Example from Bollen et al.: “Choice Overload”
What is the effect of the number of recommendations?
What about the composition of the recommendation list?
Tested with 3 conditions:
- Toprecs: 1 2 3 4 5
5:
-
- Toprecs: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
20:
-
- Lin recs: 1 2 3 4 5 99 199 299 399 499 599 699 799 899 999 1099 1199 1299 1399 1499
20:
-
INFORMATION AND COMPUTER SCIENCES
102. Analysis
Bollen et al.: “Understanding Choice Overload in Recommender Systems”, RecSys 2010
INFORMATION AND COMPUTER SCIENCES
103. Analysis
Top-20 Lin-20
vs Top-5 recommendations vs Top-5 recommendations
Simple regression
“Inconsistent mediation”
“Full mediation” + + − +
perceived + perceived + choice
recommendation recommendation
variety quality difficulty
Trade-o!
Measured by+var1-var6 +
(not shown here) + −
movie choice
+
Additional e!ect
expertise satisfaction
Bollen et al.: “Understanding Choice Overload in Recommender Systems”, RecSys 2010
INFORMATION AND COMPUTER SCIENCES
104. Analysis
Top-20 Lin-20
vs Top-5 recommendations vs Top-5 recommendations
.455 (.211) .612 (.220) .879 (.265)
-.804 (.230)
p < .05 p < .01 p < .001
+ + p < .001 − +
perceived + perceived + choice
recommendation recommendation
variety .503 (.090) quality .336 (.089)
difficulty
p < .001 p < .001
+ + -.417 (.125)
1.151 (.161)
.181 (.075) .205 (.083) p < .001 p < .005
p < .05 p < .05 + −
.894 (.287)
movie choice p < .005
+
expertise satisfaction
Bollen et al.: “Understanding Choice Overload in Recommender Systems”, RecSys 2010
INFORMATION AND COMPUTER SCIENCES
105. Analysis
“Great! Can I learn how to do this myself?”
Check the video tutorials at
www.statmodel.com
INFORMATION AND COMPUTER SCIENCES
106. Analysis
Homoscedasticity is a 20
prerequisite for linear stats
Interaction time (min)
15
Not true for count data,
time, etc.
10
Outcomes need to be
unbounded and continuous 5
Not true for binary 0
answers, counts, etc. 0 5 10 15
Level of commitment
INFORMATION AND COMPUTER SCIENCES
107. Analysis
Use the correct methods for non-normal data
- Binary data: logit/probit regression
- 5- or 7-point scales: ordered logit/probit regression
- Count data: Poisson regression
Don’t use “distribution-free” stats
They were invented when calculations were done by hand
Do use structural equation modeling
MPlus can do all these things “automatically”
INFORMATION AND COMPUTER SCIENCES
108. Analysis
Standard regression requires
Number of followed recommendations
5
uncorrelated errors
Not true for repeated 4
measures, e.g. “rate these 3
5 items”
2
There will be a user-bias
(and maybe an item-bias) 1
Golden rule: data-points 0
0 5 10 15
should be independent
Level of commitment
INFORMATION AND COMPUTER SCIENCES
109. Analysis
Number of followed recommendations
5
Two ways to account for
repeated measures:
3.75
- Define a random
intercept for each user 2.5
- Impose an error 1.25
covariance structure
Again, in MPlus this is easy 0
0 5 10 15
Level of commitment
INFORMATION AND COMPUTER SCIENCES
110. Analysis
A manipulation only causes
things
Disclosure Perceived
For all other variables: help
(R2 = .302)
privacy threat
(R2 = .565)
- Common sense + − + −
- Psych literature Trust in the
company
(R2 = .662) +
Satisfaction
with the system
(R2 = .674)
- Evaluation frameworks Knijnenburg & Kobsa.: “Making Decisions about Privacy”
Example: privacy study
INFORMATION AND COMPUTER SCIENCES
111. Analysis
“All models are wrong, but some are useful.”
George Box
INFORMATION AND COMPUTER SCIENCES
112. use structural equation models
use correct use correct
methods for methods for
non-normal data repeated measures
Analysis
Statistical evaluation of the results
use manipulations and theory to make inferences about causality
113.
114.
115. Introduction
User experiments: user-centric evaluation of recommender systems
Evaluation framework
Use our framework as a starting point for user experiments
Hypotheses
Construct a measurable theory behind the expected effect
Participants
Select a large enough sample from your target population
Testing A vs. B
Assign users to different versions of a system aspect, ceteris paribus
Measurement
Use factor analysis and follow the principles for good questionnaires
Analysis
Use structural equation models to test causal models
116.
117. “It is the mark of a truly intelligent person
to be moved by statistics.”
George Bernard Shaw
118. Resources
User-centric evaluation
Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining
the User Experience of Recommender Systems. UMUAI 2012.
Knijnenburg, B.P., Willemsen, M.C., Kobsa, A.: A Pragmatic Procedure to
Support the User-Centric Evaluation of Recommender Systems. RecSys 2011.
Choice overload
Willemsen, M.C., Graus, M.P., Knijnenburg, B.P., Bollen, D.: Not just more of the
same: Preventing Choice Overload in Recommender Systems by Offering
Small Diversified Sets. Submitted to TiiS.
Bollen, D.G.F.M., Knijnenburg, B.P., Willemsen, M.C., Graus, M.P.: Understanding
Choice Overload in Recommender Systems. RecSys 2010.
INFORMATION AND COMPUTER SCIENCES
119. Resources
Preference elicitation methods
Knijnenburg, B.P., Reijmer, N.J.M., Willemsen, M.C.: Each to His Own: How
Different Users Call for Different Interaction Methods in Recommender
Systems. RecSys 2011.
Knijnenburg, B.P., Willemsen, M.C.: The Effect of Preference Elicitation
Methods on the User Experience of a Recommender System. CHI 2010.
Knijnenburg, B.P., Willemsen, M.C.: Understanding the effect of adaptive
preference elicitation methods on user satisfaction of a recommender system.
RecSys 2009.
Social recommenders
Knijnenburg, B.P., Bostandjiev, S., O'Donovan, J., Kobsa, A.: Inspectability and
Control in Social Recommender Systems. RecSys 2012.
INFORMATION AND COMPUTER SCIENCES
120. Resources
User feedback and privacy
Knijnenburg, B.P., Kobsa, A: Making Decisions about Privacy: Information
Disclosure in Context-Aware Recommender Systems. Submitted to TiiS.
Knijnenburg, B.P., Willemsen, M.C., Hirtbach, S.: Getting Recommendations and
Providing Feedback: The User-Experience of a Recommender System. EC-
Web 2010.
Statistics books
Agresti, A.: An Introduction to Categorical Data Analysis. 2nd ed. 2007.
Fitzmaurice, G.M., Laird, N.M., Ware, J.H.: Applied Longitudinal Analysis. 2004.
Kline, R.B.: Principles and Practice of Structural Equation Modeling. 3rd ed. 2010.
Online tutorial videos at www.statmodel.com
INFORMATION AND COMPUTER SCIENCES