4. Models for Implicit feedback
â˘âŻ Classification or Learning to Rank
â˘âŻ Binary pairwise ranking loss function (Hinge, AUC loss)
â˘âŻ Sample from non-observed/irrelevant entries
Friend Not a Friend
RecSys 2012, Dublin, Ireland, September 13, 2012
4
5. Learning to Rank in CF
â˘âŻ The Point-wise Approach f (user, item) â R
â˘âŻ Reduce Ranking to Regression, Classification, or Ordinal
Regression problem, [OrdRec@Recys 2011]
â˘âŻ The Pairwise Approach f (user, item1 , item2 ) â R
â˘âŻ Reduce Ranking to pair-wise classification [BPR@UAI 2010]
â˘âŻ List-wise Approach f(user, item1 , . . . , itemn ) â R
â˘âŻ Direct optimization of IR measures, List-wise loss minimization
[CoFiRank@NIPS 2008]
RecSys 2012, Dublin, Ireland, September 13, 2012
5
6. Ranking metrics
List-wise ranking measures for implicit feedback:
1
Reciprocal Rank: RR =
ranki
|S|
Average precision: P (k)
k=1
[TFMAP@SIGIR2012] AP =
|S|
RecSys 2012, Dublin, Ireland, September 13, 2012
6
7. Ranking metrics
RR = 1
AP = 1
RecSys 2012, Dublin, Ireland, September 13, 2012
7
8. Ranking metrics
RR = 1
AP = 0.66
RecSys 2012, Dublin, Ireland, September 13, 2012
8
9. Less is more
â˘âŻ Focus at the very top of the list
â˘âŻ Try to get at least one interesting item at the top of the list
â˘âŻ MRR particularly important measure in domains that usually
provide users with only few recommendations, i.e. Top-3 or
Top-5
RecSys 2012, Dublin, Ireland, September 13, 2012
9
10. Ingredients
â˘âŻ What kind of model should we use?
â˘âŻ Factor model
â˘âŻ Which Ranking measure do we need to optimize to have a good
Top-k recommender?
â˘âŻ MRR captures the quality of Top-k recommendations
â˘âŻ But MRR is not smooth so what can we do?
â˘âŻ We can perhaps find a smooth version of MRR
â˘âŻ How to ensure the proposed solution scalable?
â˘âŻ A fast learning algorithm (SGD), smoothness - gradients
RecSys 2012, Dublin, Ireland, September 13, 2012
10
12. The Non-smoothness of Reciprocal
Rank
â˘âŻ Reciprocal Rank (RR) of a ranked list of items for a given user
N
Yij N
RRi = (1 â Yik I(Rik Rij ))
j=1
Rij
k=1
RecSys 2012, Dublin, Ireland, September 13, 2012
12
17. How can we get a smooth-MRR?
â˘âŻ Borrow techniques from learning-to-rank:
I(Rik Rij ) â g(fik â fij )
1
â g(fij )
Rij
g(x) = 1/(1 + eâx )
RecSys 2012, Dublin, Ireland, September 13, 2012
17
18. MRR Loss function
N
N
RRi â Yij g(fij ) 1 â Yik g(fik â fij )
j=1 k=1
fij = Ui , Vj
Ui , V = arg max{RRi }
Ui ,V
2
O(N )
RecSys 2012, Dublin, Ireland, September 13, 2012
18
19. MRR loss function II
â˘âŻ Use concavity and monotonicity of log function,
N
N
L(Ui , V ) = Yij ln g(fij ) + ln 1 â Yik g(fik â fij )
j=1 k=1
+2
O(n )
RecSys 2012, Dublin, Ireland, September 13, 2012
19
20. Optimization
M N
E(U, V ) = Yij ln g(UiT Vj )
i=1 j=1
N
+ ln 1 â Yik g(UiT Vk â UiT Vj )
k=1
Îť
â (U 2 + V 2 )
2
âE âE
â˘âŻ Objective is smooth we can compute:
âUi âVj
â˘âŻ We use Stochastic Gradient Descent
â˘âŻ Overall scalability linear to # of relevant items O(dS)
RecSys 2012, Dublin, Ireland, September 13, 2012
20
21. Whatâs different ?
â˘âŻ CLiMF reciprocal rank loss essentially pushes relevant items
apart
â˘âŻ In the process at least one items ends up high-up in the list
RecSys 2012, Dublin, Ireland, September 13, 2012
21
22. Conventional loss
RecSys 2012, Dublin, Ireland, September 13, 2012
22
24. Experimental Evaluation
Data sets
â˘âŻ Epinions :
â˘âŻ 346K observations of trust relationship
â˘âŻ 1767 users; 49288 trustees
â˘âŻ 99.85% Sparseness
â˘âŻ Avg. friends/trustees per user 73.34
RecSys 2012, Dublin, Ireland, September 13, 2012
24
25. Experimental Evaluation
Data sets
â˘âŻ Tuenti :
â˘âŻ 798K observations of trust relationship
â˘âŻ 11392 users; 50000 friends
â˘âŻ 99.86% Sparseness
â˘âŻ Avg. friends/trustees per user 70.06
RecSys 2012, Dublin, Ireland, September 13, 2012
25
26. Experimental Evaluation
Experimental Protocol
Given 5
Friends/ Friends/
Trustees Trustees
data
Test data
(Items holdout)
Training data
RecSys 2012, Dublin, Ireland, September 13, 2012
26
27. Experimental Evaluation
Experimental Protocol
Given 10
Friends/ Friends/
Trustees Trustees
data
Test data
(Items holdout)
Training data
RecSys 2012, Dublin, Ireland, September 13, 2012
27
28. Experimental Evaluation
Experimental Protocol
Given 15
Friends/ Friends/
Trustees Trustees
data
Test data
(Items holdout)
Training data
RecSys 2012, Dublin, Ireland, September 13, 2012
28
29. Experimental Evaluation
Experimental Protocol
Given 20
Friends/ Friends/
Trustees Trustees
data
Test data
(Items holdout)
Training data
RecSys 2012, Dublin, Ireland, September 13, 2012
29
30. Experimental Evaluation
Evaluation Metrics
|S|
1 1
M RR =
|S| i=1 ranki
P @5
ratio of test users who have at
1 â call@5 least one relevant item in their
Top-5
RecSys 2012, Dublin, Ireland, September 13, 2012
30
32. Experimental Evaluation
Competition
â˘âŻ Pop: Naive, recommend based on popularity of each item
â˘âŻ iMF (Hu and Koren, ICDM 08): Optimizes Squared error loss
â˘âŻ BPR-MF (Rendle et al., UAI 09): Optimizes AUC
RecSys 2012, Dublin, Ireland, September 13, 2012
32
33. Epinions MRR
0.4
Pop iMF BPRâMF CLiMF
0.3
MRR
0.2
0.1
0.0
5 10 15 20
RecSys 2012, Dublin, Ireland, September 13, 2012
33
34. Epinions P@5
0.30
Pop iMF BPRâMF CLiMF
0.25
0.20
P@5
0.15
0.10
0.05
0.00
5 10 15 20
RecSys 2012, Dublin, Ireland, September 13, 2012
34
35. Epinions 1âcall@5
0.8
Pop iMF BPRâMF CLiMF
0.6
1âcall@5
0.4
0.2
0.0
5 10 15 20
RecSys 2012, Dublin, Ireland, September 13, 2012
35
36. Tuenti MRR
0.20
Pop iMF BPRâMF CLiMF
0.15
MRR
0.10
0.05
0.00
5 10 15 20
RecSys 2012, Dublin, Ireland, September 13, 2012
36
37. Tuenti P@5
0.10
Pop iMF BPRâMF CLiMF
0.08
0.06
P@5
0.04
0.02
0.00
5 10 15 20
RecSys 2012, Dublin, Ireland, September 13, 2012
37
38. Tuenti 1âcall@5
0.30
Pop iMF BPRâMF CLiMF
0.25
0.20
1âcall@5
0.15
0.10
0.05
0.00
5 10 15 20
RecSys 2012, Dublin, Ireland, September 13, 2012
38
39. Conclusions and Future Work
â˘âŻ Contribution
â˘âŻ Novel method for implicit data with some nice properties (Top-k, speed)
â˘âŻ Future work
â˘âŻ Use CLiMF to avoid duplicate or very similar recommendations in
the top-k part of the list
â˘âŻ To optimize other evaluation metrics for top-k recommendation
â˘âŻ To take the social network of users into account
RecSys 2012, Dublin, Ireland, September 13, 2012
39
40. Thank you !
Telefonica Research is looking for interns!
Contact: alexk@tid.es or linas@tid.es
RecSys 2012, Dublin, Ireland, September 13, 2012
40