Conference Centric 2011, Barcelona, Spain Full paper available at : http://www.thinkmind.org/index.php?view=article&articleid=centric_2011_2_30_30049 Abstract : Recommender systems aim at automatically providing objects related to user’s interests. The angular stone of such systems is a way to identify documents to be recommended. Indeed, the quality of these systems depends on the accuracy of its recommendation selection method. Thus, the selection method should be carefully chosen in order to improve end-user satisfaction. In this paper, we first compare two sets of approaches from the literature to underline that their results are significantly different. We also provide the conclusion of a survey done by thirty four students showing that diversity is considered as important in recommendation lists. Finally, we show that combining existing recommendation selection methods is a good mean to obtain diversity in recommendation lists.

1. Institut de Recherche en Informatique de Toulouse (IRIT) - UMR 5505
Bridging the gap between users and systems
Laurent CANDILLIER – Max CHEVALIER – Damien DUDOGNON – Josiane MOTHE
27/10/11

2. Diversity in recommender systems
 How to recommend documents for a visited one
 Maximizing the chances of retrieving at least one relevant
document per user [Santos et al., 2010]
 Cover a large range of users’ interests
 Context
 Blog platform
 Unknown user => no profile
 Diversity of users, diversity of their expectations
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 2

3. Diversity in recommender systems
 How to recommend documents for a visited one
 Maximizing the chances of retrieving at least one relevant
document per user [Santos et al., 2010]
 Cover a large range of users’ interests
 Context
 Blog platform
 Unknown user => no profile
 Diversity of users, diversity of their expectations
=> Diversify the recommendations
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 3

4. What is diversity?
 Definitions from the literature
 Topicality
 Related to a particular topic [Xu and Chen, 2006]
 Diversity
 Topical diversity
 Extrinsic: solve ambiguity [Radlinski et al., 2009]
 Intrinsic: avoid redundancy [Clarke et al., 2008]
 Serendipity
 Attractive and surprising documents [Herlocker et al., 2004]
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 4

5. Approaches to diversify IR results
 Topical diversity
 Clustering
 Identify aspects
 Reorder depending on the aspects covered
 Examples
 K-Means [Bi et al., 2009]
 Hierarchical Clustering [Meij et al., 2010]
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 5

6. Approaches to diversify IR results
 Topical diversity
 Sliding Windows
 Reorder the retrieved documents
 Select documents using metrics
 Similarity with the visited document
 Similarity with the current recommended document list
 Examples
 MMR [Carbonell and Goldstein, 1998]
 Intra-list similarity [Ziegler et al., 2005]
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 6

7. Approaches to diversify IR results
 Serendipity
 Alternative to topical diversity
 Similarity not only based on the content
 Examples
 Organizational similarity [Cabanac et al., 2007]
 Temporal diversity [Lathia et al., 2010]
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 7

8. Analysis of the TREC Web 2009 results
 Hypothesis
 Diversity of approaches
 No one approach for all users’ needs
 Approaches are complementary
 Valuable to combine them
 Goals
 Analyse results obtained with approaches having
 Same goal
 Similar performances
=> To identify if diversity exists
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 8

9. Analysis of the TREC Web 2009 results
 Experimental framework
 Reference IR corpus (TREC Web 2009)
 Two IR contexts
 Adhoc task
 Diversity task
 Compare results (runs) of the 4 best approaches of each task
 Similar performances according to IR metrics
 MAP for adhoc task
 NDCG for diversity task
 Overlap for each pair of runs underlying diversity
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 9

10. Analysis of the TREC Web 2009 results
 Adhoc Task
 Top 10 documents
 Overlap: 22.4%
 Precision: 0.384
 Overlap max < 30%
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 10

11. Analysis of the TREC Web 2009 results
 Diversity Task
 Top 10 documents
 Overlap: 6.3%
 Overlap max < 15%
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 11

12. Analysis of the TREC Web 2009 results
 Conclusions
 Two distinct approaches are unlikely to return the same
(relevant) documents
 Low average overlap
 Diversity of approaches
 No approach significantly better than others
 A combination can be valuable
 TREC tasks focused on topicality and topical diversity
 Can’t be used to evaluate other types of diversity
 Users’ study necessary [Hayes et al., 2002]
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 12

13. Users’ Study
 Our intuitions
 Most of the time, users want topicality
 Get focused information
 Sometime, they want diversity
 Topical diversity
 Enlarge the subject
 Serendipity
 Discover new information
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 13

14. Users’ Study
 Goals
 Verify our intuitions
 Prove that diversified recommendations answer a larger
range of users’ needs
 Context of experimentation
 34 students in M. Sc. (Management field)
 Blog post recommendations
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 14

15. Users’ Study
 Experimental Framework
 Select a document
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 15

16. Users’ Study
 Experimental Framework
 Read the selected
document
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 16

17. Users’ Study
 Experimental Framework
 Compute the recommendation lists
Approach 1 List 1 (random)
Approach 2
Approach 3
Approach 4 List 2 (fused)
Approach 5
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 17

18. Users’ Study
 Experimental Framework
 Compute the recommendation lists
Approach 1 List 1 (random)
Approach 2
Approach 3
Approach 4 List 2 (fused)
Approach 5
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 18

19. Users’ Study
 Experimental Framework
 Compute the recommendation lists
Approach 1 List 1 (random)
Approach 2
Approach 3
Approach 4 List 2 (fused)
Approach 5
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 19

20. Users’ Study
 Experimental Framework
 Compute the recommendation lists
Approach 1 List 1 (random)
Approach 2
Approach 3
Approach 4 List 2 (fused)
Approach 5
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 20

21. Users’ Study
 Experimental Framework
 Compute the recommendation lists
Approach 1 List 1 (random)
Approach 2
Approach 3
Approach 4 List 2 (fused)
Approach 5
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 21

22. Users’ Study
 Experimental Framework
 Present recommendation lists for assessment
Which list best meets your needs?
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 22

23. Users’ Study
 Experimental Framework
 Present recommendation lists for assessment
Which list is the most diversified?
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 23

24. Users’ Study
 Experimental Framework
 Assessment of all
documents
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 24

25. Users’ Study
 Approaches used
 searchsim
 Vector-space model
 Document title as query
 mlt
Topicality
 Apache Solr MoreLikeThis module
 Document content as query
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 25

26. Users’ Study
 Approaches used






 kmeans
 K-means classification Topical diversity
 One element per cluster
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 26

27. Users’ Study
 Approaches used









 blogart
 Random selection from the same blog
 topcateg Serendipity
 Popular documents in the same category
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 27

28. Users’ Study
 Approaches used
 Same analysis than TREC
experiments
 Same results
 Overlap is low (< 10%)
=> High diversity
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 28

29. Users’ Study
 Results
 Distribution of relevant documents
blogart fused kmeans fused
35% 65% 52.5% 21.3%
0% 26.2%
mlt fused
54.7% 32.8%
12.5%
searchsim fused topcateg fused
52.4% 38.9% 8.8% 91.2%
8.7% 0%
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 29

30. Users’ Study
 Results
 Distribution of relevant documents
kmeans fused
35% 65% 52.5% 21.3%
0% 26.2%
mlt fused
54.7% 32.8%
12.5%
searchsim fused
52.4% 38.9% 8.8% 91.2%
8.7% 0%
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 30

31. Users’ Study
 Results
 Distribution of relevant documents
blogart fused
35% 65% 52.5% 21.3%
0% 26.2%
54.7% 32.8%
12.5%
topcateg fused
52.4% 38.9% 8.8% 91.2%
8.7% 0%
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 31

32. Users’ Study
 Results
 Distribution of relevant documents
 Relevant mainly retrieved by topical approaches
 But at least 20% are retrieved only by fused
 Fused matches with a larger range of needs
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 32

33. Conclusions and future work
 Conclusions
 Diversity of users’ expectations
 No one approach to rule them all
 A diversity of approaches
 Complementary
 Fused
 Diversity helps RS to fit more users’ needs
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 33

34. Conclusions and future work
 Future work
 Real scale experiment
 OverBlog platform
 Renew the user survey
 More users (international call for participation)
 Avoid revealed biases
 e.g. More detailed form
=> Deeper analysis
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 34

35. Conclusions and future work
 Future work
 Improve the model
 Refining the fusing process
 Adding a learning process to weight each approach
 For every visited document
 Find the proportion of documents coming from each
approach (log analysis)
 Better match with the real users’ needs
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 35

36. Thank you for your attention
Questions ?
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 36

37. References
W. Bi, X. Yu, Y. Liu, F. Guan, Z. Peng, H. Xu, and X. Cheng, “ICTNET at Web Track 2009 diversity task”, Text REtrieval Conf., 2009
G. Cabanac, M. Chevalier, C. Chrisment, and C. Julien, “An Original Usage-based Metrics for Building a Unified View of Corporate Documents”,
Inter. Conf. on Database and Expert Systems Applications, 2007, LNCS V. 4653, 2007, pp. 202–212
J. Carbonell and J. Goldstein, “The use of MMR, diversity-based reranking for reordering documents and producing summaries”, ACM Conf. on
Research and Development in Information Retrieval, 1998, pp. 335-336
C. L. A. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I.n MacKinnon, “Novelty and Diversity in Information
Retrieval Evaluation”, ACM Conf. on Research and Development in Information Retrieval, 2008, pp. 659-666
C. Hayes, P. Massa, P. Avesani, and P. Cunningham, « An online evaluation framework for recommender systems», Workshop on Personalization
and Recommendation in E-Commerce, 2002
J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, “Evaluating Collaborative Filtering Recommender Systems”, ACM Trans. Information
Systems, 22(1), 2004, pp. 5-53
N. Lathia, S. Hailes, L. Capra, and X. Amatriain, “Temporal diversity in recommender systems”, ACM Conf. on Research and Development in
Information Retrieval, 2010, pp. 210-217
E. Meij, J. He, W. Weerkamp, and M. de Rijke, “Topical Diversity and Relevance Feedback”, Text REtrieval Conf., 2010
F. Radlinski, P. N. Bennett, B. Carterette, and T. Joachims. “Redundancy, diversity and interdependent document relevance”, SIGIR Forum, 43(2),
2009, pp. 46–52
R. L. T. Santos, C. Macdonald, and I. Ounis, “Selectively Diversifying Web Search Results”, ACM Inter. Conf. on Information and Knowledge
Management, 2010
Y. C. Xu and Z. Chen, “Relevance judgment: What do information users consider beyond topicality”, Journal of the American Society for
Information Science and Technology, 57(7), 2006, pp. 961–973
C. Ziegler, S. McNee, J. A. Konstan, and G. Lausen, “Improving recommendation lists through topic diversification”, Inter. Conf. on World Wide
Web, 2005, pp. 22–32
27/10/11 Candillier L. – Chevalier M. – Dudognon D. – Mothe M. 37