Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Explanation Strategies - Advances in Content-based Recommender System

372 Aufrufe

Veröffentlicht am

Our lecture for the RecSys Summer School 2019 in Goteborg (Sweden).

Veröffentlicht in: Technologie
  • Get Paid To Write Articles? YES! View 1000s of companies hiring online writers now! ◆◆◆ http://t.cn/AieXS62G
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Explanation Strategies - Advances in Content-based Recommender System

  1. 1. @cataldomusto cataldo.musto@uniba.it Advances in Content-based Recommender Systems Explanation Strategies CATALDO MUSTO UNIVERSITÀ DEGLI STUDI DI BARI ‘ALDO MORO’ - ITALY
  2. 2. Recommender Systems Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 2
  3. 3. The Explanation Problem Recommendation Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 I suggest you… 3
  4. 4. The Explanation Problem Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Recommendation 4
  5. 5. A possible solution: descriptive properties Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Recommendation I suggest you The Ring because you often like movies with Naomi Watts as 21 grams and Mulholland Drive. Furthermore, you like films about ghosts such as The Sixth Sense. 5
  6. 6. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Another solution: review-based features I recommend you The Ring because people who liked the movie think that it delivers some bone-chilling terror. Moreover, people liked The Ring since the casting is pretty good. 6
  7. 7. An overview of content-based strategies to build a domain-agnostic and algorithm-agnostic explanation supporting the recommendation. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 In this talk 7
  8. 8. An overview of content-based strategies to build a domain-agnostic and algorithm-agnostic explanation supporting the recommendation. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 In this talk 8
  9. 9. 1. Content-based Explanations exploiting the Linked Open Data cloud 2. Review-based Explanation exploiting Sentiment Analysis techniques 3. Review-based Explanations exploiting Automatic Text Summarization Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Agenda 9
  10. 10. 1. Content-based Explanations exploiting the Linked Open Data cloud Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Agenda Cataldo Musto, Fedelucio Narducci, Pasquale Lops, Marco de Gemmis, Giovanni Semeraro: ExpLOD: A Framework for Explaining Recommendations based on the Linked Open Data Cloud. Proceedings of RecSys 2016: pp. 151-154 (Best Paper Nominee) 10
  11. 11. 2. Review-based Explanation exploiting Sentiment Analysis techniques Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Agenda Cataldo Musto, Pasquale Lops, Marco de Gemmis, Giovanni Semeraro: Justifying Recommendations through Aspect-based Sentiment Analysis of Users Reviews. Proceedings of ACM UMAP 2019: pp. 4-12 11
  12. 12. 3. Review-based Explanations exploiting Automatic Text Summarization Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Agenda Cataldo Musto, Gaetano Rossiello, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro: Combining Text Summarization and Aspect-based Sentiment Analysis of Users’ Reviews to Justify Recommendations. To be presented at ACM RecSys 2019☺ 12
  13. 13. Content-based Explanations exploiting the Linked Open Data cloud Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 1. 13
  14. 14. Intuition Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Descriptive features of the items can be freely gathered from knowledge graphs as DBpedia (http://dbpedia.org) 14
  15. 15. Properties from DBpedia The Ring Ghost Films Hans Zimmer Naomi Watts Psychological Horror Films Films shot in California Horror Movies Japanese Movies Gore Verbinski dcterms:subject dbo:starring dcterms:subjectdcterms:subject Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 15
  16. 16. Methodology Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 16
  17. 17. ExpLOD Framework Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 17
  18. 18. ExpLOD: Mapper Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 18
  19. 19. ExpLOD: Mapper Mapper Profile Recommendations dbp:The_Ring_(2002_film)dbp:21_grams Profile Recommendation Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 19
  20. 20. ExpLOD: Builder Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 20
  21. 21. ExpLOD: Builder Recommendation American Films Psychological Movies Films about Ghosts Naomi Watts Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 21
  22. 22. ExpLOD: Ranker Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 22
  23. 23. ExpLOD: Ranker items in the user profile and in the recommendation list property number of edges connecting the property c with the items in the user profile number of edges connecting the property c with the items in the recommendation set Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 23
  24. 24. ExpLOD: Ranker Recommendation American Films Psychological Movies Films about Ghosts Naomi Watts Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 24
  25. 25. ExpLOD: Generator Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 25
  26. 26. ExpLOD: Generator Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 26
  27. 27. ExpLOD: Generator Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Naomi Watts I suggest you The Ring because you often like movies with Naomi Watts as 21 grams and Mulholland Drive. 27
  28. 28. ExpLOD: Generator Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Naomi Watts Films about Ghosts Furthermore, you like films about ghosts such as The Sixth Sense. 28
  29. 29. ExpLOD: final output Recommendation I suggest you The Ring because you often like movies with Naomi Watts as 21 grams and Mulholland Drive. Furthermore, you like films about ghosts such as The Sixth Sense. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 29
  30. 30. Experimental Evaluation Research Question How does our explanations perform with respect to other explanation strategies? Experimental Design User Study with a Web Application 308 subjects, Movie Domain Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness [^] Between-subjects experiment Configurations: ExpLOD, popularity-based baseline, non-personalized baseline [^] Tintarev, N., & Masthoff, J. Designing and evaluating explanations for recommender systems. In Recommender systems handbook. pp. 479-510. Springer, Boston, MA. 2011 Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 30
  31. 31. Experimental Protocol Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 1. Gathering movie preferences Users rated their favourite movies 31
  32. 32. Experimental Protocol Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 2. Recommendation is obtained Personalized PageRank as algorithm 1. Gathering movie preferences Users rated their favourite movies 32
  33. 33. Experimental Protocol Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 2. Recommendation is obtained Personalized PageRank as algorithm 3. Explanation is generated Random Configuration (users not aware) 1. Gathering movie preferences Users rated their favourite movies 33
  34. 34. Experimental Protocol Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 1. Gathering movie preferences Users rated their favourite movies 2. Recommendation is obtained Personalized PageRank as algorithm 3. Explanation is generated Random Configuration (users not aware) 4. Metrics are calculated Through Questionnaires 34
  35. 35. Explanations - Results Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 MOVIES ExpLOD Non-personalized Popularity transparency 4.18 3.04 3.01 persuasion 3.41 2.84 2.59 engagement 3.48 3.28 2.31 trust 3.39 2.81 2.67 effectiveness 0.72 0.66 0.93 35
  36. 36. Explanations - Results Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 MOVIES ExpLOD Non-personalized Popularity transparency 4.18 3.04 3.01 persuasion 3.41 2.84 2.59 engagement 3.48 3.28 2.31 trust 3.39 2.81 2.67 effectiveness 0.72 0.66 0.93 «I recommend you The Ring since you should like movies by Gore Verbinski whose music composer is Hans Zimmer» 36
  37. 37. Explanations - Results Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 MOVIES ExpLOD Non-personalized Popularity transparency 4.18 3.04 3.01 persuasion 3.41 2.84 2.59 engagement 3.48 3.28 2.31 trust 3.39 2.81 2.67 effectiveness 0.72 0.66 0.93 «I recommend you The Ring since it is very popular in the community» 37
  38. 38. Explanations - Results Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 MOVIES ExpLOD Non-personalized Popularity transparency 4.18 3.04 3.01 persuasion 3.41 2.84 2.59 engagement 3.48 3.28 2.31 trust 3.39 2.81 2.67 effectiveness 0.72 0.66 0.93 Significant improvement for 4 out of 5 metrics 38
  39. 39. Explanations - Results Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Aim Question transparency I understood why this movie was recommended to me topic director distributor composer persuasion The explanation made the recommendation more convincing awards director location producer engagement The explanation helped me discover new information writer director producer distributor trust The explanation increased my trust in the recommender system awards composer producer topic effectiveness I like this recommendation director writer location composer 39
  40. 40. Review-based Explanation exploiting Sentiment Analysis techniques Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 2. 40
  41. 41. Intuition To identify relevant and distinguishing characteristics of the recommended item by mining users’ reviews Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 41
  42. 42. Intuition Intense thriller Pretty good casting Well-plotted investigation Impressive horror ...... To identify relevant and distinguishing characteristics of the recommended item by mining users’ reviews Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 42
  43. 43. Workflow 43Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
  44. 44. Aspect Extraction Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 44
  45. 45. Aspect Extraction Goal: to identify the aspects that are discussed when people talk about the item Strategy: to use natural language processing techniques (specifically, part- of-speech tagging) to extract the names mentioned in users’ reviews Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 45
  46. 46. Aspect Extraction Goal: to identify the aspects that are discussed when people talk about the item Strategy: to use natural language processing techniques (specifically, part- of-speech tagging) to extract the names mentioned in users’ reviews Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 46
  47. 47. Aspect Extraction reviews aspects Input: reviews of the item i R = {ri1, ri2 … rin} Output: aspects A = {ai1, ai2 … aik} Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 47
  48. 48. Aspect Extraction reviews aspects Input: reviews of the item i R = {ri1, ri2 … rin} Output: aspects A = {ai1, ai2 … aik} Why only names? Findings from previous work in the area Why no bigrams? No significant improvement emerged Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 48
  49. 49. Aspect Ranking Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 49
  50. 50. Aspect Ranking Goal: to identify distinguishing aspects that are discussed with a positive sentiment when people talk about the item Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 50
  51. 51. Aspect Ranking Goal: to identify distinguishing aspects that are discussed with a positive sentiment when people talk about the item How many times aspect ‘a’ appears in the reviews of item ‘i’ (frequency of the aspect) Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 51
  52. 52. Aspect Ranking Goal: to identify distinguishing aspects that are discussed with a positive sentiment when people talk about the item How many times aspect ‘a’ appears in the reviews of item ‘i’ (frequency of the aspect) How positive is the opinion of the users when they talk about aspect ‘a’ (opinion towards the aspect) Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 52
  53. 53. Aspect Ranking Goal: to identify distinguishing aspects that are discussed with a positive sentiment when people talk about the item How many times aspect ‘a’ appears in the reviews of item ‘i’ (frequency of the aspect) How positive is the opinion of the users when they talk about aspect ‘a’ (opinion towards the aspect) How distinguishing is the aspect ‘a’ (inverse popularity) Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 53
  54. 54. Aspect Ranking Goal: to identify distinguishing aspects that are discussed with a positive sentiment when people talk about the item Intuition: our formula gives an higher score to the aspects that are frequently mentioned in the reviews with a positive sentiment. Moreover, it also rewards less popular aspects (higher IAF). Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 54
  55. 55. Aspect Ranking aspects top-k aspects Input: aspects A = {ai1, ai2 … aim} Output: top-k aspects A = {ai1, ai2 … aik} Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 55
  56. 56. Generation Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 56
  57. 57. Generation Goal: to generate a template-based natural language justification that relies on the most relevant aspects identified by the ASPECT RANKING module. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 57
  58. 58. Generation Goal: to generate a template-based natural language justification that relies on the most relevant aspects identified by the ASPECT RANKING module. For each aspect ’a’ returned by the ASPECT RANKING module Browse the available reviews Look for a compliant excerpt containing ‘a’ If the sentence has a positive sentiment Add the sentence to the justification Strategy Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 58
  59. 59. Generation Question: when does an excerpt is a compliant sentence? Answer: an excerpt is compliant if it follows one of the 18 justification patterns we defined Example: the excerpt must have a third personal singular verb Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 59
  60. 60. Generation Question: when does an excerpt is a compliant sentence? “I really liked the cast” Not compliant “The cast was great” Compliant Example: the excerpt must have a third personal singular verb Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 60 Answer: an excerpt is compliant if it follows one of the 18 justification patterns we defined
  61. 61. Generation – Final Output I recommend you The Ring because people who liked the movie think that it delivers some bone- chilling terror. Moreover, people liked The Ring since the casting is pretty good. Legenda Red: randomized template sentences Green: recommendation Blue: aspects (k=2) Black: compliant excerpts Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 61
  62. 62. Experimental Evaluation Research Question 1 How effective are the justifications generated through the pipeline, on varying of different combinations of the parameters? Research Question 2 How does our justifications perform with respect to a classic feature-based explanation? Experimental Design User Study with a Web Application 286 subjects Movie and Books Domain Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness Between-subjects for Research Question 1, Within-subjects for Research Question 2 Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 62
  63. 63. Experimental Evaluation Parameters of the system - Length of the justifications (short vs. long justifications) short → top-2 aspects long -> top-4 aspects - Vocabulary of aspects (statics vs. complete) static → bounded to a fixed and pre-defined set of relevant aspects. No aspect extraction, just aspect ranking complete → not bounded. All the aspects are discovered by the Aspect Extractor - Four different configurations Implementation Details Recommendations generated through Personalized PageRank, aspect extraction through CoreNLP POS-tagger and sentiment analysis through Stanford algorithm Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 63
  64. 64. Experimental Protocol Recommendation Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 64
  65. 65. Experimental Protocol (Research Question 1) Recommendation Review-based Explanation Questionnaire Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 65
  66. 66. Experimental Protocol (Research Question 2) I propose you “Aliens” because you sometimes like movies edited by Canadian film editors, American fiction films and 1980s films, as The Terminator. I recommend you “Aliens” because people who liked this movie think that the Alien series is one of the best sci-fi movies and that the ending is awesome with some fantastic action scenes. Review-based Explanation Feature-based Explanation Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 66
  67. 67. Results (Research Question 1) MOVIES Transparency Persuasion Engagement Trust Effectiveness Static Short 3.40 3.13 3.09 3.23 0.64 Static Long 3.77 3.68 3.55 3.73 0.55 Complete Short 3.91 3.60 3.25 3.70 0.53 Complete Long 3.74 3.48 3.35 3.46 0.59 Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 67
  68. 68. Results (Research Question 1) Finding 1 With the ‘complete’ set of aspects, shorter justifications have the best results Finding 2 With the ‘static’ set of aspects, longer justifications have the best results Overall Long justifications based on static aspects have the best results in the Movie Domain MOVIES Transparency Persuasion Engagement Trust Effectiveness Static Short 3.40 3.13 3.09 3.23 0.64 Static Long 3.77 3.68 3.55 3.73 0.55 Complete Short 3.91 3.60 3.25 3.70 0.53 Complete Long 3.74 3.48 3.35 3.46 0.59 Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 68
  69. 69. Results (Research Question 2) MOVIES Review- based Feature- based Indiffer. Transparency 47.4% 38.6% 14.0% Persuasion 51.7% 43.3% 5.0% Engagement 66.7% 25.0% 8.3% Trust 53.3% 35.5% 11.7% Effectiveness 57.9% 35.0% 7.1% Outcome: Users preferred Review-based Justifications Confirmed for all the metrics and both the domains Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 69
  70. 70. Review-based Explanations exploiting Automatic Text Summarization Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 3. 70
  71. 71. Why do we need another approach that exploits users’ reviews? Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Motivations 71
  72. 72. Why do we need another approach that exploits users’ reviews? Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Motivations Our first methodology has two main weaknesses • Very naïve strategy for ASPECT EXTRACTION • Very static template-based GENERATION 72
  73. 73. To exploit automatic text summarization techniques to build an higher-quality justifications. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Intuition 73
  74. 74. To exploit automatic text summarization techniques to build an higher-quality justifications. We conceive our justification as a summary of the information conveyed by all the available reviews. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Intuition 74
  75. 75. Workflow Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 75
  76. 76. Workflow Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Same conceptual workflow, different implementations of the modules! 76
  77. 77. Aspect Extraction Statistical approach based on the Kullback-Leibler (KL) Divergence Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Measures the difference between the distribution of a term in a generic corpus (e.g. BNC) and its distribution in a domain corpus (e.g. movie reviews) 77
  78. 78. Aspect Extraction Measures the difference between the distribution of a term in a generic corpus (e.g. BNC) and its distribution in a domain corpus (e.g. movie reviews) Insight: the higher the divergence, the higher the importance of the term in the domain t = term ca = corpus A cb = corpus B Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Statistical approach based on the Kullback-Leibler (KL) Divergence 78
  79. 79. Aspect Extraction Measures the difference between the distribution of a term in a generic corpus (e.g. BNC) and its distribution in a domain corpus (e.g. movie reviews) Insight: the higher the divergence, the higher the importance of the term in the domain Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 KL(cast, BNC, movie-reviews) >> 0 KL(actor, BNC, movie-reviews) > 0 KL(city, BNC, movie-reviews) ~ 0 KL(woman, BNC, movie-reviews) ~ 0 We label as ‘aspects’ the nouns whose KL-divergence is higher than zero Statistical approach based on the Kullback-Leibler (KL) Divergence 79
  80. 80. Aspect Extraction Measures the difference between the distribution of a term in a generic corpus (e.g. BNC) and its distribution in a domain corpus (e.g. movie reviews) Insight: the higher the divergence, the higher the importance of the term in the domain Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 KL(cast, BNC, movie-reviews) >> 0 YES KL(actor, BNC, movie-reviews) > 0 YES KL(city, BNC, movie-reviews) ~ 0 NO KL(woman, BNC, movie-reviews) ~ 0 NO Statistical approach based on the Kullback-Leibler (KL) Divergence 80
  81. 81. Aspect Ranking Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 81
  82. 82. Aspect Ranking Goal: to identify distinguishing aspects that are discussed with a positive sentiment when people talk about the item Novelty: KL-divergence is used as relevance score rela,Ri Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 82
  83. 83. Generation Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Same conceptual workflow, different implementations of the modules! 83
  84. 84. Generation Intuition: we conceive our justification as a summary of the information conveyed by all the available reviews Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 84
  85. 85. Generation Intuition: we conceive our justification as a summary of the information conveyed by all the available reviews Approach: we exploited a centroid-based method for automatic text summarization. Very good performance in multi-document summarization scenarios. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Assumption: each review can be considered as ‘document’ thus the corpus of the reviews can be used to feed the algorithm 85
  86. 86. Generation Generation process is in turn split into two steps • Sentence Filtering • Text Summarization Sentence Filtering is used to feed the summarization algorithm with compliant sentences. We selected sentences that matched the following criterions: Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 86
  87. 87. Generation Generation process is in turn split into two steps • Sentence Filtering • Text Summarization Sentence Filtering is used to feed the summarization algorithm with compliant sentences. We selected sentences that matched the following criterions: Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 • The sentence contains a main aspect • The sentence is longer than 5 tokens • The sentence expresses a positive sentiment • The sentence does not contain first-person personal or possessive pronouns 87
  88. 88. Generation Text Summarization Algorithm Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Input: item i, sentences s1…sn, word limit k Output: summary for item i consisting of k words 1. Build a vector space representation for each sentence 2. Merge all the sentences in a pseudo-document that represents the item (centroid vector) 3. Until the word limit is not reached 3.1 Go through the sentences and calculate cosine similarity 3.2 Pick the one with the highest cosine similarity to the centroid (and not that similar to the sentences previously picked) 3.3 Add it to the summary 88
  89. 89. Generation Text Summarization Algorithm Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Input: item i, sentences s1…sn, word limit k Output: summary for item i consisting of k words 1. Build a vector space representation for each sentence 2. Merge all the sentences in a pseudo-document that represents the item (centroid vector) 3. Until the word limit is not reached 3.1 Go through the sentences and calculate cosine similarity 3.2 Pick the one with the highest cosine similarity to the centroid (and not that similar to the sentences previously picked) 3.3 Add it to the summary 89
  90. 90. Generation Text Summarization Algorithm Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Input: item i, sentences s1…sn, word limit k Output: summary for item i consisting of k words 1. Build a vector space representation for each sentence 2. Merge all the sentences in a pseudo-document that represents the item (centroid vector) 3. Until the word limit is not reached 3.1 Go through the sentences and calculate cosine similarity 3.2 Pick the one with the highest cosine similarity to the centroid (and not that similar to the sentences previously picked) 3.3 Add it to the summary 90
  91. 91. Generation Text Summarization Algorithm Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Input: item i, sentences s1…sn, word limit k Output: summary for item i consisting of k words 1. Build a vector space representation for each sentence 2. Merge all the sentences in a pseudo-document that represents the item (centroid vector) 3. Until the word limit is not reached 3.1 Go through the sentences and calculate cosine similarity 3.2 Pick the one with the highest cosine similarity to the centroid (and not that similar to the sentences previously picked) 3.3 Add it to the summary 91
  92. 92. Generation Text Summarization Algorithm Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Input: item i, sentences s1…sn, word limit k Output: summary for item i consisting of k words 1. Build a vector space representation for each sentence 2. Merge all the sentences in a pseudo-document that represents the item (centroid vector) 3. Until the word limit is not reached 3.1 Go through the sentences and calculate cosine similarity 3.2 Pick the one with the highest cosine similarity to the centroid (and not that similar to the sentences previously picked) 3.3 Add it to the summary 92
  93. 93. Generation – Final Output “If you like or love the blood and gore kinds of films, this movie will certainly disappoint you as the focus is on character, story, mood and unique special effects. The Ring is a story about supernatural evil therefore, it is a horror film, done very much in the style of the psychological thriller.” Legenda Red: aspects (k=4) Black: compliant excerpts Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 93
  94. 94. Experimental Evaluation Research Question 1 How effective are the justifications generated through the pipeline, on varying of different combinations of the parameters? Research Question 2 How does our justifications perform with respect to a simple review-based explanation? Experimental Design User Study with a Web Application 141 subjects Movie Domain. 300 movies. ~150k reviews. Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness [^] Parameters: Justification Length (Short=50 words, Long=100) and #Aspects (10 and 30). Between-subjects for Research Question 1, Within-subjects for Research Question 2 [^] Tintarev, N., & Masthoff, J. Designing and evaluating explanations for recommender systems. In Recommender systems handbook. pp. 479-510. Springer, Boston, MA. 2011 Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 94
  95. 95. Results (Research Question 1) MOVIES Transparency Persuasion Engagement Trust Effectiveness Top-10 Short 2.83 3.06 3.06 2.83 0.89 Top-30 Long 3.16 3.06 2.69 3.19 0.94 Top-10 Short 3.95 3.64 3.37 3.55 0.55 Top-30 Long 3.24 3.18 3.12 3.22 0.38 Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 Finding 1 Long justifications better than short justifications, on average Finding 2 Top-10 aspect provide better explanations than Top-30 aspects Finding 3 Long explanations based on Top-10 aspects lead to the best results 95
  96. 96. Results (Research Question 2) MOVIES Review+ Summary Review- based Indiffer. Transparency 54.5% 40.9% 4.6% Persuasion 77.3% 13.6% 9.1% Engagement 63.6% 27.3% 9.1% Trust 68.2% 4.5% 27.3% Outcome: automatic Text Summarization provides users with the best explanation Confirmed for all the metrics and both the domains Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 96
  97. 97. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 97 Recap and Take Home Messages
  98. 98. Recap Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 “If you like or love the blood and gore kinds of films, this movie will certainly disappoint you as the focus is on character, story, mood and unique special effects. The Ring is a story about supernatural evil therefore, it is a horror film, done very much in the style of the psychological thriller.” I recommend you The Ring because people who liked the movie think that it delivers some bone- chilling terror. Moreover, people liked The Ring since the casting is pretty good. I suggest The Ring because you often like movies with Naomi Watts as 21 grams and Mulholland Drive. Furthermore, you like films about ghosts such as The Sixth Sense. Feature-based explanation exploiting DBpedia Review-based explanation using sentiment analysis Review-based explanation using automatic text summarization 98
  99. 99. Take-home Messages Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 99 1. 2. All the methodologies can provide the users with satisfying explanations, that can support the suggestions returned by a generic recommendation algorithm How to choose the most suitable one? Available data and explanation aims have to drive the choice! Feature-based: easier approach, good transparency; Review-based: improves the persuasion and the engagement; Summarization-based: more sophisticated generation, good for long-term usage of the explanation facilities.
  100. 100. Thank you! cataldo.musto@uniba.it @cataldomusto Contacts Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies. ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 100

×