Recommender systems typically generate recommendations for a user based on their profile, or for an item given its user interactions, but there are many scenarios especially in leisure domains such as books, movies, games and music, where users have specific recommendation needs, where they want to steer the recommendation process towards certain aspects they find relevant. Currently, there are few recommender or search systems that can deal with the complexity of such directed needs, nor do we know well which data types (metadata, user ratings and reviews, item content) are useful to match against different aspects of recommendation needs. There are many discussion forums where users describe their needs and their frustration with current search and recommender systems. In this talk I will summarize our work on analyzing relevance aspects for these needs and describe experiments on dealing with these.
5. History of NDR
• Evolved from Social Book Search evaluation campaign (2011-2016)
• Evolved into analysing forum requests as complex information needs
• Books: Koolen et al. (CIKM 2012, ECIR 2014, ECIR 2015)
• Movies and Books: Bogers et al. (iConference 2018)
• Games: Bogers et al. (iConference 2019)
• Books, Movies, Games, Music: Bogers et al. (in preparation)
• Narrative-Driven Recommendation
• Bogers, Koolen (ACM RecSys 2017, KaRS 2018).
6. Scenario
• Narrative-Driven Recommendation (NDR, Bogers & Koolen 2017, 2018)
is a complex scenario:
• narrative description of desired aspects of items
• user preference info (user profile or example items)
• Related to Conversational Recommendation
• But different: e.g. human-directed, complex and often vaguely
expressed (latent) needs
• Interactions with conversational agents tend to be much simpler
• E.g. more concrete aspects (genre, creator, title, year, …)
8. ANALYSIS – WHAT DO NDR BOOKS REQUESTS LOOK LIKE?
▸ Annotated 1,457 book NDR request narratives with seven relevance aspects of
Reuters (JASIS 2007)
– Accessibility Accessibility in terms of the language, length, or level of difficulty.
– Content Aspects such as topic, plot, genre, style, or comprehensiveness.
– Engagement Books that fit a particular mood or interest or provide a particular
reading experience.
– Familiarity Books that are similar to known books or related to a previous
experience.
– Metadata Books with a certain title or by a certain author, editor, illustrator,
publisher, in a particular format, or written or published in certain year or period.
– Novelty Books with content that is novel to the reader, unusual or quirky.
– Socio-Cultural Books related to the user’s socio-cultural background or values,
books that are popular or obscure, or books that have had a particular impact.
8
9. ANALYSIS – WHAT DO THEY LOOK LIKE?
▸ Distribution of the seven narrative aspects
9
A C E F M N S
Accessibility 137 96 41 48 28 8 27
Content 598 157 267 176 26 98
Engagement 196 88 40 11 24
Familiarity 326 74 17 45
Metadata 179 11 25
Novelty 34 10
Socio-cultural 133
10. Search-Recommendation
Continuum
• Some requests are pure search
• “sci-fi books about space traders”
• Some requests are almost pure recommendation
• “Something as good as David Copperfield“
• Many requests mix search and recommendation
• “historical fiction set in 17th c. England that I’ll like
based on my profile”
11. Continuum and Latent
Interests
• Latent factors are related to amount of reading experience
• Novice readers ask for recommendations based on
examples (interests are latent)
• Experienced readers describe detailed content aspects
(interests are known)
• How does this work in other domains?
13. Comparing Domains
• We developed a relevance aspect model for leisure needs
• Five annotators together analysed forum posts in four domains
• Data from a range of discussion forums
• Books: 503 requests (LibraryThing)
• Movies: 538 requests (IMDB forums)
• Games: 521 requests (reddit)
• Music: 589 requests (reddit)
14. Domain Differences
• Noticeable differences
• NDR Movie requests often mention plot and characters, book
and game requests mention experience
• Book and game information needs are more complex, NDR
requests more prevalent on forums than movie and music needs
• Songs are short, low cost to access and consumption
• Books and games are long, high cost to access and
consumption
• Movies fall somewhere in between
16. Operationalizing Forum
Request Context
I’m looking for manly books about manly issues, that aren’t
too gritty, but make you think as much as you laugh. So far
these examples I have on my bookshelf: ‘About a Boy’ and
‘High Fidelity’ by Nick Hornby, ‘Train Man’ by Hitori Nakano.
Have you any other manly books for manly men such as I?
Requester
Post 1 ( = original request)
Thread
…
Well, of course, many of the early works of Ernest
Hemingway are an intense meditation on manliness. Esp.
The Sun Also Rises, the Nick Adams stories, the story "Fifty
Grand," but many of his other short stories too.User X
Post 2 (= 1st reply)
User preferences
Books
Metadata
User-generated
content
Curated
metadata
Narrative
Mini-profile
Book(s)
contains
Author(s)
Author(s) Book(s)
User suggestions
represented by
for
has
has
Context: requests in LibraryThing (LT) discussion forums
17. Methodology & Setup
• Requests
• 974 NDR forum requests (Bogers & Koolen, RecSys 2017)
• 331 requests have 1) examples, 2) suggestions by forum
members and 3) user profile of requester
• Datasets
• 2.8M Amazon/LT book records (metadata + user reviews/tags)
• 66K LibraryThing user profiles, 29M trans., 4.4M ratings
• Relevance judgements
• Recommendations posted by forum members
• Graded: higher relevance value if requester catalogues
recommendation
19. Baselines
• Collaborative Filtering
• LightFM toolkit (Kula 2015)
• Loss function: WARP (Weston et al. 2011), 300 factors
• Too few ratings, so transaction as implicit feedback
• 50/50 train/test (due to small no. of requests)
• Content Based Filtering
• Indri 5.4, LM with JM smoothing, Krovetz stemming, stopword
removal
• optimized based on earlier experiments (Bogers & Petras
2015, 2017)
• Narrative Driven: Narrative request as text query
• Example Driven: book metadata of examples as query
20. NDR Baselines
• CF less effective than CBF (also on standard recsys scenario)
so domain is atypical
• CBF effective with tags and esp. reviews (uses similar
language as requests?)
21. Example-Driven Recommenders
• EDR less effective than NDR
• CBF: esp. tags are effective
• CF: using examples much more effective than using full profile, less than CBF
• Example books more effective than example authors (drift?)
• requests with only example authors more difficult
22. Hybrid EDR + NDR
• EDR books + authors better than EDR books and EDR authors
• EDR + NDR more effective than NDR alone:
• Examples complement narrative (latent factors), narrative
more important
24. Popularity
• Forum recommendations that users follow up on are lower range
popularity
• People don’t need a recommender to find popular stuff
• Collab. Filtering picks mid-high range popularity items
• can use personalized popularity priors to rerank
• but still not directed towards current need
• Content-based picks more low range popularity items
• and can deal with directed needs
25. The Power of Reviews
• User reviews are highly effective
• not only for recommendation but many search tasks
(Koolen ECIR 2014)
• written in the language of the user (same as request)
• discuss broad range of aspects…
• … including reading/watching/listening/playing
experience!
26. Models for Capturing Needs -
Conversational Recommenders
• Complex narrative is difficult to interpret algorithmically
• Possible interaction: conversational models for iterative
structuring
• Kang et al., (RecSys 2017) look at queries in conversational movie
recommendation
• Objective: genre (“superhero movies”), deep features (“movies
with open endings or plot twists”)
• subjective: emotion (“sad movie”), quality (“interesting
characters, clever plot”), familiarity (“what would you
recommend to a fan of Big Lebowski?”)
27. Explanations From Reviews
• Lu et al. (RecSys 2018) used adversarial s2s learning to
generate explanations from user reviews
• generate review for new item based on own reviews of
consumed items
• Possible interaction:
• select reviews that discuss aspects and experiences
you’re interested in
• let system generate recommendations + explanations
28. Experience, Appeal and
Impact
• Many requests indicate that users are looking for a certain reading
experience or impact
• Saricks (2005) identified appeal elements based on style,
characterization, plot, pace
• We’re currently developing an impact model
• Identify and extract expressions of impact from user reviews (with
colleague Peter Boot)
• Impact type: emotional (not just binary sentiment) and cerebral
(changing your views, motivating you, bringing up memories)
• Impact cause: e.g. style, narrative
29. Conclusions
• Narrative-Driven Recommendation is a challenging task.
• Requires combination of data sources and algorithms to
solve.
• User-generated content essential for good performance
• Little explored scenario
• Use well-understood baselines to understand nature of
scenario, requests and data
• Next steps: other domains, deeper analysis, interaction models
30. References (1/2)
• Bogers, Koolen (RecSys 2017). Defining and Supporting Narrative-driven Recommendation
• Bogers, Koolen (KaRS 2018). “I’m looking for something like …”: Combining Narratives and
Example Items for Narrative-driven Book Recommendation
• Bogers et al. (iConf 2018). "What was this Movie About this Chick?" - A Comparative Study
of Relevance Aspects in Book and Movie Discovery
• Bogers et al. (iConf 2019). "Looking for an amazing game I can relax and sink hours into..." -
A Study of Relevance Aspects in Video Game Discovery
• Bogers, Petras (iConf 2015). Tagging vs. Controlled Vocabulary: Which is More Helpful for
Book Search?
• Bogers, Petras (iConf 2017). An in-depth analysis of tags and controlled metadata for book
search
• Kang et al. (RecSys 2017). Understanding How People Use Natural Language to Ask for
Recommendations
31. References (2/2)
• Koolen et al. (CIKM 2012). Social book search: comparing topical relevance judgements and
book suggestions for evaluation
• Koolen, M. (ECIR 2014). “User reviews in the search index? That’ll never work!”
• Koolen et al. (ECIR 2015). Looking for Books in Social Media: An Analysis of Complex Search
Requests
• Kula (CBRecSys 2015). Metadata Embeddings for User and Item Cold-start Recommendations
• Lu et al. (RecSys 2018). Why I like it: Multi-task Learning for Recommendation and Explanation
• Reuter (JASIS 2007). Assessing aesthetic relevance: Children's book selection in a digital library
• Saricks (2005). Readers' advisory service in the public library
• Weston et al. (IJCAI 2011). WSABIE: Scaling Up To Large Vocabulary Image Annotation
32. Thank You!
• Acknowledgements:
• collaborative work with Toine Bogers, Peter Boot, Maria
Gäde, Jaap Kamps, Vivien Petras, Mette Skov
• Questions?
