The document summarizes two studies that explored different set-based visualizations for helping users explore recommendations. The studies compared TalkExplorer, a graph-based recommender, and SetFusion, which used an interactive Venn diagram. Results showed that visualizing intersections of relevant contexts helped users discover more relevant items. SetFusion may have better supported exploration of multiple intersections through its Venn diagram interface. Future work could explore scaling SetFusion to more data sources and recommendation algorithms.

1. The Effect of Different Set-based Visualizations on User Exploration of Recommendations
KatrienVerbert, KU Leuven
Denis Parra, PUC Chile
Peter Brusilovsky, University of Pittsburgh
IntRSWorkshop at RecSys2014, Foster City, CA, USA

2. Outline
•Context of this Work in RecSysresearch
•Set-based Visual Interfaces for User Exploration
–TalkExplorer: Multimode graph
–SetFusion: Venn diagram
•Meta-Analysis
•Summary & Future Work
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 2

3. INTRODUCTION
Recommender Systems: Introduction & Motivation
3
* Danboard(Danbo): Amazon’s cardboard robot, in these slides represents a recommender system
*

4. Recommender Systems (RecSys)
Systems that help people (or groups) to find relevant items in a crowded item or information space (McNeeet al. 2006)
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 4

5. Challenges of RecSysAddressed Here
Traditionally, RecSys has focused on producing accurate recommendation algorithms. In this research, we address these challenges:
1.HCI: Implementation of visualizations that enhance users’ exploration of the items suggested.
2.Recommendation Tasks: Tackling exploration of recommendations, not only rating prediction or Top-N.
3.Meta-Analysis: Comparing results of different studies to generalizeresults.
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 5

6. Research Platform
•The studies were conducted using Conference Navigator, a Conference Support System
•Our goal was recommending conference talks
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 6
Program
Proceedings
Author List
Recommendations
http://halley.exp.sis.pitt.edu/cn3/

7. RELATED WORK OF VISUAL INTERFACES FOR RECSYS
Previous research related to this work / Motivating results from TalkExplorer study
10/06/2014
Verbet, Parra, & Brusilovsky .~ IntRs as
RecSys 2014
7

8. PeerChooser–CF movies
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 8
O'Donovan, J., Smyth, B., Gretarsson, B., Bostandjiev, S., & Höllerer, T. (2008, April). PeerChooser: visual interactiverecommendation

9. SmallWorlds–CF Social
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 9
Gretarsson, B., O'Donovan, J., Bostandjiev, S., Hall, C., & Höllerer, T. (2010, June). Smallworlds: Visualizingsocial recommendations.

10. TasteWeights–Hybrid Recommender
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 10
Bostandjiev, S., O'Donovan, J., & Höllerer, T. (2012, September). Tasteweights: a visual interactivehybridrecommendersystem

11. TALKEXPLORER: A GRAPH-BASED INTERACTIVE RECOMMENDER
11

12. TalkExplorer–IUI 2013
•Adaptation of Aduna Visualization to CN
•Main research question: Does fusion(intersection) of contexts of relevance improve user experience?
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 12

13. TalkExplorer-I
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 13
Entities
Tags, Recommender Agents, Users

14. TalkExplorer-II
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 14
Recommender
Recommender
Cluster with intersection of entities
Cluster (of talks) associated to only one entity
•Canvas Area: Intersections of Different Entities
User

15. TalkExplorer-III
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 15
Items
Talks explored by the user

16. Our Assumptions
•Itemswhicharerelevant in more that one aspect could be more valuable to the users
•Displaying multiple aspects of relevance visually is important for the users in the process of item’s exploration
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 16

17. TalkExplorerStudies I & II
•Study I
–Controlled Experiment: Users were asked to discover relevant talks by exploring the three types of entities: tags, recommender agents and users.
–Conducted at Hypertext and UMAP 2012 (21 users)
–Subjects familiar with Visualizations and Recsys
•Study II
–Field Study: Users were left free to explore the interface.
–Conducted at LAK 2012 and ECTEL 2013 (18 users)
–Subjects familiar with visualizations, but not much with RecSys
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 17

18. Evaluation: Intersections & Effectiveness
•What do we call an “Intersection”?
•We used #explorations on intersections and their effectiveness, defined as:
Effectiveness = |푏표표푘푚푎푟푘푒푑푖푡푒푚푠| |푖푛푡푒푟푒푠푒푐푡푖표푛푠푒푥푝푙표푟푒푑|
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 18

19. Results of Studies I & II
•Effectiveness increases with intersections of more entities
•Effectiveness wasn’t affected in the field study (study 2)
•… but exploration distribution was affected
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 19

20. Drawback: Visualizing Intersections
Clustermap
Venn diagram
•Venn diagram: more natural way to visualize intersections
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 20

21. SETFUSION: VENN DIAGRAM FOR USER-CONTROLLABLE INTERFACE
21

22. SetFusion–IUI 2014
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 22

23. SetFusionI
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 23
Traditional Ranked List
Papers sorted by Relevance.
It combines 3 recommendation approaches.

24. SetFusion-II
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 24
Sliders
Allow the user to control the importance of each data source or recommendation method
Interactive Venn Diagram
Allows the user to inspect and to filter papers recommended. Actions available:
-Filter item list by clicking on an area
-Highlight a paper by mouse-over on a circle
-Scroll to paper by clicking on a circle
-Indicate bookmarked papers

25. SetFusion – UMAP 2012
• Field Study: let users freely explore the interface
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 25
- ~50% (50 users) tried the
SetFusion recommender
- 28% (14 users) bookmarked at
least one paper
- Users explored in average 14.9
talks and bookmarked 7.36
talks in average.
A AB ABC AC B BC C
15 7 9 26 18 4 17
16% 7% 9% 27% 19% 4% 18%
Distribution of bookmarks per method or combination of methods

26. META-ANALYSIS
Description and Analysis of the results of the 3 user studies

27. TalkExplorervs. SetFusion
•Comparing distributions of explorations
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 27
In studies 1 and 2 over talkEplorerwe observed an important change in the distribution of explorations.

28. TalkExplorervs. SetFusion
•Comparing distributions of explorations
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 28
Comparing the field studies:
-In TalkExplorer, 84% of the explorations over intersections were performed over clusters of 1 item
-In SetFusion, was only 52%, compared to 48% (18% + 30%) of multiple intersections, diff. not statistically significant

29. CONCLUSIONS & FUTURE WORK

30. Summary of this Talk
•We presented two implementations of visual interactive interfaces that tackle exploration on a recommendation setting
•We showed that intersections of several contexts of relevance help to discover relevant items
•The visual paradigm used can have a strong effect on user behavior: we need to keep working on visual representation that promote exploration without increasing the cognitive load over the users
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 30

31. Limitations & Future Work
•Apply our approach to other domains (fusion of data sources or recommendation algorithms)
•For SetFusion, find alternatives to scale the approach to more than 3 sets, potential alternatives:
–Clustering and
–Radial sets
•Consider other factors that interact with the user satisfaction:
–Controllability by itself vs. minimum level of accuracy
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 31

32. THANKS! QUESTIONS? DPARRA@ING.PUC.CL

33. Mixed Hybridization: Item Score
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 33
M: is the set of all methods available to fuse
rankreci,mj: rank–position in the list of a recommended item
reci: recommended method i
mj, : recommendation method j
Wmj:weight given by the user to the method mjusing the controllable interface
|Mreci| represents the number of methods by which item reciwas recommended
Slider weight

34. Hybridization Methods (Burke 2002)
10/06/2014 Verbet, Parra, & Brusilovsky .~ IntRs as RecSys 2014 34
Hybridization
Description
Weighted
The scores (or votes) of several recommendation techniques are combined together to produce a single recommendation.
Switching
The system switches between recommendation techniques depending on the current situation.
Mixed
Recommendations from several different recommenders are presented at the same time
Feature combination
Features from different recommendation data sources are thrown together into a single recommendation algorithm
Cascade
One recommender refines the recommendations given by another.
Feature augmentation
Output from one technique is used as an input feature to another.
Meta-level
The model learned by one recommender is used as input to another.