Published in ACM TiiS: Verbert, K., Parra, D., & Brusilovsky, P. (2016). Agents Vs. Users: Visual Recommendation of Research Talks with Multiple Dimension of Relevance. ACM Transactions on Interactive Intelligent Systems (TiiS), 6(2), 11. Presented at IUI 2017

1. Agents vs Users: Visual
Recommendation of Research Talks
with Multiple Dimension of Relevance
Katrien Verbert – KU Leuven – @katrien_v
Denis Parra – PUC Chile - @denisparra
Peter Brusilovsky – University of Pittsburgh - @peterpaws
Presented at IUI 2017

2. Motivation
• multiple relevance prospects in
personalized social tagging
systems
o community relevance prospects
o social relevance prospect
o content relevance prospect
• existing personalized social
systems
o do not allow to explore and combine
multiple relevance prospects
o only one prospect can be explored at a given
time
2

3. Also recommendations  personalized
relevance prospect
3

4. Shortcomings recommender systems
• cold-start issues
• difficult to explain the rationale behind recommendations
• user control has positive effect
o several ways to control elements
o which are most effective for the user experience?
4

5. 5
He, C., Parra, D., & Verbert, K. (2016). Interactive recommender systems: A survey of the state
of the art and future research challenges and opportunities. Expert Systems with Applications,
56, 9-27.

6. John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: visual
interactive recommendation. CHI '08
Related work: PeerChooser
6

7. Related work:
Smallworlds
Gretarsson, B., O'Donovan, J., Bostandjiev, S.,
Hall, C. and Höllerer, T. SmallWorlds: Visualizing
Social Recommendations. Comput. Graph. Forum,
29, 3 (2010), 833-842. 7

8. Related work: TasteWeights
Bostandjiev,S.,O'Donovan,J.andHöllerer,T.TasteWeights:avisualinteractivehybrid
recommendersystem.InProceedingsofthesixthACMconferenceonRecommender
systems(RecSys'12).ACM,NewYork,NY,USA(2012),35-42.
8

9. Contributions
• new approach to support exploration, transparency and
controllability
o recommender systems are shown as agents
o in parallel to real users and tags
o users can interrelate entities to find items
• evaluation study that assesses
o effectiveness
o probability of item selection
9

10. Conference Navigator
The studies were conducted using Conference Navigator 3
http://halley.exp.sis.pitt.edu/cn3/10

11. TalkExplorer - I
11
Entities
Tags, Recommender Agents,
Users

12. TalkExplorer - II
Recommender
Recommender
Cluster
with
intersect
ion of
entities
Cluster (of
talks)
associated
to only one
entity
Canvas Area: Intersections of Different Entities
User

13. TalkExplorer - III
13
Items
Talks explored by the
user

14. Our Assumptions
• Items which are relevant in more that one aspect could be
more valuable to the users
• Displaying multiple aspects of relevance visually is
important for the users in the process of item’s exploration
14

15. User study 1
• Setup
o supervised user study
o 21 participants at UMAP 2012 and ACM Hypertext 2012 conferences
• Procedure
o Tasks
• interact with users and their bookmarks
• interact with agents
• interact with tags
o Post-questionnaire
15

16. Evaluation
• Data collection
o recordings of voice and screen using camtasia studio
o system logs
• Measurements
o effectiveness: # bookmarked items / #explorations
o yield: : # bookmarked items / sum of items in selection set
16

17. Effectiveness
results
Summary explored actions
and their effectiveness
Effectiveness = # bookmarked
items / #explorations
17

18. Summary results
Sign. effect. Sign.
yield
multiple versus one entity 0.003 <0.001
user versus (user + entity) 0.593 <0.001
agent versus (agent + entity) 0.341 <0.001

19. Post-questionnaire
19

20. Post-questionnaire
20

21. User study 2
• Setup
o Unsupervised user study
o Conducted at LAK 2012 and ECTEL 2013 (18 users)
o Subjects familiar with visualizations, but not much with RecSys
• Procedure
o Users were left free to explore the interface.
o Interactions were logged
o Post-questionnaire
21

22. Summary results
22
Sign. effect.
multiple versus one entity <0.001
user versus (user + entity) 0.3682
agent versus (agent + entity) 0.4426

23. Summary results
Sign. effectiveness
ranked list versus agents 0.7008
ranked list versus (agent + entity) <0.001
ranked list versus multiple entities 0.0012
23

24. Results of Studies 1 & 2
• Effectiveness increases
with intersections of more
entities
• Effectiveness wasn’t
affected in the field study
(study 2)
• … but exploration
distribution was affected
Average effectiveness
Total number of explorations
24

25. Drawback: visualizing intersections
Clustermap Venn diagram
Venn diagram: more natural way to visualize intersections
25
Verbert, K., Parra, D., Brusilovksy, P. (2014). The effect of different set-based visualizations on user
exploration of recommendations. In : IntRS@RecSys, 2014 (pp. 37-44).

26. IntersectionExplorer (IEx)
26
Verbert, K., Seipp, K., He, C., Parra, D., Wongchokprasitti, C., Brusilovsky, P. (2016). Scalable exploration of relevance
prospects to support decision making. In : IIntRS@RecSys. 2016 (pp. 28-35). CEUR-WS.

27. IntersectionExplorer (IEx)
27

28. Feedback very welcome!
Please participate in a short user study:
http://halley.exp.sis.pitt.edu/cn3/iestudy3.php?conferenceID=
148
28

29. Thank you!
Katrien.verbert@cs.kuleuven.be
peterb@pitt.edu
denisparra@gmail.com
29