- Machine learning algorithms used in recommendation systems must be made more transparent to users. Users currently do not understand how the systems work and feel they have no control over the recommendations. - Interviews with users revealed fears about data privacy and a lack of understanding of how user interactions influence recommendations. Users have not formed clear mental models of how recommendation systems track and use their data. - User experience designers need to help users build better mental models of recommendation systems and deconstruct the "black box" feeling. Designers must also encourage users to provide feedback and give them more control over their data and recommendations.

1. RECOMMENDATION SYSTEMS
AND MACHINE LEARNING
Mapping the User Experience
LUIZ AGNER | BARBARA NECYK | ADRIANO RENZI
Rio de Janeiro

2. • Making a choice is a challenging and
complicated task because of the amount
of information.
• Machines are more efficient than human beings
in managing data and in learning from this data.
INFORMATION
OVERLOAD
Gomez-Uribe; Hunt: The Netflix recommender system: Algorithms, business value, and innovation.

3. • Employed in:
• Netflix,
• YouTube,
• Spotify,
• Amazon,
• Facebook,
RECOMMENDATION
SYSTEMS
•Linkedin,
•Twitter,
•Trip Advisor,
•Google News,
•Others
Rocca, B.: Introduction to recommender systems. Pandey, P.: The Remarkable world of Recommender Systems.

4. RECOMMENDATION
SYSTEMS
• Prediction
data is used to predict the evaluation
a user will give to an item.
• Ranking
defines a list of items to be
presented to the user.
PROBLEMS:
Pandey, P.: The Remarkable world of Recommender Systems.

5. RECOMMENDATION
SYSTEMS
• Relevance
• Novelty
• Serendipity
• Diversity
GOALS
Pandey, P.: The Remarkable world of Recommender Systems.

6. • Methods are based on past interactions
among users and items.
• Can recommend complex items without
an understanding of its content.
COLLABORATIVE
FILTERING
Rocca, B.: Introduction to recommender systems.

7. • Need additional information about
users and items.
• Need a description of each item
and a profile of user preferences.
CONTENT-
BASED FILTERING
Rocca, B.: Introduction to recommender systems.

8. Netflix is an excellent example of hybrid
recommendation systems because its
recommendations are based not only on
browsing habits but also on similar videos.
HYBRID
SYSTEMS
Pandey, P.: The Remarkable world of Recommender Systems.

9. • ML is the science of helping computers discover
patterns and relationships in data.
• UX designers have a lot to learn about ML
to help users feel in control of the technology.
• Users can not make the systems do what they want.
USER EXPERIENCE
& MACHINE LEARNING
Lovejoy, J., Holbrook, J.: Human-Centered Machine Learning.

10. • Algorithms are not transparent to users.
• Users don’t identify actions and do
not have a clear understanding of the outputs.
• Recommendations seem random, meaningless.
• Algorithms group items according to obscure
and not mutually exclusive rules.
USER EXPERIENCE
& MACHINE LEARNING
Budiu, R.: Can Users Control and Understand a UI Driven by Machine Learning?

11. • Users perceive the system as a “black box”.
With mysterious inputs and outputs out of control.
• Users need developing a clear mental model of how the system works.
The system needs to be clear about how users can change its results.
• Repeated recommendations - and session-specific arts,
descriptions and headings - increase the cognitive cost of interaction.
USER EXPERIENCE
& MACHINE LEARNING
Budiu, R.: Can Users Control and Understand a UI Driven by Machine Learning?

12. USER EXPERIENCE
GUIDELINES
• Transparency
• Easy controls
• Don’t repeat items
Budiu, R.: Can Users Control and Understand a UI Driven by Machine Learning?

13. MORE UX
GUIDELINES
• Transparency of data source
• Improvement of recommendations by users
• Response time
Harley, A.: UX Guidelines for Recommended Content.

14. HUMAN CENTRED AI
GUIDELINES
• Microsoft: Guidelines for Human-AI Interaction.
• Comprehensibility and controllability.
Amershi, Horvitz, et al. Microsoft - Guidelines for human-AI interaction design

15. • Personalised Video Ranker (PVR)
• Top-N Video Ranker
• Trending Now
• Video-Video Similarity (Sims)
• Continue Watching
• Evidence
• Page Generation: Row Selection and Ranking
• Search
NETFLIX ALGORITHMS
Gomez-Uribe; Hunt: The Netflix recommender system: Algorithms, business value, and innovation.

16. SEMI-STRUCTURED
INTERVIEWS
• What influences users to find new video.
• How users identify where their recommendations are.
• How users explain the way Netflix's algorithms work.
• What users could do to improve their recommendations.
• About data protection and privacy.
Courage, C., Baxter, K.: Understanding Your Users: A Practical Guide…

17. USERS ANSWERS

18. USERS ANSWERS
“Conspiracy Theories”

19. • Human-algorithm interaction emerges as a new frontier of research.
• Machine learning algorithms must be transparent.
• Users do not know which interactions create the recommendation lists.
• Users have not formed a mental model about tracked data.
• Designers must help users build mental models.
• Designers must help users deconstruct the “black box” feeling.
CONCLUSIONS & NOTES

20. • Fears of political surveillance, restriction of the
information freedom, and credit card hacking.
• “Conspiracy theories” appeared.
• UX designers should encourage users to record
feedbacks and edit user navigation data.
• UX designers have a lot to learn.
• UX designers have to reinforce user’s control.
CONCLUSIONS
& NOTES

21. Pandey, P.: The Remarkable world of Recommender Systems. https://towardsdatascience.com/the-
remarkable-world-of-recommender-systems-bff4b9cbe6a7, last accessed 2020/01/28.
Gomez-Uribe, C.A.; Hunt, N.: The Netflix recommender system: Algorithms, business value, and
innovation. In: ACM Transactions Manage. Inf. Syst. 6, 4, Article 13. ACM, New York (2015).
Rocca, B.: Introduction to recommender systems. https://towardsdatascience.com/introduction-to-
recommender-systems-6c66cf15ada 1/22, last accessed 2019/12/19.
Budiu, R.: Can Users Control and Understand a UI Driven by Machine Learning? Nielsen Norman
Group, https://www.nngroup.com/articles/machine-learning-ux/, last accessed 2020/01/02.
Harley, A.: Individualized Recommendations: Users’ Expectations & Assumptions. https://
www.nngroup.com/articles/recommendation-expectations/, last accessed 2019/11/28.
Harley, A.: UX Guidelines for Recommended Content. https://www.nngroup.com/articles/
recommendation-guidelines/, last accessed 2019/11/29.
Lovejoy,J.,Holbrook,J.: Human-Centered Machine Learning. https://medium.com/google- design/
human-centered-machine-learning-a770d10562cd, last accessed 2019/12/10.
Netflix Research.: Machine Learning: Learning how to entertain the world, https://re-
search.netflix.com/research-area/machine-learning, last accessed 2019/12/28.
Basilico, J.: Recent Trends in Personalisation: a Netflix Perspective. ICML2019-Adaptative and Multi-
Task Learning Workshop, https://slideslive.com/38917692/recent-trends-in- personalization-a-netflix-
perspective, last accessed 2020/01/10.
Courage, C., Baxter, K.: Understanding Your Users: A Practical Guide to User Requirements Methods,
Tools, and Techniques. Morgan Kaufmann, San Francisco (2005).
REFERENCES

22. THANK YOU.
luizagner@gmail.com