The document discusses Netflix's research into how people perceive similarity in recommended content. It found that perceptions are influenced by three factors: 1) the person seeing the recommendations and their past experiences, 2) the current context, and 3) where the recommendations are placed. By accounting for these factors, Netflix was able to create a new similarity model that resulted in fewer perceived unreliable recommendations from users.

1. A Human Perspective on
Algorithmic Similarity
RecSys 2020
Zach Schendel, Faraz Farzin, & Siddhi Sundar
Netflix Product Innovation, Consumer Insights

2. Even though it was highly recommended by a Netflix
algorithm, I rejected this movie many times in 2017.
It just didn’t look or sound
good from the description.

3. Importantly, I like to click on “More Like This” because
I want to find out what unfamiliar movies are similar to.
I have seen a few of these.
They were decent, but the
connection between them
was not obvious

4. I eventually watched it after reading more about it
online. It remains my favorite Netflix Original movie!
...but, if I were to personalize
“More Like This”, I would include
similar titles that would be
explanatory and authentic for me
in that moment.

5. And… I wasn’t the only one who had an opinion about
similar recommendations on Netflix...

6. These and other examples got us thinking… what
exactly is similarity? How similar does something
need to be to describe another movie? When do
algorithms need to be restrictive, and when do they
have permission to be relaxed?
Where is the line between “similar” and “dissimilar”?
We decided to ask Netflix members!

7. We employed three different research methods.
An international landscape
assessment of how similarity is
used in and outside of the Netflix
category by other businesses.
Qualitative interviews with Netflix
members assessing how
appropriate and successful our
recommendations are in different
places where similarity could be
used as a driver for algorithms.
A quantitative evaluation of
perceived similarity between a series
of source titles and potential
recommendations using a technique
called inverse Multi-dimensional
Scaling (iMDS) [1].
More on this later...
[1] Kriegeskorte & Mur, 2012
1. 2. 3.

8. the Person
Who is seeing the recommendations, and what
is their past experience with the source?
the Context
What is going on at the moment? What are
the user’s needs?
1 2 3the Placement
Where are the recommendations placed within
the Netflix user interface?
The answer...it’s complicated! But we can really help our
recommendations! There are 3 sources of complexity:

9. the Person the Context
1 2 3the Placement
Where are the recommendations placed within
the Netflix user interface?

10. Many places in the Netflix interface are populated by
algorithms that factor in some dimension of similarity.

11. We found that members had higher expectations of
similarity when part of a 1:1 recommendation.
but… this doesn’t make much sense
at all. The Crown and The Office have
little in common.
This makes perfect sense. The Kissing
Booth and To All the Boys... are teen
romantic dramas with female leads.
Both cases are risky - there are no backups/other
titles to help explain the similarity.

12. But, there were lower expectations in places with
1:many recommendations, like while browsing.
Members are unlikely to say, “oh you liked Million Dollar Beach House? You should definitely watch Queer Eye.”
...but this broad placement is mostly made up of reality shows, so the link makes complete sense as a row.

13. And, like earlier, there were higher expectations in
placements that explicitly result from member action.
If they search for something specific
If they click into a specific title
and navigate to “More Like This”
Members are looking for something specifically similar.

14. the Person the Context
1 2 3the Placement
There is no one-size-fits all approach to employing
similarity signals in algorithms in all candidate
placements. Places that display 1:1
recommendations or react to explicit user
engagement have higher expectations of similarity.
Summary:

15. the Placement the Context
1 2 3the Person
Who is seeing the recommendations, and what
is their past experience with the source?

16. Method: We asked members to use iMDS[1] to
self-cluster content by similarity.
[1] Kriegeskorte & Mur, 2012
More similar =
closer together.
Will be added

17. Let’s walk through one example to illustrate what we
learned. Stranger Things is multidimensional.
Sci-fi . Fantasy . Teen . Horror . Thriller . 80’s . Nostalgia . Starring
Winona Ryder . Ominous . Scary . Exciting . Comedy . Coming of age

18. These movies are very similar to Stranger Things on
almost every one of those dimensions.
These high-similarity options would work well in 1:1 placements discussed above.
But, you quickly run out of options, and these won’t make up the bulk of placements.

19. Both broad and specific drivers of perceived similarity
can be used to fill out the bulk of placements.
Sci-fi . Fantasy . Teen . Horror . Thriller . 80’s . Nostalgia . Starring
Winona Ryder . Ominous . Scary . Exciting . Comedy . Coming of age

20. Broad similarity drivers were surface level, like genre.
They piqued interest and had clear source links.
Sci-fi . Fantasy . Teen . Horror . Thriller . 80’s . Nostalgia . Starring
Winona Ryder . Ominous . Scary . Exciting . Comedy . Coming of age

21. Specific similarity drivers were varied and difficult to
predict but were more salient final points of proof.
it stars Winona
Ryder!
‘80s nostalgia! Battling science
monsters!
Group of misfit
teens in the 80’s!

22. But...they are high-risk/high-reward. Trustbusters
often occur when a wrong or unclear link are chosen.
it stars Winona
Ryder!
‘80s nostalgia! Battling science
monsters!
Group of misfit
teens in the 80’s!

23. the Person
The degree to which something is or is not similar
is in the eyes of the beholder, all of whom might
latch on to different paths or give different
permissions. The best path for algorithms is to find
ways to balance broad and specific drivers of
similarity. Broadly similar recommendations that
differentially emphasize specific drivers for an
individual should minimize trustbusters.
the Context
1 2 3the Placement
There is no one-size-fits all approach to employing
similarity signals in algorithms in all candidate
placements. Places that display 1:1
recommendations or react to explicit user
engagement have higher expectations of similarity.
Summary:

24. the Placement the Person
1 2 3the Context
What is going on at the moment? What are
the user’s needs?

25. Finally, to add even more complexity, even if you hold
placement and person consistent, context also matters.
After you finish a movie or show,
Netflix recommends something.

26. You finished... ...try this next.
After finishing a show, members are most likely to
watch something similar.

27. 18%...watch another
reality show
#2 among options
But while this is certainly a common path, it is not
the most common or only path.
52%...watch another
Netflix Original
#1 among options
One example… After finishing a Netflix Original reality TV show...
55%...watch more
unserialized content
#1 among options

28. There are multiple contexts in which similarity is
unnecessary and in fact, the opposite of ideal.
“That was intense. I need
a change of pace,
something lighter”
“I don’t have time for
another movie. I’ll just put
on something short.”
… and more
“I just need to rewatch
something familiar, something
I have watched before.”

29. the Placement the Person
1 2 3the Context
Even if the placement and the person are
held constant, context further impacts the
perception or need for similarity. Algos
that can take context into consideration
(e.g., today it’s likely to be more of the
same day, tomorrow a change of pace) or
allow for balance across multiple paths
should be more successful.
Summary:

30. % Fewer Perceived Trustbusters comparing the old similarity
model to the new similarity model by similarity rank.
FEWERTrustbusters
In the end, the research validated a new similarity
model! There were fewer perceived trustbusters!

31. And, for me at least, these are better recommendations!
“More Like This” Similars

32. Thanks!
A Human Perspective on Algorithmic Similarity
RecSys 2020
Zach Schendel, Faraz Farzin, & Siddhi Sundar
Netflix Product Innovation, Consumer Insights

33. Slides for 3 minute intro video
A Human Perspective on
Algorithmic Similarity
RecSys 2020
Zach Schendel, Faraz Farzin, & Siddhi Sundar
Netflix Product Innovation, Consumer Insights

34. If you look at Twitter, there are
typically 3 types of comments.

35. I really liked this!
… I want more like it!1

37. uuuuh…
I don’t understand2

39. and…
a pleasant surprise!3

41. “More Like This” Similars
We are going to talk about how Netflix worked
directly with member research to get from this?!?...
?!?

42. ...to this!...
“More Like This” Similars!

43. the Person
Who is seeing the recommendations, and what
is their past experience with the source?
the Context
What is going on at the moment? What are
the user’s needs?
1 2 3the Placement
Where are the recommendations placed within
the Netflix user interface?
...by exploring how perceived similarity can be strongly
influenced by three new variables:

44. 20200903T225500Z__recsys__IN1032__a-human-perspective-on-algorit
SlidesLive ID