1. Is it Time to Abandon Abandonment?!
Abhimanyu Lad, Daniel Tunkelang!
LinkedIn!
Problems
with
click-­‐based
measures
of
search
performance
TRUST
BIAS
CLICKS
DON’T
ALWAYS
INDICATE
RELEVANCE
MORE
CLICKS
NOT
ALWAYS
BETTER!
Users
are
more
likely
to
click
on
the
top-­‐ranked
Users
might
click
on
a
result
out
of
curiosity,
or
Interpreta*on
depends
on
the
query
type:
results
irrespec*ve
of
their
relevance.
by
mistake,
e.g.,
due
to
a
misleading
snippet.
Naviga;onal
query:
[ceo
facebook]
More
clicks
signify
inability
to
find
the
correct
result.
Click-­‐through
rate
Exploratory
query:
[java
programmer
boston]
More
clicks
signify
mul4ple
relevant
results.
Wait,
what
is
this
guy
doing
in
my
search
results?
Rank
Solu;on:
Move
beyond
“raw”
interpreta;on
of
clicks
as
relevance!
DIFFERENTIATE
BETWEEN
DETECT
SEARCH
SESSIONS
CONSIDER
ACTIONS
PERFORMED
GOOD
AND
BAD
CLICKS
ON
SEARCH
RESULTS
Users
o'en
sa*sfy
their
informa*on
needs
Use
indicators
like
*me
spent
or
ac*ons
through
a
sequence
of
related
queries.
Ac*ons
like
sending
a
message,
bookmarking,
performed
on
landing
page,
presence
of
sharing,
etc.
are
more
robust
indicators
of
task
subsequent
clicks,
or
query
reformula*ons.
comple*on,
and
thus,
user
sa*sfac*on.
[john
shear]
[john
shearer]
CONSIDER
DWELL
TIME
Users
are
likely
to
spend
more
*me
on
relevant
[john
shearer
intel]
landing
pages.
[
…
]
New
Evalua;on
Measures
ROBUST
VERSIONS
OF
SEARCH
UTILITY
LOOK
AT
MULTIPLE
METRICS
COMMONLY-­‐USED
MEASURES
FOR
COMPLETE
PICTURE
(query
types)
Abandonment:
Gain
Percentage
of
search
sessions
with
no
good
clicks
Effort
Session
Abandonment
Mean
Reciprocal
Rank
(MRR):
Reciprocal
rank
of
first
or
last
good
click
in
each
Gain
=
good
clicks,
ac4ons,
dwell
4me,
…
session
Effort
=
sequen4al
scan,
4me
spent
on
Probability
of
Skip
(pSkip):
search
results
page,
reformula4on
Ra4o
of
number
of
results
skipped
(including
effort,
face4ng
effort,
…
bad
clicks)
and
number
of
results
examined
Session
length
For
further
informa4on:
Abhi
Lad
<alad@linkedin.com>
or
h@p://www.linkedin.com/in/abhilad