1. Making Sense of Search Result Pages
[Position Paper]
Jan O. Pedersen
Yahoo! Inc.
2811 Mission College Blvd
Santa Clara, CA 95054
jpederse@yahoo-inc.com
1. OVERVIEW ules, such as the Yahoo! movie direct display, inserted above,
Search engine result pages are presented hundreds of millions below or between results. The result presentations are some-
of times a day, yet it is not well understood what makes a times enriched with additional data, such as quick links for
particular page better from a consumer’s perspective. For homepages and in-line players for video content. In addi-
example, search engines spend large amounts of capital to tion, interventions in the process of constructing a query
make search-page loading latencies low, but how fast is fast are possible via search assistance technologies such as Ya-
enough or why fast is better is largely a subject of anecdote. hoo! Gossip.
Another example; search engines deploy large teams to con-
tinuously improve ranking quality, yet ranking quality does As the search industry moves forward, a deeper understand-
not fully characterize what is valued by the consumer. To ing of user searching behavior will be required to better
see this, consider a feature referred to as site collapse, the shape increasing complex products. The industry is fortu-
custom of clustering results from the same site, subresults nate to have an abundance of user-behavior data; every as-
indented below the main (most relevant) result. Common pect of the search experience is carefully metered. However,
measures of ranking quality, such as discounted cumulative this data is very complex in structure and captures only a
gain (DCG) [1], are not optimized by this feature, a seeming slice of Internet activity. An effective model of user interac-
contradiction. tion with the search result page, validated through qualita-
tive surveys and eye tracking experiments will be necessary
Much of the contradiction comes from imposing a optimiza- to make full use of this data.
tion criterion that does not account for perceptual phenom-
ena. Users rapidly scan search result pages and often make a 2. RESEARCH TOPICS
decision as to what action to take next within a few seconds. Yahoo! currently conducts research into user search be-
Presentations optimized for easy consumption and efficient havior through qualitative ethnographic studies, eye track-
scanning will be perceived as more relevant even if the con- ing experiments, analysis of query log data and live exper-
tent is identical to (or worse than) other, more awkward iments. Unfortunately, the largest scale, and hence richest,
displays. data source — the log data — is the hardest to interpret
since it is a partial, retrospective record of user actions. In
Eye-tracking experiments have shed considerable light on particular, simple, aggregate click-through data is not very
these issues [2]. It has been noted that users consume con- diagnostic because it averages over a huge range of phenom-
tent from top to bottom cycling through quick scan-cull- ena. Insights from user modeling are critical to both devel-
decide cycles. Presentation details are extremely determi- oping finer-grained, more revealing, user feedback measures
nant of whether a consumer will notice a result. For exam- and in developing tracking metrics that allow Yahoo! to
ple, the presence or absence of bolding can impact click rates monitor the health of its services.
significantly. Once a result is noticed, the decision to click is
driven by ancillary information in the result summary, such The process Yahoo! search uses to design, validate, and
as the contextual snippets and the url. optimize a new search feature includes the following steps:
Search result pages are becoming more complex; they can
no longer be characterized as a ranked list of homogeneously 1. Propose a design and build a mock-up or prototype
presented results. It is quite common to see content mod-
2. Translate the stated goals of the feature into expected
user behaviors
3. Conduct a usability test of the proposed feature; often
this includes a eye tracking experiment
4. Validate if the design achieves the desired goals; if not
iterate
5. Develop proxy measures for the desired behaviors that
can be measured in the user feedback logs

2. 6. Conduct an online test of the feature to quantify effects scanning behavior. One challenge is scaling up these exper-
iments, which are quite labor intensive, so that they can be
7. After launch, track the health of the feature through used not only to generate hypotheses but also to statistically
a feedback metric confirm them.
3. CONCLUSION
User modeling is critical for steps 2 and 5 and, of course,
We are still very early in the development of a comprehensive
reflects strongly on step 1. The next few sections describes
model for user interaction with the search results page. A
some of the the data and methods used to develop the Ya-
this time, we use operating principles, such as the hypothesis
hoo! search user model.
that users approach a search task with a fixed cognitive effort
budget or that bolding is critical is gaining user attention.
2.1 Session Analysis We will need more quantitative models to take the search
One rich source of insights into consumer search behavior is experience to the next level.
query log session analysis. A session is a sequence of user
actions, including query reformulations and url clicks, over a 4. REFERENCES
relatively short span of time associated with the same user [1] Kalervo J¨rvelin and Jaana Kek¨l¨inen, IR evaluation
a aa
goal. For example, one finds in the Yahoo query logs the methods for retrieving highly relevant documents,
following session fragments: SIGIR 2000, pages 41–48
[2] A.T. Duchowski, Eye Tracking Methodology: Theory
Lead query Follow query and Practice, Springer-Verlag, New York, Inc., 2003
samsung lcd tv best buy store [3] Yun-Fang Juan and Chi-Chao Chang, An Analysis of
samsung lcd tv consumer reports Search Engine Switching Behavior using Click
37 inch lcd tv best 37 inch lcd tv Streams, WWW 2005, Chiba Japan, pages 1050–1051
shelving units costco wholesale
shelving units home library
The user’s intention is made clearer in the reformulation.
Joining this data with user clicks actions is even more infor-
mative. Yahoo! actively mines session data to build search
assists, spell correct queries, and to automatically expand
queries (in sponsored search). A newer area is mining this
data to measure the effectiveness of ranking and presenta-
tion algorithms, with the underlying assumption that better
selection and ranking algorithms will anticipate user refor-
mulations. For example, one study examines conditions un-
der which users switch from one search engine to another [3].
However, the surface has only been scratched in the analysis
of session data.
2.2 Toolbar data
Search engine query logs only reflect a small slice of user
behavior — actions taken on the search results page. A
more complete picture would include the entire click stream;
search result page clicks as well as offsite follow-on actions.
This sort of data is available from a subset of toolbar users
— those that opt into having their click stream tracked.
Yahoo! has just begun to collect this sort of data, although
competing search engines have collected it for some time.
We expect to derive much better indicators of user satis-
faction by consider the actions post click. For example, if
the user exits the clicked-through page rapidly then one can
infer that the information need was not satisfied by that
page. User satisfaction indicators can then be correlated
with search result page context to shed light on usefulness.
2.3 Eye Tracking
Ultimately a direct measurement of what the user perceives
on a search results page would be most useful. A closely
proxy of this ideal is measuring eye movement and fixations
across the page followed by user interviews. Aggregate mea-
sures reveal which page elements get the greatest attention,
while tracking individual sessions is very revealing about