The Australian Search Experience Project collects data from citizen scientists using a browser plugin to analyze how search results differ for various search platforms and groups over time. Preliminary results show Google Search results are generally stable while Google News is fast-changing. Limited personalization is seen based on location and device type. Future plans include more detailed analysis of each platform, non-organic results, and search quality for different demographic and event-based topics. The project aims to better understand information inequality and filter bubbles.

1. THE AUSTRALIAN
SEARCH EXPERIENCE
PROJECT
ADM+S News & Media Symposium 2021

2. Search results and personalisation:
• Are search results personalised? If so, how?
• Does this produce ‘filter bubbles’ and information inequality?
• How do search results for emerging topics evolve over time?
Browser plugin:
• Queries Google Search, Google News, Google Video, YouTube
• Drawing on the profiles Google has assembled for participants
• Presents as desktop or mobile browser
• Data donation philosophy, involving the general public
• Building on 2017 AlgorithmWatch study in Germany
• Launched in late July 2021
Axel Bruns, Jean Burgess, Nicolas Suzor, Mark Andrejevic, Kim Weatherall,
James Meese, Damiano Spina, Tim Graham, Daniel Angus, Falk Scholer,
Abdul Obeid + AlgorithmWatch
https://www.admscentre.org.au/searchexperience/
AUSTRALIAN SEARCH EXPERIENCE
ADM+S News & Media Symposium 2021

3. admscentre.org.au/
searchexperience

4. ADM+S News & Media Symposium 2021

5. DEMOGRAPHICS
TO DATE
5
ADM+S News & Media Symposium 2021

7. PRELIMINARY
RESULTS
7
ADM+S News & Media Symposium 2021

8. Calculating rankflow
For a specific search term on a given day,
1. take all donated results up to a maximum list length (e.g. 10),
2. filter for results that appear in at least n donations (e.g. 50),
3. calculate the average list position at which they appear,
4. list them in order of average list position, per day and over time.
Extensions
Standard deviation: across all donated search results for the same
day, how varied is the list position of a given item?
Demographic distinctions: do different participant groups see
different lists of results? Why?
Technological distinctions: do desktop and mobile users see
different lists of results? Why?
How stable are the search results seen for the same search term by
different users of the same platform from day to day?
This measures the change of search results over time, both in
general and for specific groups of users as defined by demographics
and other personal attributes.
Limitations for now:
● first page of 10 (Google Search, News, Video) or 20 (YouTube)
results only
● organic results only (no promoted links, side boxes, etc.)
● non-representative demographics of participants
Further possible extensions:
● shorter/longer timeframes of analysis
● results variation within specific demographic groups
● results variation across different browser types (desktop/mobile)
● non-organic search results and other elements
● domain- rather than URL-level analysis
8
SEARCH RANKINGS OVER TIME
ADM+S News & Media Symposium 2021

9. Each line represents
a single URL.
URLs are ranked by
their average list
position on the day.
Line thickness indicates
standard deviation: the thinner
the line, the more stable the list
position on the day.
Search keyword for
this example:
Barnaby Joyce.
Dots mean the URL
appeared only on that
day (e.g. news articles).
The longer the list for each day, the more
diverse the results encountered by
participants. Short lists mean everyone
saw the same results, all day long.

21. Calculating variability
For a specific search term on a given day,
1. take all donated results with the same list length (e.g. 10),
2. order them by frequency of occurrence,
3. count how many of them it takes to account for 80% of all
results seen by our participating users,
4. divide 80% of the list length (e.g. 10 × 80% = 8) by that count,
5. subtract that value from 1 to produce a scale of
0 (no variability) to 1 (maximum variability).
Examples
20 participants report seeing the exact same 10 results on the day.
Any 8 of those 10 are enough to account for 80% of all 200 results
donated. Our variability is 1 – (8/8) = 0.
20 participants report seeing entirely different 10 results on the day.
To account for 80% of all 200 results we need 160 donated results.
Our variability is 1 - (8/160) = 0.95.
How diverse are the search results seen for the same search term
by different users of the same platform on the same day?
This measures the level of personalisation based on demographics
and other personal attributes.
Limitations for now:
● first page of 10 (Google Search, News, Video) or 20 (YouTube)
results only
● organic results only (no promoted links, side boxes, etc.)
● arbitrary threshold of 80% in variability calculation
Further possible extensions:
● shorter/longer timeframes of analysis
● results variation within specific demographic groups
● order of search results received
● non-organic search results and other elements
21
INTRA-DAY VARIABILITY
ADM+S News & Media Symposium 2021

22. 1. Search results
Search for ‘mortgage broker’
on 3 Sep. 2021.
Results with a list length of 9.
2. Frequency of occurrence
Top URL seen 250 times, etc.
Total count: 2,259 results.
 Target: 2,259 × 80% = 1,807
3. Cumulative count
Adding the frequencies for
each search result, we reach
1,807 after item #31.
Variability calculation:
1 – (9 × 80%) / 31 = 0.77

28. FIRST
OBSERVATIONS
28
ADM+S News & Media Symposium 2021

29. Very different patterns across platforms:
• Google Search largely stable.
• Google News very fast-moving.
• Google Video quite static.
• YouTube often stable in top ~5 results, then highly changeable.
Limited evidence of search personalisation so far:
• Personalisation in Google Search largely driven by user location.
• Critical search topics appear manually curated, at least in part.
• Some results differences based on browser type (desktop vs. mobile).
• Other platforms (especially YouTube) need further detailed analysis.
EARLY INSIGHTS
ADM+S News & Media Symposium 2021

30. FUTURE
PLANS
30
ADM+S News & Media Symposium 2021

31. Further analysis:
• Detailed analyses per platform, and cross-platform comparisons.
• Further breakdown by demographic attributes and browser features.
• Extension to non-organic search results.
• Evaluation of search result quality.
Additional outreach:
• Demographic profile of citizen scientist cohort still uneven.
• Need to compensate for participant attrition over time.
• Gradual variation of search terms to address emerging topics.
• Focus especially on major upcoming events – e.g. federal election.
NEXT STEPS
ADM+S News & Media Symposium 2021

32. ADM+S News & Media Symposium 2021
THANKS
Contact for more information
Prof. Axel Bruns
a.bruns@qut.edu.au — @snurb_dot_info
www.admscentre.org.au