Every month, the English-language Wikipedia is edited 2.9 million times. Is anyone reading all these changes? How does Wikipedia decide what information is trustworthy and what is better left out? In this session, we'll look at how Wikipedia works behind the scenes to scrutinize new content, and how this challenge has shaped Wikpedia over time. Without any easy formulas to decide whether Wikipedia articles are reliable, we'll see how an understanding of Wikipedia's inner workings can help us use it with more educated skepticism. Originally presented to the University of Victoria L.E.A.D. Lab in December, 2014

1. Trusting Wikipedia
Su-Laine Yeo Brodsky
December 2014

2. Today’s Talk
• How does
Wikipedia vet its
own content?
• Questions are
welcome
anytime, or at the
end
Image:Robert Lawton, distributed under a CC BY-SA 2.5 license

3. The Encyclopedia Anyone
Can Edit
• Hosted by the non-profit
Wikimedia Foundation,
in 250+ languages
• Almost anyone is
encouraged to edit
almost all text and
media on the site
• ~10% of edits are
reverted (rejected)
Image: xkcd.com, distributed under a CC BY-NC 2.5 license

4. What Happens When You
Edit This Page?

5. Right Away

6. Exceptions: Edits You Can’t
Save
• URLs on Wikipedia’s spam blacklist
• Use of certain images, except in particular articles
• Certain types of blatant vandalism

7. Page History

9. User Contributions Page
Accounts used only
for vandalism are
easy to spot

10. Summary: Wiki Mechanics
• All edits are tracked, and
summarized in the article
History page
• All edits can be tracked to an
IP address or username
• Edits can be quickly
reverted by another editor

11. People and Processes

12. Roles at Wikipedia
• Editor: Anyone who writes or changes articles, or
uploads images
• Administrator: 1,386 elected administrators have
additional tools:
• Can delete/undelete articles or past versions of
articles
• Can block editors or other administrators
• Can “protect” an article to prevent editing

13. Reality: For the most part,
everybody has equal privileges
when it comes to controlling article
content
Assumption: Moderators decide
what changes to an article will be
accepted

14. How Wikipedia Monitors
Edits: A 2-Stage Process

15. Stage 1: Recent Changes
Page
30 seconds worth
of edits to all
English Wikipedia
articles

16. Link to “diff”
showing what text
was changed in this
edit
Link to list of editor’s
other edits
Heading of the article
section that was
changed
Editor’s
explanation for
the edit
Red indicates
editor is new

17. Recent Changes Patrol
• Volunteer patrollers and
robots monitor all
changes to the site
• Quickly revert blatantly
inappropriate changes

18. Stage 2: Watchlists
Changes to articles
I’ve chosen to
watch, grouped by
date

19. Watchlists
• Each user has a watchlist of
articles they are interested in
• The Watchlist page shows
recent changes in those
articles
• Primary tool for fact-
checking

20. Will an Edit Stick?
• Usually decided by the editors who watch the
page
• If no consensus, the editors who watch the page
ask the wider Wikipedia community for input and
mediation

21. Reality: Wikipedia articles summarize
what reliable published sources say
on the subject…
…as determined by the consensus of
editors who show up
Assumption: Wikipedia articles
summarize what a majority of its
editors on the subject believe

22. How Editors Scrutinize
Changes
Consider:
• My own knowledge
• Sources cited?
• Who is the editor?
• What does the source say?
Userpage of a Wikipedia
editor
Image: Userpage of Kim Dent-Brown on Wikipedia
• Improve instead of reverting

23. Dispute Resolution
• Discuss the issue on
the article Talk Page
• Ask the wider
community for input
• Problematic editors
can be blocked by
administrators

24. What Can Go Wrong?
• Patrollers/watchlisters
not paying attention
• Watchlisters lacking the
necessary expertise
• Persistent, highly
motivated agenda-
pushers

25. Perspective: Core Facts
In an article with a reasonable number of watchers, the core
facts of the article tend to be reliably monitored.
“West Bengal
is a state in
the eastern
region of
India.”
(no source given)

26. Wikipedians: Question 1
Wikipedia currently has 31,000 active, registered
editors. According to surveys of Wikipedia editors,
what percentage are male?
A. 85%
B. 60%
C. 45%

27. Wikipedians: Question 2
Which of the following statements is false?
A. According to editor surveys, more than 10% of
Wikipedia editors are under 18 years old
B. Anyone can track down the IP address that
each edit comes from
C. According to editor surveys, about half of
Wikipedia editors have some post-secondary
education

28. Systemic Bias
• Demographic bias:
• Technical
• Male
• Childless
• First world
• Recentism: Recent
news coverage &
online sources

29. Summary: People and
Processes
• All editors are equal, in
theory
• Articles should draw only
from reliable published
sources
• 2-stage review process
means that subtle problems
last longer than obvious
ones
Image: Adam Novak, distributed under a CC BY-SA 3.0 license
Wikimania Conference 2013

30. Summary (cont’d)
• In an article with a
reasonable number of
watchers, the core facts
tend to be relatively
reliable
• Wikipedia has systemic
biases stemming from
community demographics

31. Trends in Trust
Image: Mariuszjbie, distributed under a CC BY-SA 3.0 license

32. How Many Watchers?
Article
# of
Watchers
Barack Obama 2,592
Angela Merkel 225
Influenza 228
Cephalexin 46
Many articles on living people <5

33. Four Trends
1. More complaints from the subjects of Wikipedia
articles, even articles with very low readership
2. Higher expectation to cite sources when adding
new content
3. More complete content -> Ratio of good to bad
edits changes
4. More sophisticated PR from organizations

34. Problematic Edits in Medical
Articles
Source: https://en.wikipedia.org/wiki/Wikipedia:Conflicts_of_interest_%28medicine%29

35. Looking Out for Bias
• Would any personal,
corporate, or ideological
interest benefit from a
certain article slant?
• Does the article Talk page
and/or history show any
concerns with conflict-of-
interest editing or disputes
over neutrality?
Buy!

36. • Libel
• Hoax
• Advertising
• Pseudoscience
• Propaganda
Don’t bite the
newbies
Reject questionable
edits
Many edits by new
editors are imperfect
and do not cite
sources

38. A Reversible Trend?
• Community needs to
grow to:
• Maintain quality and
update facts
• Reduce systemic bias
• Requires Wikipedia
community to be skilled
in both skepticism and
openness

39. Enjoy Wikipedia
• Reliability is variable
• Project reflects huge
volunteer effort
• Future depends on a
strong editor
community
Image: Takeaway, distributed under a CC BY-SA 3.0 license

40. Thank You!
Any questions?
Su-Laine Yeo Brodsky
syeo@interelement.ca
www.interelement.ca
@sulaineyeo