WikiSym 2012 presentation

1. Digital Enterprise Research Institute www.deri.ie
Deletion Discussions in Wikipedia*:
Decision Factors and Outcomes
Jodi Schneider, Alexandre Passant, & Stefan Decker
WikiSym 2012 Wednesday 29th August 2012
Linz, Austria
*enWP
Copyright 2011 Digital Enterprise Research Institute. All rights reserved.
Enabling Networked Knowledge
1

2. Big questions about WP
Digital Enterprise Research Institute www.deri.ie
 Is crowdsourcing sustainable?
 Is content bias manageable?
 Does it matter who writes WP?
 How can newcomers be welcomed and socialized?
Enabling Networked Knowledge
2

3. … are related to Deletion
Digital Enterprise Research Institute www.deri.ie
 Is crowdsourcing sustainable?
 How do we maintain content through deletion?
 Is content bias manageable?
 Are new articles needed? Are they welcomed?
 Does it matter who writes WP?
 … or who makes deletion decisions?
 How can newcomers be welcomed and socialized?
 Deletion threatens editor retention
– 1 in 3 editors begin by creating a new article
– 7 times as likely to stay if their article is kept
Source: [[User:Mr.Z-man/newusers]] via
[[Wikipedia:Wikipedia_Signpost/2011-04-04/Editor_retention]]
Enabling Networked Knowledge
3

4. Overall Goals
Digital Enterprise Research Institute www.deri.ie
 Understand outcomes of deletion discussions
 What are good outcomes for articles?
 ... for the community?
 Provide support to various groups
 Readers/New Editors
 Debate Closers
 People Reading Archived Debates
Enabling Networked Knowledge
4

5. This Study’s Research
Questions
Digital Enterprise Research Institute www.deri.ie
1. What factors contribute to the decision about whether
to delete a given article?
2. When multiple factors are given, what is the relative
importance of those factors?
3. What are the outcomes of deletion discussions, both
for articles and for the community?
Enabling Networked Knowledge
5

6. Overview
Digital Enterprise Research Institute www.deri.ie
 Outcomes (RQ3)
 Data, Methods, Previous Research
 Factors (RQs 1&2)
 Future Work on Support (Demo)
Enabling Networked Knowledge
6

7. Articles: Good Outcomes
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
7

8. … Content Expansion
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
8

9. Good Rationale
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
9

10. Good Outcome?
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
10

11. Community: Good Outcomes
Digital Enterprise Research Institute www.deri.ie
 Learning to argue effectively
 Becoming more detached from content
 Introducing new editors to community values
 Developing new editors’ editing skills
Enabling Networked Knowledge
11

12. Example: Good
Community Outcomes
Digital Enterprise Research Institute www.deri.ie
 William Vickers (fiddler)
 1 main author – their first article
 Nominated for deletion after 1 hour and 20 minutes
 Shaped during the process
Enabling Networked Knowledge
12

13. Changes During AfD
Digital Enterprise Research Institute www.deri.ie
 Article renamed to William Vickers manuscript
 Discography added
 26 edits from this author
Enabling Networked Knowledge
13

14. Supporting the Editor
Digital Enterprise Research Institute www.deri.ie
 First article this editor created.
 Overall 11 articles later created by this editor.
 Creator made many more edits to this article.
 26 edits, compared to 3-9 edits to his later articles.
Enabling Networked Knowledge
14

15. 15

16. Overview
Digital Enterprise Research Institute www.deri.ie
 Outcomes (RQ3)
 Data, Methods, Previous Research
 Factors (RQs 1&2)
 Future Work on Support (Demo)
Enabling Networked Knowledge
16

17. Discussion-based Deletion
Digital Enterprise Research Institute www.deri.ie
“Articles for Deletion” (AfD)
 Most contentious
 Articulated decision-making
 500+ deletion discussions/week
 ~12% of deletions Lam & Riedl. “Is Wikipedia growing a longer tail?”
GROUP ’09
Enabling Networked Knowledge
17

18. Dataset
Digital Enterprise Research Institute www.deri.ie
 Data Corpus: “Typical Day”
 72 deletion discussions
 January 29, 2011
 English Wikipedia only
Enabling Networked Knowledge
18

19. Methods
Digital Enterprise Research Institute www.deri.ie
Deep analysis of a moderate-sized dataset
 Representative sample
 Intensive manual analysis
 Annotation with multiple coders
 Descriptive statistics
Enabling Networked Knowledge
19

20. Previous Research
Digital Enterprise Research Institute www.deri.ie
Shallow analysis of large datasets
 Redacted content
– West & Lee, “What Wikipedia deletes” WikiSym 2011
 Vote sequencing
– Taraborelli & Ciampaglia “Beyond notability” SASOW 2011
 Decision quality
– Lam, Karim & Riedl “The effects of group composition on decision
quality in a social production community”, GROUP 2010
 Who participates, what & how much gets deleted
– Priedhorsky, Chen, Lam, Panciera, Terveen, & Riedl. “Creating,
destroying, and restoring value in Wikipedia”, GROUP 2007
– Geiger & Ford “Participation in Wikipedia’s article deletion processes”,
WikiSym 2011
Enabling Networked Knowledge
20

21. From Reading to Editing
Digital Enterprise Research Institute www.deri.ie
 How can newcomers be welcomed and socialized?
 Deletion threatens editor retention
– 1 in 3 editors begin by creating a new article
– 7 times as likely to stay if their article is kept
Source: [[User:Mr.Z-man/newusers]] via
[[Wikipedia:Wikipedia_Signpost/2011-04-04/Editor_retention]]
Enabling Networked Knowledge
21

22. Instructions?
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
22

23. Notabili-what?
Digital Enterprise Research Institute www.deri.ie
 22% of all deletions are speedy deleted for
A7: No indication of importance
Geiger & Ford WikiSym 2011
Enabling Networked Knowledge
23

24. Reader’s View of Deletion
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
24

25. Novices vs. Experts in
deletion discussions
Digital Enterprise Research Institute www.deri.ie
 Worthwhile content that is poorly defended -> deleted
 Need Wikipedia knowledge (procedural knowledge)
 Need content knowledge
Enabling Networked Knowledge
25

26. Articulate Values/Criteria
Digital Enterprise Research Institute www.deri.ie
 4 Factors in Deletion Discussions cover
 91% of comments
 70% of discussions
Enabling Networked Knowledge
26

27. Articulate Values/Criteria
Digital Enterprise Research Institute www.deri.ie
 4 Factors in Deletion Discussions cover
 91% of comments
 70% of discussions
 The best way to avoid deletion is for readers to
understand these criteria.
Enabling Networked Knowledge
27

28. Article Feedback
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
28

29. 4 Factors (RQ1)
Factor Example (used to justify `keep')
Notability Anyone covered by another encyclopedic
reference is considered notable enough
for inclusion in Wikipedia.
Sources Basic information about this album at a
minimum is certainly verifiable, it's a
major label release, and a highly notable
band.
Maintenance …this article is savable but at its current
state, needs a lot of improvement.
Bias It is by no means spam (it does not
promote the products).
Other I'm advocating a blanket "hangon" for all
articles on newly- drafted players
Jodi Schneider, Alexandre Passant & Stefan Decker
Deletion Discussions in Wikipedia: Decision Factors and Outcomes

30. Articulate Values/Criteria
Digital Enterprise Research Institute www.deri.ie
 4 Factors in Deletion Discussions cover
 91% of comments
 70% of discussions
 The best way to avoid deletion is for readers to
understand these 4 criteria:
 Notability
 Sources
 Maintenance
 Bias
Enabling Networked Knowledge
30

31. Other [Size], is not some little stub article, either. If you don't
Maintenance like the way the article is written, then fix it. …
Factors in Context
Digital Enterprise Research Institute www.deri.ie
Decision Messages
Factors
Sources, Read likes an WP:OR book report, only two citations,
Notability no content about why book is notable
Agreed... Also somewhat biased in tone. Merge and
Bias Redirect
Keep. This article has been in existence since 2004. It
Other [Size], is not some little stub article, either. If you don't
Maintenance like the way the article is written, then fix it. …
Enabling Networked Knowledge
31

32. Relative importance (R2)
Digital Enterprise Research Institute www.deri.ie
Notability trumped by other values
 Comprehensiveness > Notability (given Sources)
 Keeping a (non-notable) Velvet Underground album
we shouldn’t mechanically apply notability guidelines in this
instance, where it would “[punch a] hole in their otherwise
comprehensive discography.”
 Maintenance > Notability
 Deleting a notable topic due to maintenance
this is the rare case where notability is not the main argument in
favor of deletion. It has been demonstrated that the subject is
already covered in numerous other articles and that those articles
do a much better, more thorough job of covering the topic.
Enabling Networked Knowledge
32

33. Issues
Digital Enterprise Research Institute www.deri.ie
 Discussions fail without comments
 Interactions with article creators
 Contentious
 Learning opportunity
 Conflicts around consensus values
 Notability
– Why just because it is a small team and not major does it not
deserve it’s (sic) own page on here?
 Reliable sources
 Policy development is separated from case debates
 Frankly, the basis of my disagreement with you here is that I
don’t agree with the guideline.
Enabling Networked Knowledge
33

34. Future Work
Digital Enterprise Research Institute www.deri.ie
 Factor-based view of deletion
 Please give me feedback!
Enabling Networked Knowledge
34

35. 35

36. 36

37. 37

38. Thanks!
Digital Enterprise Research Institute www.deri.ie
jodi.schneider@deri.org
http://jodischneider.com/jodi.html
@jschneider
User:Jodi.a.schneider
Enabling Networked Knowledge
38

39. Deletion Workflow
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
39

40. Articles for Deletion (AfD)
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
40

41. Friction with outside
Digital Enterprise Research Institute www.deri.ie
Enabling Networked Knowledge
41

43. Novices don’t understand
notability
Digital Enterprise Research Institute www.deri.ie
 Notability vs. real-world importance
 Emsworth Cricket Club is one of the oldest cricket clubs in the world, and this
really is worth a mention. Especially on a website, where pointless people …
gets a mention.
 Why just because it is a small team and not major does it not deserve it’s (sic)
own page on here?
Enabling Networked Knowledge
43