This is a presentation I gave at 2019 Wikimania Research in Sweden (August 17). It gives an overview of our research projects related to Wikipedia's Article for Deletion (AfD) discussions, with a focus on the rationales in the discussions

1. Hidden Gems in the Wikipedia Discussions:
The Wikipedians’ Rationales
Lu Xiao
School of Information Studies
Syracuse University
lxiao04@syr.edu
Wikimania, Stockholm – August 17, 2019
1

2. Context: Wikipedia’s Article for Deletion
(AfD) Discussions
 A Wikipedia article proposed to be deleted may undergo the community discussion
before a decision is made (e.g., to delete the article or to keep it). The article
tagged for Deletion Discussion is called “Article for Deletion” (AfD).
 Comment: participant’s view (keep, delete, etc.) and his/her rationale
 AfD discussion: keep, delete, other
2
Fig. 1. A Wikipedia AfD discussion example

3. How do participants’ rationales affect AfD
discussions?
3
Wikipedians’ rationales
in AfD discussions
Perceived Effects
Xiao & Askin, 2012
Sentiment of the Rationale
Xiao & Sitaula, 2018
Content of the Rationale
Xiao & Askin, 2014;
Javanmardi & Xiao, 2019
Language use of the Rationale
Mao, Mercer, & Xiao, 2014; Xiao
& Nickerson, 2019
Xiao, 2018

4. Online questionnaire: the perceived effects of AfD
participants’ rationales (Xiao & Askin, 2012)
 Data Collection
wiki-research-l, Wikimedia-l, wikiEN-l, and Wikipedia-l
41 valid responses in over a month, 8 female, 28 male, and 5 chose “prefer not to
say”
 Data Analysis and Result
 Most respondents have been a Wikipedia editor for over 5 years (59%), while only one
had less than a year of experience.
 “people understand other votes better by knowing rationales” (29 responses), “sharing
rationales helps control the quality of the discussion outcome” (25 responses), and “the
shared rationales help educate participants about Wikipedia’s policies” (24 responses)
 “sharing rationales makes it more challenging to propose different opinions” (4
responses), sometimes “lengthy debates are a waste of time” because of misuse of
policy or processes” (5 responses)
 Three types of most influential rationales: rationales that are about the article’s notability,
that cite Wikipedia’s policies, and that are from established Wikipedia editors (7
responses)
4
Xiao, L., & Askin, N. (2012) Deliberation in Wikipedia: Rationales in Article Deletion Discussions, the 75th Annual Meeting of Association for
Information Science and Technology (ASIS&T) (Oct. 26-30, Baltimore, MD, USA), 49: 1–4. doi: 10.1002/meet.14504901234

5. What’s in an AfD rationale: a traditional content
analysis (Xiao & Askin, 2014)5
Xiao, L., & Askin, N. (2014). What Influences Online Deliberation? A Wikipedia Study, Journal of the American Society for Information Science and
Technology, 65: 898–910
 Data
 Data Analysis: open and iterative coding
Day # of Articles Total votes for
Keep
Total votes for
delete
Total other
votes 9merge,
userfy, etc.)
June 1, 2010 89 127 280 37
June 1, 2011 73 119 212 23
Jan. 15, 2012 67 109 200 63

6. A traditional content analysis: Results
 Most participants offered their rationales including when they express their
agreement/disagreement, as requested by Wikipedia’s policy on voting
 A good portion of rationales referred to Wikipedia policies or topics
 notability far outweighs the second most frequent rationale, credibility.
 articles about people, forprofit organizations, and definitions are slightly more likely to be
deleted than expected
 articles about locations or events are more likely to be kept than expected
 articles about nonprofit organizations and media are more likely to be suggested for other
options (e.g., merge, redirect, etc.) than expected.
 AfD participants not only provide arguments and rationales about their opinions but also
take actions to help improve the articles and make them acceptable to the community
6
Xiao, L., & Askin, N. (2014). What Influences Online Deliberation? A Wikipedia Study, Journal of the American Society for Information Science and
Technology, 65: 898–910

7. What’s in an AfD Rationale: a computational and
visualization approach (Javanmardi & Xiao, 2019)7
Javanmardi, A., Xiao, L. (2019). What’s in the Content of Wikipedia’s Article for Deletion Discussions? Towards a Visual Analytic Approach, the Sixth
Wiki Workshop at the 2019 Web Conference (May 14, San Francisco, USA), 1215-1223
 AfD discussions dated from May 15, 2013, to May 15, 2015
 To prepare the dataset for further computational analysis techniques, we removed
5% of the dataset that caused difficulties when parsing the HTML content to extract
the parts.
 39, 177 discussions from the raw about 40, 000 discussions.
 A SQL Database that organizes AfD discussion data (e.g., content, title, outcome,
etc.) as well as additional information (e.g., total number of comments, category of
the article, policies mentioned in the comments)
Dataset (https://minio.webservices.ischool.syr.edu/minio/research-data/)

8. 8
Participant’s vote in following choices Vote
stored as
Number of Top level
comments in the
analysis
Strong delete, speedy delete, delete, weak delete Delete 21,589
Weak keep, keep Keep 7,976
Note, comment, question, speedy close, closing
administrator, relisted, text, reviews, speedy
decline, withdraw, userfy, move and dab, move,
oppose, merge, redirect, redirect and merge
Other 8,196
Table 1. a comment’s vote and its distribution in the analyzed data
What’s in an AfD Rationale: a computational and
visualization approach (Javanmardi & Xiao, 2019)
Javanmardi, A., Xiao, L. (2019). What’s in the Content of Wikipedia’s Article for Deletion Discussions? Towards a Visual Analytic Approach, the Sixth Wiki
Workshop at the 2019 Web Conference (May 14, San Francisco, USA), 1215-1223

9. The Policies Mentioned in the Participants’ Rationales
9
http://www.mandanemedia.com/afd/view/diagram3.php
What’s in an AfD Rationale: a computational and
visualization approach (Javanmardi & Xiao, 2019)
The Mapping of Article Categories and Mentioned
Policies in the Rationales
http://www.mandanemedia.com/afd/view/diagram5.php

10. AfD Rationales’ Sentiments and the Discussion
Outcome (Xiao & Sitaula, 2018)
VADER for sentiment measure
Our analysis suggests that:
• Discussions that had delete outcome are more than expected to
have negative and neutral sentiment and less than expected to have
positive sentiment.
• Discussions that had keep or other outcome are more than expected
to have positive sentiment and less than expected to have negative
and neutral sentiment.
10
Xiao, L. Sitaula, N. (2018). Sentiments in Wikipedia Articles for Deletion Discussions. Proceeding of
iConference (Mar. 25 – 28, Sheffield, UK), 81-86. Springer. https://link.springer.com/chapter/10.1007/978-3-
319-78105-1_10

11. Language use of AfD Rationales I: Imperatives
11
Mao, W. T., Mercer, R., & Xiao, L. (2014). Extracting Imperatives from Wikipedia Article for Deletion Discussions, First Workshop on Argumentation
Mining at the 52nd Annual Meeting of the Association for Computational Linguistics (ACL) (June 22 – 27, Baltimore, MD, USA), retrieved on Mar. 31,
2016, http://acl2014.org/acl2014/W14-21/pdf/W14-2117.pdf
Xiao, L., & Nickerson, J. V. (2019). Imperatives in past online discussions: Another helpful source for community newcomers? Proceedings of the 52nd
Annual Hawaii International Conference on System Sciences (Jan. 8 – 11, Grand Wailea, Maui, USA), IEEE, http://hdl.handle.net/10125/59966
 Data
• One week of the Wikipedia AfD discussions for each month of 2013, i.e., 84 days of AfD
discussion pages – 4,593 discussions
• Imperative extraction tool by Mao, Mercer, and Xiao (2014) extracted 2, 768 imperative
statements
 Data cleaning
• 1, 272 of 2, 768 imperative statements were kept for further analysis
• 59% of discarded statements (N = 884): non-imperatives
• Remaining discarded statements: did not contain decontextualized knowledge (e.g.,
Listen, Go ahead, Come on here)
 Data analysis: open and iterative coding process

12. Type of Request/Suggestion Number of Imperatives (Percentage)
Type 1: Article Content 196 (15.4%)
Type 2: Wikipedia Technical 165 (13.0%)
Type 3: Discussion Norm or Practice 150 (11.8%)
Type 4: Reference 364 (28.6%)
Type 5:Reasoning/Evaluation
Perspectives
397 (31.2%)
Language use of AfD Rationales I: Imperatives (Mao,
Mercer, & Xiao, 2014; Xiao & Nickerson, 2019)
Results
Table 3: Types Of Requests Or Suggestions in these AfD Imperative Rationales
12

13. Language use of AfD Rationales II: Persuasive vs.
Non-Persuasive Rationales13
Xiao, L. (2018). A Message's Persuasive Features in Wikipedia's Article for Deletion Discussions. In Proceedings of the 9th International Conference on
Social Media and Society (July 18 – 20, Copenhagen, Denmark), 345-349. ACM.
Identifying Persuasive and Non-Persuasive Rationales
 Persuasive rationale: rationale of a participant’s view that is the same as the
discussion outcome
 Non-persuasive message: a rationale of a participant’s view that is the opposite of the
discussion outcome
 75,942 persuasive comments and 8,687 non-persuasive comments
 Included in our analysis: 1710 persuasive vs. 2195 non-persuasive comments
 only kept those discussions that have both persuasive and non-persuasive comments and
have equal or more number of non-persuasive comments than the persuasive ones – to
alleviate the situation that the persuasive power is accumulated from multiple persuasive
comments.
 Removed noised data (“#NAME”, “#name”, word count < 3)

14. Language use of AfD Rationales II: Persuasive vs.
Non-Persuasive Rationales (Xiao, 2018)14
Xiao, L. (2018). A Message's Persuasive Features in Wikipedia's Article for Deletion Discussions. In Proceedings of the 9th International Conference on
Social Media and Society (July 18 – 20, Copenhagen, Denmark), 345-349. ACM.
Language use in Persuasive vs. Non-Persuasive Rationales
 Linguistic Inquiry & Word Count (LIWC) Analysis Tool
 Results
 The use of punctuation marks
 The use of emotion words
 The drives by power and risk
 Length of a comment

15. How do participants’ rationales affect the
outcome of an AfD discussion?
15
Wikipedians’ rationales
in AfD discussions
Perceived Effects
Xiao & Askin, 2012
Sentiment of the Rationale
Xiao & Sitaula, 2018
Content of the Rationale
Xiao & Askin, 2014;
Javanmardi & Xiao, 2019
Language use of the Rationale
Mao, Mercer, & Xiao, 2014; Xiao
& Nickerson, 2019
Xiao, 2018
Future Work
• Participant
(e.g., Gender, Status)
• Temporal
• Topic/Category
…

16.  The Categories of the AfD articles
16
www.mandanemedia.com/afd/view/diagram4.php
Three categories have the highest
percentages of the delete comments:
Martial Art, Football, and Sportspeople.
Three categories tend to have higher
percentage of keep comments: History,
Crime, and Architectures.
Other Visualizations in this Study (Javanmardi &
Xiao, 2019)

17. Acknowledgement
Collaborators:
Nicole Askin, Ali Javanmardi, Niraj Sitaula, Jeffrey Nickerson, Wanting
Mao, Robert Mercer
Funding support:
The Natural Sciences and Engineering Research Council of Canada (NSERC)
17

18. Thank You!
Lu Xiao
School of Information Studies
Syracuse University, USA
lxiao04@syr.edu
18

19. Background
 The rationales provided by the others can make an impact on the
individual and on the collective activity (Xiao, 2013; Xiao &
Carroll, 2013; Xiao & Carroll, 2015; Xiao, 2014)
 Awareness of the others’ intellectual contribution and domain expertise
 The individual’s reflective thinking
 The group work’s quality
19
Xiao, L. (2013) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software, 86(7), 1727 –
1737
Xiao, L. (2014). Effects of rationale awareness in online ideation crowdsourcing tasks, Journal of the American Society for Information Science and
Technology, 65, 1707-1720, doi: 10.1002/asi.23079
Xiao, L., & Carroll, J. M. (2013) The Effects of Rationale Awareness on Individual Reflection: Processes in Virtual Group Activities, International Journal
of e-Collaboration, 9(2), 78 – 95
Xiao, L., & Carroll, J. M. (2015). Shared practices in articulating and sharing rationale: An empirical study. International Journal of e-Collaboration,
11(4), Article 2, 11-39.

20. A traditional content analysis: Coding Schema
20
Table 3 in Xiao, L., & Askin, N. (2014). What
Influences Online Deliberation? A Wikipedia
Study, Journal of the American Society for
Information Science and Technology, 65:
898–910

21. AfD Rationales’ Sentiments and the Discussion
Outcome (Xiao & Sitaula, 2018)
Results (VADER for sentiment measure)
Label # of Delete discussions # of Other discussions # of Keep discussions
Positive 2460 1498 1739
Neutral 17641 6389 6011
Negative 1488 309 226
Table 2. Sentiment labels on the outcome of a discussion
We conducted a chi-square test based on this result and obtained a p-value < 0.001. The
test results suggest that:
• Discussions that had delete outcome are more than expected to have negative and
neutral sentiment and less than expected to have positive sentiment.
• Discussions that had keep or other outcome are more than expected to have
positive sentiment and less than expected to have negative and neutral sentiment.
21

22.  The Outcomes vs the No. of Keep/Delete Opinions
22
www.mandanemedia.com/afd/view/diagram1.php www.mandanemedia.com/afd/view/diagram2.php
Other Visualizations in this Study (Javanmardi &
Xiao, 2019)