Annotation examples. This is an overview of some of the software I have used for annotation (and a few extra features some of this software has.) This was presented in the SwissUniversities Doctoral Programme, Language & Cognition, in the Module: Linguistic and corpus perspectives on argumentative discourse. Screenshots are given of GATE, UAM Corpus Tool, Excel, BRAT, EPPI Reviewer, and a custom tool. In most cases there are references to one of my papers for further details. I briefly describe a typical annotation process: Find text of interest Find phenomena of interest Draft an annotation manual Iteratively test annotation & revise manual Find questionable annotations, check disagreements. Revise the manual. Iterate. Annotate

1. Annotation
Jodi Schneider
Linguistic and corpus perspectives on argumentative
discourse, SwissUniversities Doctoral Programme
Language & Cognition
University of Fribourg, Fribourg, Switzerland
2019-09-02

2. A typical annotation process
• Find text of interest
• Find phenomena of interest
• Draft an annotation manual
• Iteratively test annotation & revise manual
– Find questionable annotations, check disagreements.
– Revise the manual.
– Iterate.
• Annotate

3. Examples of annotation software
• GATE: https://gate.ac.uk Free & Open source
– NLP pipeline integration, robust developmnet community, ingests lots of
formats,
• UAM CorpusTool: http://www.corpustool.com Free
– Comparative statistics, corpus search, annotation schemes easy to set up
• Excel
– Great for simple annotation
• BRAT: http://brat.nlplab.org Free & Open source
– Run your own instance, browser-based for collaboration
• EPPI Reviewer:
https://eppi.ioe.ac.uk/cms/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4
– Data extraction for systematic review
• Custom tools

4. GATE
Jodi Schneider, Alexandre Passant, and Stefan Decker “Deletion
Discussions in Wikipedia: Decision Factors and Outcomes.”
In WikiSym2012. Linz, Austria, August 27-29, 2012.

5. UAM CorpusTool(V 2.8.16)
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments
about Deletion: How Experience Improves the Acceptability of Arguments in Ad-
hoc Online Task Groups”. In CSCW 2013, San Antonio, TX, February 23-27, 2013.

6. Excel
Dong, Xiaoru; Xie, Jingyi; Hoang, Linh (2019): Inclusion_Criteria_Annotation. University of
Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-5958960_V2 for Text
Mining Pipeline to Accelerate Systematic Reviews in Evidence-Based Medicine

7. Excel
Hoang, Linh; Schneider, Jodi (2018): Citation context analysis of RobotReviewer core papers circa
2018-06. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-
1075526_V1 for Text Mining Pipeline to Accelerate Systematic Reviews in Evidence-Based
Medicine

8. BRAT
Halil Kilicoglu, Zeshan Peng Shabnam Tafreshi, Tung Tran, Graciela Rosemblat, Jodi Schneider.
“Confirm or Refute?: A Comparative Study on Citation Sentiment Classification in Clinical Research
Publications.” Journal of Biomedical Informatics, Vol 91, 103123. doi: 10.1016/j.jbi.2019.103123

9. EPPI-Reviewer
Work in progress, Systematic Review of Empirical Research about Retracted
Publications project team

10. Custom Tools
Halil Kilicoglu, Graciela Rosemblat, Zeshan Peng, Mario Malicki, Tony Tse, Jodi Schneider, Gerben
ter Riet. Annotating Clinical Trial Publications to Assess CONSORT Adherence: A Feasibility Study.
6th World Conference on Research Integrity, Hong Kong, 2019.

12. Some tools have additional features

13. GATE - sentiment (gazeteer)
Jodi Schneider & Adam Wyner. “Identifying Consumers' Arguments in Text”
In SWAIE 2012: Semantic Web and Information Extraction at EKAW 2012.

14. GATE - ,
,
(gazeteers)
Jodi Schneider & Adam Wyner. “Identifying Consumers' Arguments in Text”
In SWAIE 2012: Semantic Web and Information Extraction at EKAW 2012.

15. GATE – semantic search
Jodi Schneider & Adam Wyner. “Identifying Consumers' Arguments in Text”
In SWAIE 2012: Semantic Web and Information Extraction at EKAW 2012.

16. UAM CorpusTool(V 2.8.16)
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments
about Deletion: How Experience Improves the Acceptability of Arguments in Ad-
hoc Online Task Groups”. In CSCW 2013, San Antonio, TX, February 23-27, 2013.

17. UAM CorpusTool(V 2.8.16)
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments
about Deletion: How Experience Improves the Acceptability of Arguments in Ad-
hoc Online Task Groups”. In CSCW 2013, San Antonio, TX, February 23-27, 2013.

18. EPPI-Reviewer
Work in progress, Systematic Review of Empirical Research about Retracted
Publications project team