Slides for my talk at the Scientific Meeting at Centrum Wiskunde & Informatica, 29/05/2015 A standard procedure in humanities research is to evaluate the aptitude of a source by performing source criticism. This includes the examination of the validity, reliability and relevance of a source in the context of a given research task. By turning the physical manifestations of the sources into digital objects, the sources are transformed by the technology applied to them. Some of the used tools, such as OCR engines, do not yield error-free results. For humanities researchers, this means that they do not only have to deal with source-inherent biases, but they also have to take into account tool-induced bias. The aim of our research is to develop methods that enable humanities researchers to perform tool criticism by revealing uncertainty in the data by tracing tool-induced bias.

1. Tool Criticism
Myriam C. Traub, Jacco van Ossenbruggen, Lynda Hardman!
Information Access

2. “Amsterdam”
1642
“Amfterdam”
1624
“?”
16xx?
2
First mention of …
1618
… in the OCRed newspaper archive of the KB?
“Amslerdam”
1673
earliest !
document

3. 01
Digital
Humanities
✤ digital archives / large data
collections!
✤ collection bias / digitization policy!
✤ digital representation ≠ physical
object!
✤ tools are imperfect !
✤ source cannot be considered
independent from tools!
✤ need methods to detect tool-
induced bias
3

4. 01
Source criticism
✤ well established method in the
humanities to detect bias in a
source!
✤ Who is the author?!
✤ Is the information current?!
✤ Is the information objective
and credible?…!
✤ need a similar approach for
tool-induced bias: !
! tool criticism
4

5. 01
Methodology
✤ interviews with humanities scholars!
✤ classification of common research
tasks!
✤ lack of trust blocks progress!
✤ use case: digital newspaper archive
of KB !
✤ no formal OCR evaluation!
✤ useful for scholars?!
✤ mismatch between two
perspectives
5

6. We care about
average performance
on representative
subsets for generic
cases.
I care about
actual performance
on my non-
representative subset
for my specific
query.
6
Two different perspectives of quality evaluation

7. No silver bullet
✤ we propose novel strategies that solve
part of the problem:!
✤ critical attitude !
(awareness and better support)!
✤ transparency !
(provenance, open source,
documentation, …)!
✤ alternative quality metrics!
(taking research context into account)
7
M. Traub, J. van Ossenbruggen, L. Hardman, !
Impact Analysis of the OCR Quality Problem in Digital Archives, TPDL2015 (under review)

8. 8
“Amsterdam”
Context-aware quality indicators
query
data
task

9. 01
Future work
✤ What strategies should a tool
support to help scholars discover
and dealing with bias?!
✤ What is a good way of
estimating uncertainty for a
specific task?!
✤ Can we crowdsource (part of the
data for) better estimates?!
✤ What is a good way of conveying
the estimated impact to scholars?
9
… questions?