Presentation on the perception of machine written news at the Computation and Journalism Conference @Columbia University, New York.

1. Robot Reporters or Human Journalists:
Who Do You Trust More?
Hille van der Kaa & Emiel Krahmer
Credit: Shutterstock/Ociacia

2. Link to paper:
http://tinyurl.com/computertrust
@Hillevanderkaa

3. ‘influence of
technology
on storytelling’

4. automated newsroom
!!!!
can algorithms be used to
collect
information
analyze
information
transform
data into a
story

6. sensor journalism

7. social network analysis
infiniti.taalmonsters.nl

8. text-to-text generator

9. natural language generation
‘The process of automatically creating natural
language text based on non-linguistic input.
It combines knowledge about language and
the application of the domain to produce
documents and reports automatically.’
(e.g., Reiter & Dale, 1997, 2000)

10. nlg & news production
Although template-based
text-generation algorithms
have been under
development since the
1990s, news companies
have only recently started to
work with algorithms
operating along similar
lines.
(e.g. Buseman & Horacek, 1998; McRoy, Channarukul, &
Ali, 2003; Narayan, Isbell, & Roberts, 2011; Theune,
Klabbers, Odijk, De Pijper, & Krahmer, 2001; Nichols,
Birnbaum, & Hammond, 2014)

11. nlg
content selection
text planning
sentence planning
linguistic realization
journalist
research
selecting
structuring
writing

12. can robot writers replace journalists?
A large part of this decision is based on
economics: is it cheaper to hire and train a
person or to create and maintain a software?
(Reiter & Dale, 1997)

13. YES!
‘Robot journalism will free resources,
allowing reporters to focus on assignments
they are more qualified for and leaving the
descriptive summaries to the software’.
(Clerwall, 2014; Flew et al., 2012)

14. NO!
Van Dalen (2012) states that machines do
not have the creativity to avoid clichés and
add humor, although others might disagree.
(e.g., Binsted & Ritchie, 1994; Petrovic & Matthews, 2013)
Moreover, they do not have the flexibility, to
write non-routine stories, and they do not
have analytical skills to offer their stories.

15. YES!
NLG techniques may be preferred over
manual document creation because they
increase accuracy and reduce updating
time, or because they guarantee
conformance to standards.
Van Dalen (2012) adds that journalists see
the advantages of an algorithm as an
unbiased reporter.

16. does the audience agree?
Does the audience think automatically
generated news stories are as trustworthy as
human- written articles?

17. previous research
NLG
Focus on the
text quality —
grammatically
correct and
coherent?
CREDIBILITY
Focus on human
written stories –
source, message
and medium
credibility.

18. Fogg and Tseng (1999)
state that in discussing
the credibility of a
computer product, one
is actually discussing the
perception of the
credibility.
trustworthiness and
expertise are hereby
two key components

19. trustworthiness
holds the
perceived
goodness or
morality of a
source
expertise is
defined by
knowledge,
experience,
and
competences
highly credible
computer products will
be perceived as having
high levels of both
trustworthiness and
expertise
(Fogg & Tseng, 1999)

20. ‘In this research, we take a look into the
perceived trustworthiness and expertise of
robot-written news articles, searching
specifically for differences and similarities
between journalists and news consumers.’

22. RKC Waalwijk speelt gelijk tegen PEC
Zwolle
RKC Waalwijk ging op bezoek bij PEC Zwolle
en speelde gelijk. Het duel eindigde in 1–1.
Twaalfduizend toeschouwers kwamen naar
het IJsseldelta stadion.
De ploeg uit Zwolle nam na 44 minuten de
leiding door een treffer van Saymak. Na
achtenveertig minuten bracht Joachim van
RKC Waalwijk de teams op gelijke hoogte.
De wedstrijd werd gefloten door
scheidsrechter Kamphuis. Hij deelde geen
rode kaarten uit. Tomas en De Boer van PEC
Zwolle en Sno van RKC Waalwijk liepen tegen
een gele kaart aan.
‘This article is written by a computer’
RKC Waalwijk speelt gelijk tegen PEC
Zwolle
RKC Waalwijk ging op bezoek bij PEC Zwolle
en speelde gelijk. Het duel eindigde in 1–1.
Twaalfduizend toeschouwers kwamen naar
het IJsseldelta stadion.
De ploeg uit Zwolle nam na 44 minuten de
leiding door een treffer van Saymak. Na
achtenveertig minuten bracht Joachim van
RKC Waalwijk de teams op gelijke hoogte.
De wedstrijd werd gefloten door
scheidsrechter Kamphuis. Hij deelde geen
rode kaarten uit. Tomas en De Boer van PEC
Zwolle en Sno van RKC Waalwijk liepen tegen
een gele kaart aan.
‘This article is written by a journalist’

23. expertise
exper&se
trustworthiness
trustworthiness
intelligence
authority
bias
accuracy
completeness
fact-­‐based
text-­‐quality
honesty

24. WITHIN the group of news consumers we
found no difference in the perceived
credibility of the news source.
Further, they were neutral about the level of
expertise of a computer writer and
journalist.

25. WITHIN the group of journalists we found a
significant difference in the perceived
credibility of the news source.
Journalists perceive the credibility of a
journalist higher.

26. comparison of both groups
Journalists perceived the expertise of a
computer writer as greater than it was
perceived by news consumers.

27. Finally, the research also suggests that story
topic has an influence on a news item’s
perceived trustworthiness.
Overall, respondents perceived the
trustworthiness of the sports article to be
lower than that of the finance article.

28. new questions
Do journalists overestimate their level of
trustworthiness compared to a computer
writer? And if that is true, what are the reasons?
Does this create an increased risk that
journalists will be displaced by the rise of
automated journalism, since they might not see
the need to expand their repertoire to include
more areas where computers can’t compete?

29. Why do journalists differ from news
consumers in their perception on the level of
expertise?
Is it because of their training, ethics, a
greater awareness of robot reporting — or
something else?

30. Since the topic of the text seems to
influence the results, what subjects do
people perceive algorithmic approaches as
having more credibility?

31. January 2015
entertainment
tech
hard versus soft news
repetitive effect

32. @HillevanderKaa
Please visit ARJ.org:
http://tinyurl.com/computertrust