Improve your value at work: Using evidence to influence decision making

Improve your value at work:
Using evidence to influence decision making
Karen Schriver
KSA Communication Design & Research
@firstwren
08 March 2019

People in organizations who are not part of our field
may have trouble understanding what we do
@firstwren #stc19

Document features
of products or services
From blog of Sarah Maddox, technical writer (Google Australia)
https://ffeathers.wordpress.com/2019/04/01/illustrating-the-
multifaceted-role-of-a-tech-writer/#comment-53004
What they think we do
@firstwren #stc19

Sarah Maddox, “Illustrating the multifaceted role of a tech writer” (2019).
https://ffeathers.wordpress.com/?s=a+tech+writer
What we may actually do
@firstwren #stc19

Relevant research is interdisciplinary
Artificial IntelligenceUser Experience Design
Inclusive Design
@firstwren #stc19

Critique
Claim
A practical explanation or reason
for why a claim has merit.
An evaluation of a claim by citing
quantitative or qualitative evidence
that supports (or refutes) it.
A conclusion or argument a
speaker is trying to persuade you
to accept, the merits of which
need to be convincing.
Rationale
Does the evidence support the claim?
Questioning arguments
@firstwren #stc19

Critique
Claim
Texts using simple language in
English will be easier to translate
and localize.
Texts using simple language benefit
both expert and novice users—helping
everyone. Simpler texts are usually
easier to translate and localize.
Texts should be written using simple
language because weak readers will
understand them faster.
Rationale
Does the evidence support the claim?
Questioning arguments
@firstwren #stc19

Take a rival hypothesis stance
§ What is the claim?
§ Does the rationale for the claim have merit?
§ What is the evidence for the claim?
§ Is the claim supported by evidence?
§ Who were the participants?
§ Might the evidence tell a different story?
@firstwren #stc19

Ways of drawing faulty conclusions about data
§Exaggerate the evidence
§Fail to qualify the evidence
§Authorize the evidence: “experts say or do X”
§Misapply the evidence
§Hold different values for different participant groups
§Assume data apply across contexts
§Cherry pick from the evidence
§Assume correlation means causation
§Overgeneralize from a single population
@firstwren #stc19

Consider these claims
§Since pictures can be more effective than words,
images are always a better choice than text because
they are visual.
§Madcap Flare is technical communication experts’
favorite tool.
§Using contractions in texts makes content friendlier
and adds a more personal tone.
§Writers who use AI-based language analysis software
produce better content.
@firstwren #stc19

Since pictures can be more effective than words,
images are always a better choice than text
because they are visual.
Claim
@firstwren #stc19

What kind of picture? For whom?
Hey,
show
me the
data!
@firstwren #stc19

Research says
§ Images are powerful and can be more effective than words
§ Images can be more memorable and enjoyable than words
BUT
§ Whether an image is the best choice depends on
– Type of image and clarity of signals
– Quality of image
– Relatedness of image to context
– Knowledge of viewer
@firstwren #stc19

Critique
Claim
A picture is worth a thousand
words.
Images can be more effective than
words, BUT comprehension depends on
– Type of image, quality, relatedness
– Knowledge of user
Since pictures can be more effective than
words, images are always a better choice
than text because they are visual.
Rationale
Critical consumer’s conclusion?
§ Claim needs to be qualified
§ Claim exaggerates the evidence @firstwren #stc19

Madcap Flare is technical communication
experts’ favorite tool.
.
Claim
@firstwren #stc19

Showmethedata!
Which experts?
@firstwren #stc19

Rationale for claim
Ferry Vermeulen of Berlin-based
INSTRKTIV asked 83 technical
communication experts to
nominate their top 3 favorite
technical writing tools and found...
@firstwren #stc19

Experts’ top 10 technical writing tools
1. MadCap Flare
2. DITA
3. oXygen XML Editor
4. Adobe FrameMaker
5. NotePad ++
6. Camtasia Studio
7. GitHub
8. Snagit
9. Acrolinx
10. Adobe Captivate
See Vermeulen (2019) https://instrktiv.com/en/technical-writing-tools/
@firstwren #stc19

Critique
Claim
Madcap Flare is expert technical
communicators’ favorite tool.
Participants may not represent
expert technical communicators’
preferences more generally.
An interview study showed
participants preferred Madcap
Flare over other software.
Rationale
§ Claim overgeneralizes the evidence
§ Results may not apply beyond the study’s participants
@firstwren #stc19

Using contractions in texts makes content
friendlier and adds a more personal tone.
Claim
@firstwren #stc19

Hey,
show
me the
data!
All contractions? Friendlier for everyone?
@firstwren #stc19

Types of contractions
Positive
– You’ve
– You’re
– We’ll
– We’d
Negative
– Don’t
– Won’t
– Can’t
– Shouldn’t
Modal
– Should’ve
– Could’ve
– Would’ve
– Might’ve
@firstwren #stc19

What research says
Contractions make the text friendlier and add a
more personal tone for native speakers of
English who are also fluent readers.
@firstwren #stc19

What research says
Language experience in English matters
§ Experience in listening to and reading English plays a
role in whether people will have trouble
§ Contractions can be hard for those just learning to
read English
@firstwren #stc19

What research says
Literacy matters
§ People who are poor readers in their native language are
also likely to have challenges in L2
§ Native English speakers with low literacy skills may find
contractions that express a modal verb harder to
understand
– should’ve – could’ve
– shouldn’t – couldn’t
– mightn’t – mustn’t
@firstwren #stc19

What research, best practices say
Some contractions cause problems
§ Negative contractions may place heavier cognitive
demands on some L2 readers
§ Some people rely on reading the NOT
– ChangePeople (2016)
@firstwren #stc19

The UK’s Government Digital Service
found that negative contractions led
some users to make errors that
spelling it out did not.
– Lord & Lanman (2016)
@firstwren #stc19

What research, best practices say
Not all contractions cause problems
§ Positive contractions are quite effective
– You’ve, you’re, you’ve, you’ll, you’d
– We’ve, we’re, we’ll, we’d
@firstwren #stc19

Critique
Claim
Using contractions in texts
makes content friendlier and
adds a more personal tone.
Contractions can make the text
friendlier, BUT some types may lead to
problems for people with low literacy or
who are non-native speakers.
Since contractions make the
text more friendly, it’s a good
idea to use them.
Rationale
§ Claim is supported but needs to be qualified
– Neglects evidence from different populations
@firstwren #stc19

Growing body of empirical evidence
Plain Language and
Information Design
Discourse
Level
Sentence
Level
Typographic
Features
Writing
Visual
Design
Layout
Features
Whole-text
Graphic
Features
Word Level
@firstwren #stc19

Noun strings
Word length
Word frequency
Word difficulty
Word concreteness
Nominalizations
Word-level features
@firstwren #stc19

Negatives
Syntax
Voice
Anaphora
Prepositions
Given-new
Sentence-level features
Embedding
ConditionalsLexical density
Contractions
@firstwren #stc19

Repetition
Framing
Headings
Connectives
Signals
Genre cues
Whole-text-level features
Grouping
Text density
@firstwren #stc19

Tone
Typeface
Typesize
Type weight
Linespacing
Justification
Typographic features
Case
Color
Line length Contrast
@firstwren #stc19

Line length
Columns
White space
Use of grid
Layout features
Proximity
Chunking
@firstwren #stc19

Tone
Visual impression
Hierarchy
Grouping
Use of lists
Density
Whole-text graphic features
Repetition
Consistency
Images Contrast
@firstwren #stc19

Stay tuned…
Karen A. Schriver
@firstwren #stc19

Writers who use AI-based language
analysis software produce better content.
Claim
@firstwren #stc19

Better content than what? For all audiences?
@firstwren #stc19

Study 1: Hypothesis
Using Acrolinx will enable editors to produce better plain
language summaries than using guidelines.
Key Question
Would automated feedback on style lead medical editors to
develop better plain language summaries?
Alessandra Rossetti (2018a, 2018b)
Dublin City University
@firstwren #stc19

Research context
Example: 4 million Spanish summaries searched
between 2012 and 2014
Goal:
Timely health
information for
English and
non-English
speakers
@firstwren #stc19

Methods: Study 1
Test for editors
12 experienced Cochrane editors created
§a set of summaries using Cochrane guidelines
§another set after using Acrolinx
– Editors rated satisfaction with both
Test for readers
§59 native English speakers & 23 non-native speakers
– Recall test about the content for the two sets
@firstwren #stc19

What editors saw
@firstwren #stc19

Results: Study 1
What editors thought
§No significant preference for Acrolinx over guidelines
§Preference for using both for authoring support
§Complaints
– Long lists of Cochrane guidelines were cumbersome
– Acrolinx flagged non-problems (e.g., using “very”)
@firstwren #stc19

Results: Study 1
What readers remembered
§ NO difference in recall of main ideas between two sets of
summaries
– Acrolinx did not improve native or non-native speakers
recall of the content more than using paper-based
guidelines
@firstwren #stc19

Hypothesis: Study 2
Using Acrolinx will improve the machine translation (MT)
quality of plain language summaries from English to Spanish
better than not using it.
Key Questions
Would using Acrolinx’s controlled language checker to revise
§ Improve abstracts in English?
§ Lead to more accurate and fluent MT of Spanish?
Rossetti (2018a, 2018b)
@firstwren #stc19

Methods: Study 2
Stage 1: Revise and translate
12 health professionals used Acrolinx to check for
translation, content, and style issues
§ 12 “Before” texts written in English with Spanish MT
outputs
§ 12 “After” texts (revised using Acrolinx) in English with
Spanish MT outputs
@firstwren #stc19

Methods: Study 2 (cont.)
Stage 2: Rate the translations
§ Compare “before” and “after” Spanish translations
§ 41 native Spanish speakers (also health domain experts)
randomly assigned to score the translations sentence by
sentence
– Accuracy: Maintains source content
(1 = none of it; 4 = all of it)
– Fluency: Grammatical and in fluent Spanish
(1 = incorrect and disfluent; 4 correct and fluent)
@firstwren #stc19

Results: Study 2
Revising with Acrolinx did not lead to a significant
difference in translation quality
Translation Quality Pre-Acrolinx Post-Acrolinx
Adequacy (1–4) 3.72 3.78
Fluency (1–4) 3.27 3.33
@firstwren #stc19

Critique
Claim
§ Claim is not supported by current evidence
§ Rationale is unrelated to data about content quality
§ Claim is an unproven overstatement
Writers who use AI-based
language analysis software
produce better content.
Using AI-based software did not
– produce significantly better content
– lead to significantly better translations.
Since AI-based software can score
content with significant economies
of speed and scale, writers will
produce better content using it.
Rationale
@firstwren #stc19

Faulty conclusions about evidence
§Overstate the evidence
§Fail to qualify the evidence
§Authorize the evidence: “experts say or do X”
§Exaggerate the implications of evidence
§Misapply the evidence
§Hold different values for different participant groups
§Assume data apply across contexts
§Cherry pick from the evidence
§Assume correlation means causation
§Overgeneralize from a single population
@firstwren #stc19

Conclusion: Become a critical consumer of evidence
§Be skeptical of claims and rationales
§You don’t have to be a researcher to ask good questions
§Don’t be bamboozled by not knowing the research
§If a claim sounds too good to be true…
§Things are not always what they seem: Don’t settle for
first impressions
@firstwren #stc19

Thank you!
Karen Schriver, PhD
KSA Communication Design & Research
412.828.8791
kschriver@earthlink.net
karenschriverassociates.com [relaunch coming]
linkedin.com/in/karenschriver
©Karen Schriver (2019)
@firstwren
#stc19

References
Change People. (2016). How to make information accessible: A guide to producing easy read
documents. 1-52. Retrieved from https://www.changepeople.org/Change/media/Change-
Media-Library/Free%20Resources/How-to-make-info-accessible-guide-2016-Final.pdf
Chavan, A. L. (2012). User experience research and market research. Retrieved from
http://www.humanfactors.com/whitepapers/user_experience_research_and_market_rese
arch.asp
Lord, A., & Lanman, J. (2016, Mar. 31). GOV.UK Verify and the government design
standards. Retrieved from https://identityassurance.blog.gov.uk/2016/03/31/gov-uk-
verify-and-the-government-design-standards/
Richards, S. (2018). Content Design. London: Content Design London.
Rossetti, A. (2018a). Cochrane plain language summaries: A study of authors' satisfaction
and users' comprehension. Paper presented at the EARLI Special Interest Group on Writing,
Antwerp, Belgium.
https://www.academia.edu/37333783/Cochrane_plain_language_summaries_A_study_of
_authors_satisfaction_and_users_comprehension
@firstwren #stc19

Rossetti, A. (2018b). Spanish translations of Cochrane plain language summaries:
Assessing the impact of a controlled language checker on machine translation
quality. Retrieved from
https://dcu.academia.edu/AlessandraRossetti/Conference-Presentations
Rossetti, A., Vázquez, S. R., & O’Brien, S. (2017). Interaction of Cochrane authors
with a non-automated and automated editing approach for the production of
Plain Language Summaries. Retrieved from
https://www.academia.edu/34464622/_Report_Interaction_of_Cochrane_auth
ors_with_a_nonautomated_and_automated_editing_approach_for_the_produc
tion_of_Plain_Language_Summaries
Maguire, L. K., & Clarke, M. (2014, Nov.). How much do you need: A randomized
experiment of whether readers can understand the key messages from
summaries of Cochrane Reviews without reading the full review. Journal of the
Royal Society of Medicine, 107(11), 444–449. doi:10.1177/0141076814546710
Redish, J. C. (2012). Letting go of the words: Writing web content that works (2nd
ed.). San Francisco, CA: Morgan Kaufmann/Elsevier.
Schriver, K. A. (1997). Dynamics in document design: Creating texts for readers.
New York, NY: John Wiley & Sons.
@firstwren #stc19

Schriver, K. A. (2012). What we know about expertise in professional
communication. In V. W. Berninger (Ed.), Past, present, and future
contributions of cognitive writing research to cognitive psychology (pp. 275-
312). New York, NY: Psychology Press.
Schriver, K. A. (2013). What do technical communicators need to know about
information design? In J. Johnson-Eilola & S. Selber (Eds.), Solving problems
in technical communication (pp. 386-427). Chicago, IL: Univ. of Chicago Press.
Schriver, K. A. (2017, Dec.). Plain language in the United States gains
momentum: 1940-2015. IEEE Transactions on Professional Communication,
60(4), 343–383.
Schriver, K. A. (Speaker). (2018, Jan. 18). On plain language [Audio podcast].
Retrieved from http://sites.ieee.org/pcs/podcast-karen-a-schriver/
Schriver, K. A. (in preparation). Reaching busy readers: Evidence-based plain
language and information design.
@firstwren #stc19

Improve your value at work: Using evidence to influence decision making

Recommended

Recommended

More Related Content

Similar to Improve your value at work: Using evidence to influence decision making

Similar to Improve your value at work: Using evidence to influence decision making (20)

Recently uploaded

Recently uploaded (20)

Improve your value at work: Using evidence to influence decision making