Content usability testing - a case study in how to measure the effectiveness of your content

1. Kate Kenyon @kate_kenyon
Content Usability Testing:
Getting to Good

2. Hello, I’m Kate
‘80s – bookish child of an IT dev and
a PM
‘90s – linguist, magazine writer
‘00s – marketer, journalist,
technologist, content strategist
I design content, and I design the
teams and systems that make it.

3. What we’re covering today
We are here
Lunch is here

4. The Goal
is to deliver
a better customer experience
using effective content

5. Testing content shows
the gap between strategy
and practice

6. Testing provides data for content
decision making

7. Testing content informs IA, CS, and Design
Image: http://www.socialmediaconnections.biz/wp-content/uploads/2015/12/responsive_web_design.jpg

8. Testing content keeps
stakeholders aligned
Image courtesy of Shutterstock

9. Testing content explodes expectations

10. Testing enables content to become a
quantifiable, manageable asset with genuine
business value

11. The Challenge
is to create a content testing process
that will yield regular, usable data

12. Testing is part of a cycle, not a process
Plan
Make
TestReview
Publish

13. Testing content needs to be a repeatable exercise
Plan
Make
TestReview
Publish
Plan
Make
TestReview
Publish
Plan
Make
TestReview
Publish
Plan
Make
TestReview
Publish

14. It’s not the testing, it’s what you do with the results
Repeated testing builds
data around how content
works on your site, and
what you should be making.
That data needs to be part
of your decision making
processes
Plan
Make
TestReview
Publish

15. Old production processes + a bit of testing
won’t deliver the benefits you’d like.
You need a plan to change your processes,
before you start testing.

16. Approach

17. How I do testing
Image courtesy of Shutterstock
• Content testing principles
• Mapping the landscape
• Testing the things that matter
• Writing test cases
• Running tests
• Writing up findings

18. Content Testing Principles
Image courtesy of Shutterstock
• Content is there to do a job. Know what that job is, and test
for how well the content is doing that.
• Test by asking questions that are important to your key
people. These could be customers or the business, or both.
• Test using questions with definitive answers.

19. Mapping the landscape
Image courtesy of Shutterstock

20. Test for the things that matter
Image courtesy of Shutterstock
• What is the significant influencing driver within the business?
• How do your testing goals align with that driver?
• What will you do with the results of the testing?
• Who will potentially be impacted by the testing?

21. Writing test cases
• Write tasks: one task per question
• Questions must have ONE definitive answer
• No clues in the questions
• Use plain language and common speech
• Be concise
• Make tasks independent from each other
• Make sure tasks don’t require confidential information.

22. Writing test cases: samples
• How many days holidays is a new, full-time employee
entitled to in UK offices?
• Who should be the first person you contact if you have an
accident while on site with a client?
• How many bottles of hair dye did the company sell in the UK
in 2015?

23. Running tests
Image courtesy of Shutterstock

24. Running Tests
• Choose volunteers carefully: screen for prior knowledge and
bias.
• Give volunteers plenty of notice, and tell them in advance
what will happen.
• Stress that you’re testing the site, not them.
• Check your environment! Equipment, connections, facilities,
room bookings
• Have water, pens, paper, tissues, food available.

25. Recording results
• Use VIDEO! Jing is free, easy screen recording software.
There are many others, and for remote testing,
GotoMeeting has worked well.
• Ask participants to comment and narrate as they go
• Have one test observer sit beside the participant to guide
and lead the test
• Have another observer sit behind them to one side, so
they can look over their shoulder, and note down any
behaviours or comments.

26. Writing up results
Image courtesy of Shutterstock
• Use a meaningful format. Powerpoint, doc, video: choose a
format that will best deliver the message to the intended
audience
• Exec summaries are essential, and recommendations.
• If quoting from participants, maintain anonymity, and quote
accurately.
• Provide the data in appendices

27. Some notes on things I’ve learned
during testing

28. Testing: have a clear view of who is involved
Image courtesy of Shutterstock

29. Be clear that this is about making hard choices
Image: https://www.lighthousewebdesigns.com/wp-content/uploads/2015/04/bad-content.png

30. Testing content makes you align what you’ve
got with what you want

31. Case Studies

32. Mapping the landscape
Image courtesy of Shutterstock
• A global intranet site hosting over 100,000 pages of content
• One (rather ancient) implementation of Sharepoint
• Thousands of employees, across multiple countries
• A head of employee comms fighting to get her message across
• No way of showing the impact that the intranet was having.

33. Image courtesy of Shutterstock
Image courtesy of Shutterstock

34. Mapping the landscape
Image courtesy of Shutterstock
How effective is the Bank’s intranet in supporting Global Standards? Does the
intranet help employees implement their day-to-day activities, or actively contribute
to risk within the Bank?
Cognifide, a technology and consulting firm, was commissioned to conduct content
usability research to look answering these questions.
The research was conducted between July and August 2016, and focussed on how
well Bank employees could use the intranet to answer 5 questions related to
Global Standards.
It found that, on average, employees were only able to find the answers to just over
half (51%) of the questions.
This means that for this test the failure rate was 49%, a significant exposure to risk
for the Bank.

35. Writing test cases
Image courtesy of Shutterstock
• Asking each user to find the answers to a set of questions relating to Global Standards.
They had to use the intranet only to complete the task.
• Questions were worded to have an unambiguous answer.
• Additional self-assessment questionnaire to provide a subjective view on how easy it was
to do the tasks
Outcome Meaning Metrics
Success The information can be found in a reasonable amount of time.
User can complete the task/give the right
answer in the allotted time.
Fail
The information cannot be found in a reasonable amount of time. User cannot complete the task/give the right
answer in the allotted time, or gives up.
Disaster The information is confusing or misleading.
User completes a different task/gives the
wrong answer.

36. Participants & Location
Image courtesy of Shutterstock
• Participants were recruited among Bank new joiners who had completed their mandatory
training. They took part anonymously and they are identified in this report by a progressive
number (Participant 1, Participant 2, etc.).
• Anonymity is a pre-requisite for this type of research, to ensure participants can freely
express their opinion. All participants were already familiar with the intranet, and most of
them stated that they use the intranet daily for their job.
• Sessions were run at the Bank in a common meeting room, without external observers, and
they were attended by the participants, one at a time. Two facilitators from Cognifide were
attending as well, one running the session and another one taking notes.

37. Running tests
Image courtesy of Shutterstock
Task # Question
Task 1 What are the goals of the Bank’s Global Standards?
Task 2
If you suspect an account is being used for money laundering, can you tell me one of the ways to escalate
your concern?
Task 3 Can you name three unusual activities that might indicate a customer is involved in money laundering?
Task 4 A customer wants to send money to their relatives in Syria: are they allowed?
Task 5
Following an Unusual Activity Report, a customer asks to see their financial crime risk rating. Can you
informally tell them?

38. Writing up results
Image courtesy of Shutterstock
• Report written, with exec summary, methodology, test cases, results per question,
overall results, outcomes and recommendations
• PowerPoint of recommended next actions, with timelines and rough costs

39. Results into Actions
Image courtesy of Shutterstock
• Data used to build business case for
$10m investment in a new intranet CMS
• Data used to acquire new funding for UX
review, Content Strategy work and new
intranet governance
• Repeat testing scheduled to show
changes following updates.

40. Well, that’s a wrap!

41. Plan
Plan for the
smallest start
Design for
outcomes,
not problems
Image courtesy of Shutterstock

42. Run
Start fast
Expect
headwinds
as results
appear
Image courtesy of Shutterstock

43. Repeat, often
Don’t overthink it
Hold your ideas
lightly
Expect to retest
monthly, quarterly,
yearly
Image courtesy of Shutterstock

44. Final things!
PLAN:
agree on what the goal is
plan how to get there
know what you’ve got to work with
know how you measure ‘good’
REPEAT:
keep moving on, like water
RUN:
brief producers fully
keep track of what’s live
own your errors, and successes 

45. @ k a t e _ k e n y o n •
Kate.Kenyon@cognifide.com