How to execute a collaborative review of usability testing to facilitate design and development prioritisation consensus. Case studies of how the approach has worked at the University of Edinburgh. Presented at UX Scotland conference 2019

1. Shared research
experiences
RAPID REPORTING AND SHOWCASING OF USABILITY TESTING
1NEIL ALLISON @USABILITYED #UXSCOT

2. Hi I’m Neil…
UX SERVICE MANAGER AT THE UNIVERSITY OF EDINBURGH
EX-CONTENT MANAGEMENT SYSTEM SERVICE MANAGER
SERVICE DESIGNING BEFORE I KNEW IT WAS A THING
NEIL ALLISON @USABILITYED #UXSCOT 2

3. Hands up…
If you’ve ever conducted a usability test
If you’ve ever watched a usability test
If all this is totally new to you
3NEIL ALLISON @USABILITYED #UXSCOT

4. Usability testing
IT’S EASY, RIGHT?
NEIL ALLISON @USABILITYED #UXSCOT 4

5. What can be challenging
Getting the go ahead to use your time on usability testing
Getting colleagues to take on board what you uncover
Getting fixes to problems implemented
NEIL ALLISON @USABILITYED #UXSCOT 5

6. These are common
challenges…
NEIL ALLISON @USABILITYED #UXSCOT 6

7. Standing on the shoulders of…
Caroline Jarrett and Steve Krug surveyed
the UX community in 2012.
They blogged about why usability
problems go unfixed:
http://bit.ly/krug-jarrett-unfixed
NEIL ALLISON @USABILITYED #UXSCOT 7

8. A story about change
MY CHALLENGE IN 2013…
NEIL ALLISON @USABILITYED #UXSCOT 8

9. Moving out of our comfort zone…
I was the University’s CMS Service Manager, responsible for user support and training,
community engagement, user research and requirements validation, co-design facilitation…
We had a vendor-supplied CMS that everyone hated, with a service around it that people quite
liked.
Features were introduced in waterfall projects. We had a small working group delivering that
was stable, working in a user-centred fashion. And we trusted each other.
Then in 2012 it was announced we were switching CMS, going open source and developing our
own system…
NEIL ALLISON @USABILITYED #UXSCOT 9

10. Changing times, new challenges
New people, new methodology, new urgency…
A development team of 3 plus a business analyst, working agile
to deliver a new CMS and transition the service in 2 years.
We didn’t have a culture of user-centred design.
The project team needed to be won over.
The CMS user community wanted reassurance.
Old ways of working were not going to fit this new situation…
NEIL ALLISON @USABILITYED #UXSCOT 10

11. Steve Krug to the rescue
Free to use observation session materials
Everyone listing top 3 issues for
each test participant
Bringing the development team closer
to the CMS user group that the
service team knew so well
11NEIL ALLISON @USABILITYED #UXSCOT

12. Good, but not great (sorry Steve)
Testing every 4 weeks.
Live observations, the CMS user group keen to get involved.
Team (broadly) happy to dedicate time.
But after the tests, discussion meandered.
People hung on to pet issues. Tech solution chat dominated.
We would run out of time, no closure.
NEIL ALLISON @USABILITYED #UXSCOT 12

13. Plus we wanted to go public
We wanted to bring in the user community as contributors, not just as test participants
The post-test conversations were meandering with 6-8 people.
What would happen with 40?
NEIL ALLISON @USABILITYED #UXSCOT 13

14. Enter David Travis
“RED ROUTE” USABILITY TO THE RESCUE
NEIL ALLISON @USABILITYED #UXSCOT 14

15. An objective prioritisation flowchart
Three questions to grade every usability issue
Post observation prioritisation complete
in 30-40 minutes
Scalable: works the same with 30+ people
NEIL ALLISON @USABILITYED #UXSCOT 15

16. The process
HOW TO DO IT YOURSELF
NEIL ALLISON @USABILITYED #UXSCOT 16

17. Step-by-step
1. Get the right people in a room
2. Watch a small number of short sessions with users doing something
3. Prioritise the issues they see
4. Collaboratively consolidate their priority lists
5. Agree actions for usability issues
6. Repeat every few weeks
NEIL ALLISON @USABILITYED #UXSCOT 17

18. Who are the right people?
Everyone with a stake in the service
If you’re making design decisions you should be there
“Have you had your recommended
dose of research?”
http://bit.ly/gds-research-dose
18NEIL ALLISON @USABILITYED #UXSCOT

19. What do we watch?
Real users doing real tasks
Facilitated usability testing sessions
Agree the focus of testing within the team
“Research shows that teams make better services when
everyone on a project team observes users first hand.”
Jared Spool
http://bit.ly/spool-exposure
19NEIL ALLISON @USABILITYED #UXSCOT

20. How many do we watch?
As many as you can fit into the time you have
Example:
◦ Intro – 10 min
◦ Test session 1 + 5 minutes
◦ Test session 2 + 5 minutes
◦ Test session 3 + 5 minutes
◦ Issue prioritisation – 40 minutes
“The most striking truth of the curve is that
zero users give zero insights.”
Jakob Nielsen
http://bit.ly/nielsen-5-users
20NEIL ALLISON @USABILITYED #UXSCOT

21. How do we prioritise?
Rocket Surgery template:
◦ Individual notes while observing
◦ Distil to 3 issues after each participant
“Running a usability test has been compared
with taking a drink from a fire hydrant…”
David Travis
21NEIL ALLISON @USABILITYED #UXSCOT
http://bit.ly/usability-prioritise-travis

22. How do we consolidate?
“If you prioritise usability problems using
'gut feel' or intuition, you run the risk
of being exposed as a fraud…”
David Travis
http://bit.ly/usability-prioritise-travis
22NEIL ALLISON @USABILITYED #UXSCOT

23. NEIL ALLISON @USABILITYED #UXSCOT 23
xxx

24. Taking action on the insight
This matrix is a great way to separate issues
from ‘solutions’
You might put the same issue on the matrix
more than once, if multiple solutions exist
Agree the fastest, cheapest way to validate
your solution
Minor
issue for
users
Major
issue for
users
Easy
solution
available
No easy
solution
available
24NEIL ALLISON @USABILITYED #UXSCOT

25. Top tips
Testing focus
◦ Agree priority tasks together
◦ Trial test with the team
◦ No surprises!
Participants
◦ Recordings are far less stressful than live
review sessions
◦ Krug – “Recruit loosely, grade on a curve”
Do whatever it takes to get observers in the
room
◦ Start over lunch break
◦ Supply refreshments
◦ Bribery, favours, threats…
Be well organised so you don’t waste the
observer group’s time
◦ Test everything before hand
◦ Provide a schedule in advance
Stick to the process
◦ It’s easy to digress when you’ve all seen so
much
Encourage collective reflection on the session
◦ Admitting usefulness is first step to getting
observers to turn up next time
◦ Have the next one scheduled ASAP
NEIL ALLISON @USABILITYED #UXSCOT 25

26. Everything you need
Steve Krug’s Rocket Surgery resources
http://bit.ly/rocket-surgery-krug
David Travis’ Red Route prioritisation flowchart
http://bit.ly/usability-prioritise-travis
Digitise with Post-it Plus app:
http://bit.ly/post-it-plus
26NEIL ALLISON @USABILITYED #UXSCOT

27. What happened next
A TOOL FOR CULTURE CHANGE & ADOPTION
NEIL ALLISON @USABILITYED #UXSCOT 27

28. Usability testing showcases
I approached a number of business-critical services and offered usability testing for free on the
condition that we reviewed as a collaborative event.
This made the case for a pilot usability testing service which then grew into a pilot UX
consultancy service.
These ‘freebies’ have developed into longer-term funded initiatives.
NEIL ALLISON @USABILITYED #UXSCOT 28

29. Learn VLE service enhancements
Collaborating with schools on a rolling basis
to expose teaching staff and learning
technologists to student experiences.
Making the case for greater consistency and
developing guidelines in the open.
Exposing the institution to the range of
approaches in existence, and the impact
on the student.
http://bit.ly/VLE-UX
NEIL ALLISON @USABILITYED #UXSCOT 29

30. IT self help iterative improvements
Bringing frontline and second line
support teams together around the
highest volume support call themes.
Creating a cycle of testing,
analytics and content development.
http://bit.ly/IT-self-service
NEIL ALLISON @USABILITYED #UXSCOT 30

31. In conclusion
Involve the team at every stage if you want usability testing to have impact
◦ From setting a focus through to review and prioritisation
Work to establish a virtuous cycle that others can adopt
◦ Focus on inclusion and empowerment
◦ Little of this is “specialist” – it’s everyone’s responsibility
◦ This only works if you keep it going
Few read your reports and fewer believe what you write
◦ Give them real exposure to users
◦ Channel the group to write their own “report”
Go as public as you can get away with
◦ The greater the exposure, the bigger the organisational ripples
NEIL ALLISON @USABILITYED #UXSCOT 31

32. Thank you
QUESTIONS?
N E I L A L L I S O N
U X M A N A G E R , U N I V E R S I T Y O F E D I N B U R G H
@ U S A B I L I T Y E D
NEIL ALLISON @USABILITYED #UXSCOT 32