In this talk from Bunnyfoot founder Dr Jon Dodd at Unboxed 2019 (held at the Curzon cinema on the 16th of October), he explained how to determine if your customer research was giving you good data.
Jon showed how accurate customer testing is a vital part of user-centred design and how the data from it reduces knowledge gap between customer and management expectations. It can help any marketing campaign - but only if we understand our limitations and what good data looks like.
4. 4
A SUCCESSFUL FRAMEWORK: USER CENTRED DESIGN
An established process from the software
industry – the underpinning of UXD, Service
design etc.
THE IMPORTANT THING IS:
Consideration of, and engagement with, your
audience at each stage of the development
Essentially driving design decisions based on
evidence rather than assumption
aka ‘Evidence based design’
5. 5
IN USER CENTRED DESIGN
learn about your
Business
¡ Business requirements/
stakeholder workshop
¡ Participatory
design/concept testing
¡ Strategy and goal
workshops
¡ Requirement analysis
planning
¡ Business to customer
uncovery
¡ Desk research
learn about your
Competitors
¡ Customer research/
benchmarking
¡ Best practice
analysis/feature analysis
learn about your
Customers
¡ Behaviour observation
§ Observed browsing
§ Customer Inquiry
§ Ethnography
§ Competitor testing
¡ Diary studies
¡ Audience surveys
¡ Depth interviews
¡ User/Customer requirement
workshop (focus groups)
define the
Information
architecture
¡ Information architecture
development
¡ Content strategy
develop the
Interaction design
¡ Paper prototyping
¡ Iterative wireframe
development
¡ Clickable prototype
development
document with
Blueprints /instructions
¡ Information architecture
blueprints
¡ Interaction design
documentation
§ Wireframes
§ Templates
§ Elements
long term
Support
¡ ‘Rent-a-guru’ ongoing ‘’on
call’ immediate access to
our experts
¡ Workshop, training and
seminars
¡ Policy documentation
regular
Monitoring
¡ Advanced visitor analysis
¡ Benchmark quantitative
testing
¡ Competitor
monitoring/trend spotting
¡ Social media monitoring
¡ Satisfaction surveys and
interviews
ongoing
Optimisation
¡ A/B testing
¡ Multi-variate testing
¡ Updated customer
intelligence
¡ Design Modification
updates
¡ Persuasion auditing
define
Customer Models
¡ Persona development
¡ Scenario and use case
setting
explore
Customer interaction
¡ Content audit and analysis
¡ Ideation/concepting
¡ Participatory design
workshop
¡ Mental modeling
¡ Mental model/content
model
develop
Testable concepts
¡ Concept rationalisation
¡ Information architecture
concept generation
¡ Storyboarding
¡ Lo-fi prototype sketching
research
Creative concepts
¡ Creative research
exploration
¡ Creative workshop
evolve and
Develop the creative
¡ Creative concept
generation
¡ Creative application
§ Templates
§ Elements
develop
Design guides/assets
¡ Design guide production
¡ Design asset production
§ Layered psd, png, etc,
§ Presentation layer code
RESEARCH MODEL ARCHITECT DESIGN OPTIMISE
¡ Expert usability eval
¡ User testing
¡ Accessibility audit
¡ Prototype testing
¡ Eye tracking
¡ Card sorting
¡ Navigation testing
¡ Look and feel testing
¡ Remote testing
¡ Analytics
…and more tailor made for you
TESTING AND EVALUATION
8. 8
IN SERVICE DESIGN
Management
expectations
Customer
expectations
Knowledge gap Do your research to reduce this
gap
Management
expectations
Service
specification
Try to set spec to meet customer
expectations
Policy gap
Service
Specification
Service
delivery
Attempt to deliver the spec
(resources etc.)
Delivery gap
Service
delivery
Customer
coms
Communication realistically and
truthfully
– help set expectations
Communication gap
Note: this is a model we use called the gap model to help frame service design
Customer
perception
Overall goal:
Reduce this gap!
but be careful with
exceeding
expectations
Reduce the knowledge gap
this is the difference between what the org thinks its users/customers want and need – and what they actually need.
9. 9
IN DESIGN SPRINTS
UXBoost
Define the
problem &
start the UX
Canvas
Learn from
others &
develop ideas
Rapidly
evolve ideas
& produce
visualisation(s)
Build suitable
prototype(s)
Test &
determine
next steps
Monday Tuesday Wednesday Thursday Friday
15. 15
THE BIG PICTURE
Kruger & Dunning 1999
…people who lack the knowledge or wisdom to
perform well are often unaware of this fact.
We attribute this lack of awareness to a deficit in metacognitive
skill. That is, the same incompetence that leads them to make
wrong choices also deprives them of the savvy necessary to
recognize competence, be it their own or anyone else’s.
“
23. 23
ASK YOURSELF REGULARLY – WHERE ARE YOU?
UX and research etc.
I hope I am here – every
day a school day
I did stand-up comedy
(4 years later I have
booked on a new
comedy writing course)
I’m about to
do my fourth
marathon
24. 24
This proves this
RESEARCH: SOME THINGS I HEAR REGULARLY
We have loads of
[survey] data
ask them this…
We need more
people (a bigger
sample)
I want [eyetracking, focus group,
interviews…]
You’re an expert –
why do we need
research
That wasn’t a good test…
(moderator didn’t say enough)
(a child kept running in)
But we know our
customers
Just ask them what
they think about
this
26. 26
RESEARCH: WHAT WE WOULD LIKE TO SEE MORE OF
What’s the best
approach(s)…
These are our
research
objectives
How can we
improve…
This is our problem
Every day is a
school day
27. 27
You are not your audience/user/customer *
A FUNDAMENTAL TRUTH
You do not…
¡ See things like they do
¡ Know what they know
¡ Want what they want
¡ Work how they work
This is critical information when designing a product or service
…and btw your bosses have even less clue than you do…
* delete as appropriate!
…that you will often have to remind yourself and your colleagues about
30. 30
DON’T BASE DECISIONS ON ASSUMPTIONS
Don’t Make Me Think (Revisited), 2014
It’s only natural to assume that everyone uses
the Web the same way we do, and - like everyone
else – we tend to think that our own behaviour is
much more orderly and sensible than it really is
Steve Krug, 2014
“
Give this book a read, or buy
it for your boss
31. 31
GARBAGE IN – GARBAGE OUT
It is essential to choose the
right methodology !
39. 39
ULTIMATELY...
People make decisions in the unconscious...
People don’t remember...
People lie (confabulate)...
People post rationalise...
People hate to appear stupid...
People are bad at introspection...
People will be kind to you and tell you what they think you want to
hear
40. 40
and…most of our behaviour including ‘complex’
behaviour is non-conscious (sub-conscious)
- So we need to design (and design our
research) with this in mind
41. 41
THIS OLD CHESTNUT...
If I had asked people what they wanted...
...they would have said faster horses
Henry Ford
51. 51
LETS LEARN FROM SOME BLOOPERS & HORROR STORIES
Every day is a school day…
52. 52
RESEARCHING THE WRONG PEOPLE…
Under-specification
Over-specification
Bias
What about the ‘unknown unknowns’?
53. 53
RESEARCHING THE WRONG PEOPLE…
‘Professional’ research participants
Fakes
Dunning Kruger sufferers (usage, expertise)
The lovelorn
(the smelly)
54. 54
RESEARCHING THE RIGHT PEOPLE…
Take recruitment very seriously
Test screeners and modify appropriately
Know the flex points
Go where they flock
55. 55
FALLING IN LOVE WITH A METHOD – IF YOU HAVE A HAMMER EVERYTHING IS A NAIL…
The tyranny of eyetracking
The ease of the survey deployment
The fame of the focus group
56. 56
= BE PROMISCUOUS WITH METHODS
Start with the research objectives
Use a mix of methods
Be innovative
60. 60
= BE FLEXIBLE, BE INTERESTED
Be prepared (and confident) to explore
Be interested (and look like you are)
Have empathy
Respond appropriately
61. 61
THE WRONG NUMBER
Often misinterpreted
Too many = wasted ROI = too much to
analyse (paralysis through analysis)
Scientific significance v trend v
confidence
63. 63
AND LOTS MORE TO WATCH OUT FOR
Confirmation bias (selective nature for research particularly important)
Expectation (experimentors/observer) bias (along with confirmation bias dangerous for the truth)
Availability heuristic (take notes, independent observers, group analysis)
Courtesy bias (look out for people doing this – occurs a lot in focus groups, at the end of user testing)
Curse of knowledge (especially important in research – domain experts lack empathy for users)
Group think (focus groups especially)
Interoceptive bias (well fed etc. changes outlook of researcher/judge)
Mere exposure effect (familiarity breeds… in this case positivity – important in ratings etc. e.g. satisfaction)
False consensus effect (you are not your user… and neither is your husband, wife, mother etc…)
64. 64
CONCLUSIONS
Assess where you are on the curve
Every day is a school day
Observe behaviour where possible
Use mix methods and be flexible
Avoid and learn from bloopers
65. 65
CONCLUSION
Communicating and interacting with customers and ‘converting them’ is a complex proposition
Only by understanding them and their needs can we hope to support their goals and influence their decision
making
Question your existing methods and existing suppliers…
Using a mix of methods is important
¡ choose from the matrix of techniques (and both big and little data) and apply appropriately at the
appropriate time
¡ Don’t fall in love with the method or an output
¡ Focus on research objectives and outcomes (outcomes not outputs!)