Evaluation Methodologies and How To Apply Them - Evaluation methodologies are the principal means by which the UX specialist answers the questions ‘What needs to be done?’, and after an interface manipulation has occurred, ‘What is the effect of this manipulation?’

1. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
The User Experience
from 30,000ft
#comp33512
Week 07 – Lectures 15/16
Simon Harper
University of Manchester
Semester 2 – 2013/14
last update: March 19, 2014
The User Experience from 30,000ft 1 / 22

2. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
UX Pop Quiz
1. What is the scientific method and why is it important?
2. What do we mean by internal and external validity?
3. What is the single most important reason for having a set of
ethical procedures?
4. What are the eight key ethical principles (give a brief
rationale for each)?
5. Why is conforming to scientific principles key to good ethical
designs?
The User Experience from 30,000ft Preamble 2 / 22

3. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Recap – Hat Racks for Understanding #1
Data Collection/Analysis
The User Experience from 30,000ft Recap 3 / 22

4. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Recap – Hat Racks for Understanding #1
Data Collection/Analysis
Participant Observation;
Task Analysis;
Focus Groups;
Interviews;
Social; and
Archive.
The User Experience from 30,000ft Recap 3 / 22

5. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Recap – Hat Racks for Understanding #2
Dissemination
The User Experience from 30,000ft Recap 4 / 22

6. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Recap – Hat Racks for Understanding #2
Dissemination
User Stories;
Use Cases;
Persona;
Scenario;
Wireframe;
Mock-up / Wizard of Oz;
Storyboard;
Flow Chart;
State Transition / State Machine; and
UML.
The User Experience from 30,000ft Recap 4 / 22

7. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Expert Evaluation via the Audit
Walkthroughs; and
Heuristic Evaluation.
Think!
Kinds of questions you are asking, expertise of the ‘expert’.
The User Experience from 30,000ft Techniques 5 / 22

8. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
What Questions Would YOU Ask?
The User Experience from 30,000ft Techniques 6 / 22

9. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Fieldwork
In-Situ / In the Wild;
Anthropological - small sets of people.
Difference is in the intention - Not to inform design - but to
evaluate the correctness!
The User Experience from 30,000ft Techniques 7 / 22

10. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Unobtrusive
Problems with ’normal’ Observation.
Guinea Pig: people feel like guinea pigs, therefore
change their behaviour patterns;
Role Selection: participants see the experimenter as
having an elevated status, therefore
following that experimenters lead;
Measurement as Change: aspects of experiment introduces real
changes, and
Response Sets: respondents will more frequently
endorse a statement than disagree with
its opposite.
Design criteria to avoid these.
The User Experience from 30,000ft Techniques 8 / 22

11. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
What Criteria Would YOU Enact?
The User Experience from 30,000ft Techniques 9 / 22

12. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Longitudinal Observation
Observation of user behaviour when interacting with applications
and the Web – especially for the skill gaining process – is better
observed at a longitudinal fashion. This statement is founded on
the fact that increased intervals enable the consolidation of
declarative knowledge in long-term memory, where consolidation
does not happen automatically and it is not determined at the
time it has been learned. In order to gain insights into the user
experience in the context of your development, we should conduct
a longitudinal analysis of those users if at at all possible.
The User Experience from 30,000ft Techniques 10 / 22

13. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Let’s Have a Break!
Back in 10 Minutes!
Come see me now if you have
Questions Regarding this Lecture!
The User Experience from 30,000ft Break! 11 / 22

14. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Quantitative
Time etc
Laboratory, experimental - not well used in company based UX.
Card Sorting
Sorting representations into an order based on some criteria.
Triadic elicitation
Understand differences which are not immediately obvious.
The User Experience from 30,000ft Break! 12 / 22

15. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Analytics
PULSE Page views, Uptime, Latency, Seven-day
active users, and Earnings and
HEART Happiness, Engagement, Adoption,
Retention, and Task success.
Notice Anything?
The User Experience from 30,000ft Break! 13 / 22

16. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Analytics
Figure: ‘Google Analytics’
The User Experience from 30,000ft Break! 13 / 22

17. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Analytics
Experience monitoring a qualitative representation of a single
session
Mindshare goals qualitative measures such as awareness,
branding effectiveness.
Customer support responsiveness and Customer satisfaction
evaluation.
The User Experience from 30,000ft Break! 13 / 22

18. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Analytics
Figure: ‘Facebook Happiness Measure’
The User Experience from 30,000ft Break! 13 / 22

19. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Think-Aloud
Classic of the UX evaluation process evolving mainly from design
based approaches. It produces qualitative data and often occurs
as part of an observational process, as opposed to a direct
measurement of participant performance, as would be normal in
laboratory settings.
The User Experience from 30,000ft Break! 14 / 22

20. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Co-Operative Evaluation & Participatory Design
Enabling each individual to fully interact within the discussion
process while the UX specialist remains outside of the discussion
just acting as a facilitator for the participants views and thoughts
is a key factor in the process design.
Think Pair-Programming with Users!
The User Experience from 30,000ft Break! 15 / 22

21. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Survey Questionnaires – Reprise
The User Experience from 30,000ft Break! 16 / 22

22. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
The Great Measurement Debacle of 2012
Nominal Scale: Which denotes identity;
Ordinal Scale: Which denotes identity and magnitude;
Interval Scale: Denotes identity, magnitude and has the benefit of
equal intervals; and
Ratio Scale: Which has the positive properties of the three we
have already seen as well as a true zero point.
The User Experience from 30,000ft Break! 17 / 22

23. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Hybrid Methods
...known also as mixed methods or triangulation
Are terms used to denote the use of many complimentary
methods because the UX specialist recognises the inadequacies of
a single method standing alone. Indeed, the hallmark of being a
field practitioner is flexibility in relation to theoretical and
substantive problems on hand.
The User Experience from 30,000ft Break! 18 / 22

24. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Tools of the Trade
Pen and Paper
The User Experience from 30,000ft Tools of the Trade 19 / 22

25. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Tools of the Trade
Field Audio
The User Experience from 30,000ft Tools of the Trade 19 / 22

26. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Tools of the Trade
Field AV
The User Experience from 30,000ft Tools of the Trade 19 / 22

27. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Tools of the Trade
Electro-Encephalo-Graphy (EEG)
The User Experience from 30,000ft Tools of the Trade 19 / 22

28. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Tools of the Trade
Eye-Tracking
The User Experience from 30,000ft Tools of the Trade 19 / 22

29. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Tools of the Trade
Galvanic Skin Response
Figure: ‘GSR’
The User Experience from 30,000ft Tools of the Trade 19 / 22

30. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Pop Quiz for next week...
1. What are qualitative methods and how do they differ from
quantitative ones?
2. What are the key problems with laboratory based work?
3. What problems may exist when undertaking single method
evaluation?
4. Why is co-operative evaluation different from other methods?
5. What tools are at the disposal of the ‘poor’ UXer?
The User Experience from 30,000ft Wrapping Up 20 / 22

31. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
To Do for next week...
1. Revision Questions;
2. Pop Quiz, Discuss Next Week; and
3. Read your notes.
The User Experience from 30,000ft Wrapping Up 21 / 22

32. Preamble Recap Techniques Break! Tools of the Trade Wrapping Up
Any Questions?
Simon Harper 2.44 Kilburn Building
0161 275 0599 (OR x50599)
simon.harper@manchester.ac.uk
Office Hours: Friday 14:00–18:00
Figure: ‘Still Here!’
The User Experience from 30,000ft Wrapping Up 22 / 22