Conference presented at the UX Strat Europe 2016 conference in Amsterdam by Dr. Carine Lallemand (University of Luxembourg).
Abstract:
While conducting UX research, we make several conclusions that will in turn provide the foundation for our UX strategy. But what if these inferences happen to be wrong, based on invalid findings and false beliefs? How critically would this impact your organization and projects? How can you safeguard a UX strategy by ensuring the quality of research conclusions?
There might be numerous threats to validity in UX research, some of which might depend on the method used or the way it is used. A method is only a guide to action that needs to be configured, adapted, and complemented to match specific project requirements. To be successful, it is essential to ensure validity in strategic UX research methods. Failing to do so is taking the risk to base strategic decisions on false beliefs. In this talk, we will therefore see how to tackle validity issues and make the most out of UX research to stand out from the crowd by delivering value and differentiation. Through the presentation of validated cutting edge UX methods and business cases, you will be able to spot opportunities for improvement in your UX strategy!
3. 01
WE NEED TO BASE STRATEGIC DECISIONS
ON VALID FINDINGS
4. 01
COLLECTING VALID AND RELIABLE DATA?
â> VALID & RELIABLE â> CONTEXTUALIZED & DYNAMIC
Asking SIRI?
Making a good
guess?
On-site live wind
measurement
Meteorological information
provided by ofïŹcial services
5. How can you safeguard a UX strategy by
ensuring the quality of research
conclusions?Â
6. Whenever we measure or observe we should be
concerned with whether we are measuring what we
intend to measure or with how our observations are
inïŹuenced by the circumstances in which they are made
8. 01
WORLD IA DAY 2016 PRESENTATION TITLE HERE
<STRATEGIC THINKING
>Doing the right things
= Using the right methods
TACTICAL THINKING
Doing things right
= Using the methods right
10. « Strangely, while I ïŹnd the proposition to consider the experience
before the thing quite a radical change, many practitioners and
academics of HCI happily embrace experience â however, without
changing much in their approach. »
-Prof. Marc Hassenzahl (2013)
11. The nature and complexity of UX involves a deep
change in the methods we use
UX is highly dynamic
The memory of an experience matters more than the
experience itself
UX is highly contextual
UX is holistic1
2
3
4
5 UX is about emotions and psychological needs
12. UX is holistic
ThĂŒring & Mahlke, 2007
A systemâs perceived attractiveness is based on the perception of
its pragmatic and hedonic qualities
System
User
Context
Interaction
characteristics
Perception of non-instrumental qualities
Emotions
Perception of instrumental qualities
Components of User Experience
Consequences
overall evaluation,
acceptance, intention to
use, choice of alternatives
1
13. System
User
Context
Interaction
characteristics
Perception of non-instrumental qualities
Emotions
Perception of instrumental qualities
Components of User Experience
Consequences
overall evaluation,
acceptance, intention to
use, choice of alternatives
Usability scales
(SUS, QUIS, SUMI, WAMMI, etc)
established usability questionnaires focus on pragmatic aspects
only⊠this is not enough!
1
UX is holistic
ThĂŒring & Mahlke, 2007
14. We need to assess both pragmatic and hedonic
perceived qualities of a system
AttrakDiff scale
(Hassenzahl et al., 2003)
User Experience
Questionnaire
(Laugwitz et al., 2008)
meCUE scale
(Minge & Riedel, 2013)
1
Using standardized and validated UX scales
15. The AttrakDiïŹ scale: a standardized UX assessment tool
28 items (word pairs) divided into four subscales:
âą Pragmatic qualities
âą Hedonic qualities - stimulation
âą Hedonic qualities - identiïŹcation
âą Attractiveness
âą Single evaluation
âą Comparison product A - product B
âą Comparison Before - After
An abridged version (10 items)
Portfolio of results (comparison A vs. B)
(Hassenzahl et al., 2003)
16. UX is highly contextual
Context
User System
Social context
Technical context
Temporal context
Task context
Physical context
Time
2
17. user testing in a
controlled
environment
expert evaluation
traditional evaluation methods assess UX in an
artiïŹcial environment
Context
User System
Time
2
UX is highly contextual
18. How does UX alter laboratory evaluation?
Study conducted in 2015 (Lallemand et al., 2015)
Experiment involving 70 users, who were asked to
evaluate their UX with two systems
Research objective: assessing the quality and limitations
of « lab testing » for the evaluation of UX.
Results:
- validity of our assessment was limited to only the
pragmatic aspects of the interaction
- signiïŹcant order effects
- impact of the scenarios of use on the felt experience
- impact of the environment and the lack of ecological
validity
- lack of evaluation of the dynamics of the experience
19. We need to evaluate UX in a natural or realistic setting
Field testing and
observation
"In-sitro" user testing
(Kjeldskov et al., 2004)
Experience sampling
(Csikszentmihalyi , 1990)
2
Ecological validity and the « turn to the wild »
20. Before usage
Anticipated UX
Imagining experience
During usage
Momentary UX
Experiencing
After usage
Episodic UX
ReïŹecting on an
experience
Over time
Cumulative UX
Recollecting multiple periods
of use
When:
What:
How:
UX White Paper, 2010
There are several time spans of UX
UX starts before the interaction and doesnât end immediately
after the interaction
UX is highly dynamic
3
21. traditional or psychophysiological evaluation
methods focus on momentary UX⊠this is
not enough!
UX White Paper, 2010
user testing psychophysiological
measurements
Before usage
Anticipated UX
Imagining experience
During usage
Momentary UX
Experiencing
After usage
Episodic UX
ReïŹecting on an
experience
Over time
Cumulative UX
Recollecting multiple
periods of use
When:
What:
How:
UX is highly dynamic
3
22. The memory of an experience matters more than the
experience itself
Episodic UX is a reconstruction,
a remembered experience biased by cognitive
processes
The momentary experience is not as important
as the way it is remembered.
Itâs the memory of an experience that inïŹuences
userâs behavior and the way he talks or
recommends the product to someone
4
23. We need to assess UX across time and to focus
on the memory of experiences
UX Curve
(Kujala et al., 2011)
Diary methods
Retrospective UX assessment
Analytic scale
(Karapanos et al., 2010)
Longitudinal study
3 4
24. The UX Curve method: retrospective UX evaluation
What they will tell you is biased by their memory, it is not similar to how they really felt
What matters is how they remember the experience with your system because they will
behave on this basis.
Unvalid, yet reliable?
performed simply on the basis of whether the starting point of the
curve was higher or lower compared to the end point. For example,
the curve in Fig. 2 was categorized as being improving as its start-
ing point was lower than its ending point, even though the curve
deteriorates in the middle. If the starting and ending points were
at the same level, the curve was categorized as stable. As the curves
were freehand drawings, they were categorized as stable if there
was a very small deviation (less than one millimeter) between
the vertical values of the starting and ending points of the curve.
However, it can be seen from Figs. 3â10 that the categorization
was rather straight-forward to do with the three trend type catego-
ries. The relationships between the curve types and the key
Fig. 4. The deteriorating and stable general UX Curves with user IDs.
Fig. 5. The improving Attractiveness curves with user IDs.
Fig. 7. The improving ease of use curves with user IDs.
Fig. 8. The deteriorating and stable ease of use curves with user IDs.
Fig. 4. The deteriorating and stable general UX Curves with user IDs.
Fig. 5. The improving Attractiveness curves with user IDs.
Fig. 6. The deteriorating and stable Attractiveness curves with user IDs.
Fig. 7. The improving ease of use curves with user IDs.
Fig. 8. The deteriorating and stable ease of use curves with user IDs.
Fig. 9. The improving utility curves with user IDs.
Results: Mean attractiveness curves
8
3.6.2011
Facebook Mobile phoneLong-term UX curves (for a speciïŹc UX dimension)
Kujala et al., 2011
25. UX is about emotions and psychological needs
Thinking about the experience ïŹrst
Designing for emotions and psychological
needs
Using science-based design tools
5
28. Established evaluation methods only explore a limited
part of UX
single user testing
sessions
psychophysiological
measurements
expert evaluationusability scales
As we gain a deeper understanding of UX, we have to adapt
the methods we use to ensure validity
29. 01
WORLD IA DAY 2016
Some kind of illustration or image?
HEADER OPTION
SUB HEAD OR SHORT DESCRIPTION
Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.
USING THE
METHODS RIGHT
2
30. UX RESEARCH METHODS ARE
« Ingredients and Meals Rather Than Recipes »
âŠjust as the quality of what is cooked reïŹects the quality of its
ingredients, so does the quality of UX work reïŹect the quality of
resources as conïŹgured and combined. Woolrych et al., 2011
32. 01
SAMPLING: TARGETING THE RIGHT USERS
Making the most out of opportunistic sampling?
Probability sampling: process that gives all the individuals in the population equal
chances of being selected
Opportunistic sampling: the availability of participants guides on-the-spot sampling
decisions
Sample size vary in diïŹerent research settings. All else being equal, large sized sample
leads to increased precision in estimates of various properties of the population.
33. 01
WHAT ABOUT GUERRILLA RESEARCH?
A reasonable option?
Fast and cheap way to get a certain type of
feedback
âą Only for consumer-oriented product
âą Testing the understanding of the Value
Proposition or the usability of one speciïŹc
feature
Not always âbetter than no researchâ
35. 01
RESEARCH BIASES A few examples
Selection bias: one relevant group in the population has a higher probability of being
included in the sample.
â> Choosing a random or representative sample
Experimenter / interviewer bias: diïŹerential treatment of participants
â> Standardized procedures and instructions.
Expectancy / observer bias: the researcherâs expectations aïŹect the outcome of a study
â> Having independent observers and computing inter-raters agreement
Social desirability bias: the tendency of respondents to answer questions in a manner that
will be viewed favorably by others
â> Careful formulation of questions and items. Use of projective techniques.
37. 01
COMBINING QUANTITATIVE & QUALITATIVE
Understanding « how » and « why »
Quantitative research methods: rely on using large sample sizes to establish trends and
conclusions.
Qualitative research: appropriate for getting a more in-depth, contextual understanding
of why those trends occur.
The best research strategies incorporate both approaches
38. ProïŹle UnïŹnished sentence UX dimension
non ebook-
reader
Compared with a paper book, a digital book is⊠Comparison between products
In my opinion, digital books are addressed to⊠Identity / product image
I have never read digital books because⊠Frustrations / Barriers to use
I would read a digital book if⊠Expectations and needs
I expect a digital book to / that⊠Expectations and needs
When I read a paper book, I feel⊠Affects
ebook reader
Compared with a paper book, a digital book is⊠Comparison between products
The reading experience on a digital book is⊠Global UX
The problem with ebooks is⊠Issues and frustrations
What I love about ebooks is⊠Positive aspects / Appropriation factors
What frustrates me the most with a digital book is⊠Issues and frustrations
I find that the interface of a digital book is⊠SpeciïŹc UX - Interface
I dream of a digital book that⊠Expectations / Dreams
Ongoing study (Lallemand & Mercier, 2015)
Designing an optimal e-reading experience
39. 01
LIKERT SCALE VS. SENTENCE COMPLETION
On a 7-points Likert scale, how would you rate
your overall e-reading experience? (N = 1284)
Self-reported overall e-reading experience
(7 points Likert scale)
Valence Frequency Percent
Negative 228 17,8Â %
Positive 817 63,9Â %
Neutral 160 12,5Â %
Mixted 74 5,8Â %
« The reading experience on a
digital book isâŠÂ »
Valence analysis of sentence L_SC_2
Ongoing study (Lallemand & Mercier, 2015)
Designing an optimal e-reading experience
40. 01
LIKERT SCALE VS. SENTENCE COMPLETION
The problem with ebooks isâŠ
- the price
- the lack of availability and choice
- the absence of a sensual experience (feeling the
paper in oneâs hands)
- the navigation and information architecture
- the battery / the need for a network connexion
- their bad quality
- the impossibility to lend the book to a friend
- DRM (digital rights management)
- the bad reading experience
- the screen and visual fatigue
- it is dematerialized
- âŠ
that you donât see what people are reading
because you donât see the book coverâŠ
you can't skim or ïŹip through easily
Iâm not able to physically track my
progress in the book
Designing an optimal e-reading experience
Ongoing study (Lallemand & Mercier, 2015)
41. 01
VALIDITY & RIGOR, YET SCALED TO THE NEED
Basing strategic decisions on valid ïŹndings
Rigor should be proportional to the risk
Catching up on emerging research and using freely available valid
yet lightweight UX methods developed in Academia
Rethinking your Unique Value Proposition thanks to UX theories
42. THANK YOU FOR YOUR ATTENTION
Dr. Carine Lallemand
Twitter @carilall http://uxmind.eu
43. 01
REFERENCES
âą Csikszentmihalyi, M. (1990). Flow. The psychology of optimal experience, Harper and Row.
âą Hassenzahl, Marc (2013): User Experience and Experience Design. In: Soegaard, Mads and Dam, Rikke Friis (eds.). "The Encyclopedia of
Human-Computer Interaction, 2nd Ed.". Aarhus, Denmark: The Interaction Design Foundation.
âą Hassenzahl, M., Burmester, M., & Koller, F. (2003). AttrakDiff : Ein Fragebogen zur Mes- sung wahrgenommener hedonischer und pragmatischer
QualitĂ€t. In J. Ziegler & G. Szwillus (Eds.) Mensch & Computer 2003. Interaktion in Bewegung, 187â196. Stuttgart: B.G. Teubner.
âą Kahneman, D., et al., (2004). A survey method for characterizing daily life experience: The Day Reconstruction Method, Science, CCCVI(5), 702.
âą Karapanos, E., Martens, J.-B., & Hassenzahl, M. (2010). On the Retrospective Assessment of Usersâ Experiences Over Time : Memory or Actuality
? Proc. of CHI 2010, 2689-2698.
âą Kjeldskov, J., & Skov, M.B. (2007). Studying Usability In Sitro : Simulating Real World Phenomena in Controlled Environments. International Journal
of Human-Computer Interac- tion, 22(1-2), 7â36.
⹠Kujala,S., Roto,V., VÀÀnÀnen-Vainio-Mattila,K., Karapanos,E., &SinnelÀ,A. (2011). UX Curve: A method for evaluating long-term user experience.
Interacting with Computers, 23, 473-483.
âą Lallemand, C. (2015). Towards Consolidated Methods for the Design and Evaluation of User Experience. (Doctoral dissertation). University of
Luxembourg. https://publications.uni.lu/handle/10993/21463
âą Laugwitz, B, Held, T., & Schrepp, M. (2008). Construction and evaluation of a user expe- rience questionnaire. In A. Holzinger (Ed.) USAB 2008,
LNCS 5298. Berlin: Springer Verlag.
âą Lucero, A., & Arrasvuori. J. (2010) PLEX Cards : a source of inspiration when designing for playfulness. Proc. of Fun and Games 2010. New York,
USA: ACM, 28-37.
44. 01
REFERENCES
âą Minge, M., & Riedel, L. (2013). meCUE â Ein modularer Fragebogen zur Erfassung des Nutzungserlebens. Presented at Mensch und Computer
2013, Bremen.
âą Roto, V., Law, E., Vermeeren, A., & Hoonhout, J. (2011) User Experience White Paper: Bringing clarity to the concept of user experience. Result
from Dagstuhl Seminar on Demar- cating User Experience, Finland.
âą ThĂŒring, M., & Mahlke, S. (2007). Usability, aesthetics and emotions in human-technology interaction. International Journal of Psychology, 42(4),
253-264.
âą Yoon, J., Desmet, P. M. A., & Pohlmeyer, A. E. (2013). Embodied Typology of Positive Emotions: The Development of a Tool to Facilitate Emotional
Granularity in Design (pp. 1195â1206). Presented at the 5th International Congress of International Association of Sciences of Design Research,
Tokyo, Japan.
âą Woolrych, A., HornbĂŠk, K., FrĂžkjĂŠr, E. & Cockton, G. (2011). âIngredients and meals rather than recipes : a proposal for research that does not
treat usability evaluation methods as indivisible wholesâ. International Journal of Human-Computer Interaction, 27(10), 940-970
âą Adam Cooper, Cetis Blog 2014 - http://blogs.cetis.org.uk/
âą http://www.uxbooth.com/articles/complete-beginners-guide-to-design-research/
Download the UX Cards : http://uxmind.eu/portfolio/ux-design-and-evaluation-cards
Download the PLEX Cards : http://www.funkydesignspaces.com/plex/
Download the Positive Emotional Granularity Cards : www.diopd.org/emotioncards