Michael Herdman

1. 1
Using Patient Reported Outcome measures
in a cross-cultural context:
conceptual and linguistic issues
Michael Herdman,
Insight Consulting & Research
Relatively little
known (still) about
the interaction
between culture and
PRO measurement
PRO instruments
increasingly used in
cross-cultural or
multi-country studies
and in a wide range
of countries
Important to take
into account
because of:
• impact on study
results
• relevance of
culture at all points
in the measurement
process

2. 2
DEVELOPMENT
VALIDATION
ADAPTATION
APPLICATION
ANALYSIS
INTERPRETATION
Relevance of the concept across cultures
Operationalisation of the concept across
cultures (domain/items/response formats)
Translation and linguistic testing
Psychometric performance in different
cultural settings
• Response style
• Mode of administration
•MIDs
• Effect sizes
Testing for a ‘culture/country effect’ in
multi-country studies & taking any such
effect into account
Herdman M, Fox-Rushby J, Badia X. A model of equivalence in the cultural adaptation of HRQoL instruments: the universalist
approach. Qual Life Res 1998;7(4):323-35.
DEVELOPMENT
VALIDATION
ADAPTATION
APPLICATION
ANALYSIS
INTERPRETATION
Relevance of the concept across cultures
Operationalisation of the concept across
cultures (domain/items/response formats)
Translation and linguistic testing
Psychometric performance in different
cultural settings
• Response style
• Mode of administration
• MIDs
• Effect sizes
Testing for a ‘culture/country effect’ in
multi-country studies
Based on: Herdman M, Fox-Rushby J, Badia X. A model of equivalence in the cultural adaptation of HRQoL instruments: the
universalist approach. Qual Life Res 1998;7(4):323-35.

3. 3
Cross-cultural relevance of the concept measured, and its
operationalisation
• An alternative to the ‘export’ approach, is the simultaneous development of a
measure in in different cultural settings
• Evidence in the literature or elsewhere that suggests the concept (construct) is
structured/defined similarly in different cultures? Anthropological literature, sociology, clinical
experts, etc. E.g. if we are assessing health status, what are the important domains in different
cultures? How is the concept expressed?
• Post-development testing of instrument relevance in other cultural settings, e.g. in individual
interviews or focus groups
• Use of an instrument outside the cultural setting in which it was developed
implies ‘exporting’ the particular model of health/QOL/well-being, etc on
which it is based
• We do little to assess how appropriate the model might be outside its original
setting, e.g.:
13 European
countries
Most (all?)
content derived
in Europe
Fourteen countries
worldwideSix European
countries

4. 4
Cross-cultural relevance of concept, and construct
operationalisation
• What about formal aspects, such as response formats? Do we need to
test these across cultures for:
 Acceptability/preference (e.g. 3 level vs 5 level scales in EQ-5D)
 Susceptibility/robustness to culture-related effects, e.g. response style?
• Outcome relevance: how appropriate are existing (Western) measures
of health for use in all countries that might be included in a clinical
trial? How well do they reflect local conceptions of a construct such as
health status?
• Western-centric measurement bias in PROs?

5. 5
Linguistic/adaptation issues; some examples with EQ-5D
• Standard approach (FT/BT/cognitive testing) to cross-cultural adaptation may
not be sufficient
• EQ Version Management Committee incorporated a VAS severity rating
exercise into the cultural adaptation process + stricter requirements for
cognitive debriefing + experimentation with choice-based exercises
• Translation of severity labels a key concern in PBMs. Labels used for both
description and valuation of health.
• e.g. ‘slight’, ‘moderate’, ‘severe’, ‘extreme’ in EQ-5D-5L.
• Importance of adaptation process, when instruments are to be used in multi-
country trials or if we want to compare/aggregate data across countries.
I have slight problems doing my usual activities
I have moderate problems doing my usual activities
I have severe problems doing my usual activities

6. 6
Source
English
Wording
What does this mean to you in
your own words?
EXAMPLES OF POOR QUALITY
RESPONSES
What does this mean to you in
your own words?
EXAMPLES OF GOOD QUALITY
RESPONSES
I have slight
problems
in walking
around
R1 – hard to walk
R2 – I have some difficulty walking
R3 – it’s not easy for me to walk
R4 - As written
R5 – just what it says
R1 – “A part or the totality of my leg is
hurting or prevents me from
walking. Like when I choose the
wrong shoes. Sometimes I forget
my working shoes and it gives me
discomfort. My mother has to move
slowly when she goes to the
supermarket”
R2 –“It describes I have some
discomfort when I walk. When I do
my toe nails and sometimes it hurts.
My neighbor uses portable oxygen
for walking, so it makes him more
difficult to walk around”
R3 –“Mild discomfort when walking.
The day after I go for a soccer
match and I was kicked. It is not
that hard. When my mum was
operated from the breast, she had
discomfort. It was difficult for her to
move around”
Improving
cognitive
debriefing
results
Testing for ‘preference inversion’ in the US English 5L (Craig et al, 2015)
When applied to Brazilian Portuguese translation, effects were more noticeable:
approximately 60% preferred ‘graves’ over ‘extremos’ [first option reminds
respondents they are close to the ‘grave’??]

7. 7
• Substantial differences in response style and response patterns
between cultures
Response style and response patterns
• ‘Response style’ refers to a respondent’s tendency to systematically
respond to questionnaire items in a particular way regardless of item
content (Baumgartner and Steenkamp, 2001). E.g. extreme response style
and acquiescent response style
• Relatively little attention paid to these aspects in the PRO field
• Investigation of ‘response patterns’ involves descriptive exploration
of data at the population level
Chuansheng
Chan et al 1995
 Response style on rating scales between high school students from Japan (n=944), Taiwan,
Canada, the US.
 Responses to 57 7-point Likert-type scales were analyzed.
 Japanese and Chinese students more likely than North American groups to use the
midpoint on the scales
 U.S. subjects more likely than the other three groups to use extreme values.
Stening E et al,
1984
 Asessed response style using 7 point semantic differential items in 9 different nationalities,
including Japanese (males/managers)
 Japanese showed greatest tendency to use the scale (‘neutral’) mid-point and to avoid
scale extremes
Zax M et al, 1967  Assessed tendency to extreme responses style in samples of American and Japanese college
students (n=80 in each country, evenly split by male and female)
 Ratings on Semantic Differential scales using Rorschach ink-blots
 Japanese respondents tended to use Neutral (or non-rateable) responses more often
than American respondents, and also used the Extreme responses less
Studies on response style in Japanese respondents

8. 8
Meta-analysis of VAS scores from two studies comparing dienogest (DNG) versus gonadotropin-
releasing hormone (GnRH) analogues for the treatment of endometriosis-related pain in European and
Japanese populations (Gerlinger et al, 2012)
Response patterns
On EQ-5D, almost 87% of respondents reported 11111 [full
health] in China, compared to 45% in the UK, 42% in
Sweden, and 66% in Germany.
In general population studies, three health states (11111,
11121, 11122) accounted for 90.1% of Japanese
respondents compared with twelve health states in the UK
(90.6% of UK respondents)

9. 9
Conclusions
• There’s a lot we don’t know! So, great scope for further research into cross-
cultural issues around PRO use, and optimisation of current methodologies, e.g.:
move away from a Western-centric approach + more investigation of
the outcome relevance outside Western settings
more research on response style, and how to deal with it
more research into response patterns and their underlying causes
experimentation with response formats – cross-cultural suitability
and robustness to cultural effects
new/improved translation testing methods, especially for PBMs
• Caution with comparing, aggregating, transferring data
from/between different cultural settings