1. Is it a good idea to optimise question
format for mode of data collection?
Gerry Nicolaas National Centre for Social Research
Pamela Campanelli Independent Survey Methods Consultant
Steven Hope University College London
Annette Jäckle University of Essex
Peter Lynn University of Essex
2. Overview
• Background
• Research question
• ‘Mark all that apply’ vs ‘yes/no for each item’
• Literature, Hypotheses, Methods, Results
• Branched questions vs non-branching
• Literature, Hypotheses, Methods, Results
• Cognitive interview results
• Methods, Results
• Conclusion
3. Background
• Face-to-face interviewing is dominant mode in UK
• Interviewer presence
• Aural stimuli & Visual stimuli (e.g. show cards)
• Many questions designed to be ‘optimal’ for face-to-
face interviews
• abundant use of show cards, long response lists, ‘mark all
that apply’, fully labelled scales, etc
• Push towards cheaper data collection modes and
mixed modes
• Some questions used in face-to-face interviews need to be
adapted for other modes
4. Research question
• Is it a good idea to ‘optimise’ question format for
mode of data collection?
• Increasing risk of differences in measurement across modes?
• 2 question formats that tend to be changed for
telephone interviews
• Mark all that apply
• Scales
6. Example of ‘mark all that apply’
F2F & Self-Completion Telephone interview
This card shows a number of I am now going to ask you a
different ways for reducing number of questions about
poverty. In your opinion, which of different methods for reducing
the following would be effective in poverty. In your opinion, which of
reducing poverty? the following would be effective?
SHOW CARD / MARK ALL Would increasing pensions reduce
o Increasing pensions poverty?
o Investing in education for children o Yes
o Improving access to childcare o No
o Redistribution of wealth
o Increasing trade union rights Would investing in education for
o Reducing discrimination children reduce poverty?
o Increasing income support o Yes
o Investing in job creation o No
o None of these
etc……….
7. ‘mark all’ vs ‘yes/no for each’
• Difficult to interpret absence of being marked
with ‘mark all’
• Does not apply or item non-response?
• Sudman & Bradburn (1982)
• Deeper processing of each item with ‘yes/no’
• More items selected with ‘yes/no’
• Respondents take more time to complete ‘yes/no’
• Primacy effects with ‘mark all’
• Rasinski et al (1994), Smyth et al (2006), Smyth et al
(2008)
• ‘yes/no’ performs similarly in tel & web modes
• Smyth et al (2008)
8. Our research
• Replicate & extend Smyth et al (2006 & 2008)
• Face-to-face mode as well as telephone and web
• UK general population with internet access
• Cognitive interviews as well as experimental data
9. Hypotheses
• Higher % of items chosen in tel mode with ‘y/n’
than in F2F & web with ‘mark all’
• If using ‘y/n’ in all 3 modes, no differences
• If using ‘mark all’ in F2F & web, no differences
• Deeper processing with ‘y/n’
• ‘y/n’ takes longer to complete than ‘mark all’
• more primacy with ‘mark all’ respondents with below
average completion times
• ‘mark all’ respondents with at least average completion
times select as many items as ‘y/n’ respondents
• Differences greater for difficult questions
10. Methods
• 2 split ballot experiments
• Easy attitude question about neighbourhood with 8 items
• Difficult attitude question about poverty with 8 items
• Mean number of endorsements for each format in
each mode
• Response latencies for each format in each mode
• OLS regression
• Control variables for differential non-response
11. 2 split ballot experiments
(1) Neighbourhood (2) Reducing poverty
This card shows a number of different
What are the things that you like ways for reducing poverty. In your
about your neighbourhood? opinion, which of the following would
be effective in reducing poverty?
SHOW CARD
o Its community spirit SHOWCARD
o It feels safe o Increasing pensions
o The neighbours o Investing in education for children
o The character of its buildings o Improving access to childcare
o Its cleanliness o Redistribution of wealth
o Its location o Increasing trade union rights
o Its quietness o Reducing discrimination
o Its transport facilities o Increasing income support
o Investing in job creation
o None of these
12. Results
• Hypotheses supported:
• higher % of items selected in tel with ‘y/n’ than F2F &
web with ‘mark all’
• ‘y/n’ takes longer to complete than ‘mark all’
• no differences when using ‘mark all’ in F2F & web
• But:
• when using ‘y/n’ in all modes, differences observed
between tel (& F2F) and web
• differences were greater for the difficult question
• no primacy among ‘mark all’ respondents with less than
average completion times
• ‘mark all’ respondents with at least average completion
times still selected fewer items
13. Differences with Smyth et al (2006 & 2008)
• Differences in study population
• Smyth et al: students at USA university
• This study: general population who used internet
• Differences in question design
• Smyth et al: 9-15 items
• This study: 8 items for both questions
15. Example of scalar question
Fully labelled scale Branched question
Please tell me whether you consider Please tell me whether you consider
your local shopping facilities to be your local shopping facilities to be
……. …….
o extremely poor, o poor or
o very poor, o good?
o poor,
o good, Would this be …….
o very good, or o (poor/good),
o extremely good? o very (poor/good), or
o extremely (poor/good)?
16. Full scale vs branching
• Branching makes the task easier for respondents
• Decomposition principle
• Armstrong et al (1975), Groves & Kahn (1979), Krosnick &
Berent (1993), Yu et al (2003), Malhotra et al (2008)
• Branching produces more extreme responses
• Reduces reluctance to select extreme responses
• Nicolaas et al (2000), Yu et al (2003)
17. Hypotheses
• More extreme responses in tel mode with
branching than in F2F & web with no branching
• This effect is more prevalent for attitudinal than factual
questions
• Within each mode, more extreme responses with
branching
• This effect is more prevalent for attitudinal than factual
questions
• If using branching in all 3 modes, no differences
18. Methods
• 4 split ballot experiments
• Easy attitude question about shopping facilities
• Difficult attitude question about standard of living
• Easy factual question about rent/mortgage
• Difficult factual question about grocery shopping
• Proportion of extreme & non-extreme responses
• Highest versus the lowest category
• Two highest versus the two lowest categories
• Logistic regression
• Control variables for differential non-response
19. 4 split ballot experiments
(1) Please tell me (2) Thinking back to the
whether you consider last general election,
your local shopping would you say that the
facilities to be ... standard of living has ...
READ OUT SHOW CARD
oextremely poor, oincreased by a large amount,
o very poor, o increased by a medium amount,
o poor, o increased by a small amount,
o good, o stayed the same,
o very good or o decreased by a small amount,
o extremely good? o decreased by a medium amount, or
o decreased by a large amount?
20. 4 split ballot experiments (continued)
(3) How much did your (4) How much did your
household spend last month household spend last month
in rent or mortgage for the on grocery shopping?
accommodation you live in?
READ OUT READ OUT
oless than £100 o less than £100
o £100 - £199 o £100 - £199
o £200 - £299 o £200 - £299
o £300 - £399 o £300 - £399
o £400 - £499 o £400 - £499
o or £500 or more? o or £500 or more?
21. Results
• Hypotheses supported or partly supported:
• more extreme responses in tel with branching than F2F
& web with no branching
• difference was greater for attitudinal than factual
questions
• within each mode, branching produced more extreme
responses but only for attitudinal questions
• Hypotheses not supported:
• within each mode no branching seemed to produce
more extreme responses than branching for factual
questions
• many differences observed across modes when using
branched format in all 3 modes
23. Mode effects with Y/N (CAPI vs CAWI) (1)
Subtleties going on that could affect aggregate
mode comparisons.
1. Instances of clear and possible satisficing
• Most of these were in the ‘yes’ category
• Most of these were in CAWI
2. Instances of Rs in the middle ground between
‘yes’ and ‘no’ (e.g., qualified their answer or said
it depends)
• Almost all chose ‘yes’ rather than ‘no’
• More in CAPI than CAWI
24. Mode effects with Y/N (CAPI vs CAWI) (2)
3. Of 6 questions, 2 were more sensitive (increasing
income support benefit and redistribution of wealth),
• slightly more ‘yes’ answers in CAPI than in CAWI
Finding (1) –
• Opposite to quant findings, but not surprising - more
satisificing in CAWI on other hypotheses
Findings (2) and (3) –
• In the direction of the quantitative findings
Resultant quantitative effects would depend on
prevalence of respondents in these different
categories
25. Branching versus No Branching (1)
• Needed direct evidence rather than
aggregate evidence
• 12 Rs asked the 2 attitude questions in
branched format as part of the survey
questions
• Much later in cognitive interviews, R given
a showcard with ‘non-branched’ format of
same question
• Interviewer was to probe for any
inconsistencies
26. Branching versus No Branching (2)
• Some Rs were inconsistent
• 2 themes:
• Vagueness of the answer categories
• EXAMPLE: “To me, extremely good and
very good are the same thing”
• Confusion in scope of question
• EXAMPLE: Were ‘local shopping facilities’
for food shopping or clothes shopping?
• Although not explaining the branching effect, per
se, findings suggest how variable respondents’
answers can be.
28. Conclusions
• ‘Y/N for each’ and branching formats are not
necessarily functionally equivalent across all
three modes
• Optimising question format for each mode should
not be done without empirical evidence that this
will produce comparable data across modes