SlideShare a Scribd company logo
1 of 29
Download to read offline
Is it a good idea to optimise question
format for mode of data collection?

Gerry Nicolaas      National Centre for Social Research
Pamela Campanelli   Independent Survey Methods Consultant
Steven Hope         University College London
Annette Jäckle      University of Essex
Peter Lynn          University of Essex
Overview


• Background
• Research question
• ‘Mark all that apply’ vs ‘yes/no for each item’
   • Literature, Hypotheses, Methods, Results

• Branched questions vs non-branching
   • Literature, Hypotheses, Methods, Results

• Cognitive interview results
   • Methods, Results

• Conclusion
Background

• Face-to-face interviewing is dominant mode in UK
   • Interviewer presence
   • Aural stimuli & Visual stimuli (e.g. show cards)

• Many questions designed to be ‘optimal’ for face-to-
  face interviews
   • abundant use of show cards, long response lists, ‘mark all
     that apply’, fully labelled scales, etc

• Push towards cheaper data collection modes and
  mixed modes
   • Some questions used in face-to-face interviews need to be
     adapted for other modes
Research question


• Is it a good idea to ‘optimise’ question format for
  mode of data collection?
   • Increasing risk of differences in measurement across modes?



• 2 question formats that tend to be changed for
  telephone interviews
   • Mark all that apply
   • Scales
‘Mark all that apply’
        versus
‘yes/no for each item’
Example of ‘mark all that apply’

F2F & Self-Completion                   Telephone interview
This card shows a number of             I am now going to ask you a
different ways for reducing             number of questions about
poverty. In your opinion, which of      different methods for reducing
the following would be effective in     poverty. In your opinion, which of
reducing poverty?                       the following would be effective?

 SHOW CARD / MARK ALL                   Would increasing pensions reduce
o Increasing pensions                   poverty?
o Investing in education for children   o Yes
o Improving access to childcare         o No
o Redistribution of wealth
o Increasing trade union rights         Would investing in education for
o Reducing discrimination               children reduce poverty?
o Increasing income support             o Yes
o Investing in job creation             o No
o None of these
                                        etc……….
‘mark all’ vs ‘yes/no for each’

• Difficult to interpret absence of being marked
  with ‘mark all’
   • Does not apply or item non-response?
   • Sudman & Bradburn (1982)

• Deeper processing of each item with ‘yes/no’
   •   More items selected with ‘yes/no’
   •   Respondents take more time to complete ‘yes/no’
   •   Primacy effects with ‘mark all’
   •   Rasinski et al (1994), Smyth et al (2006), Smyth et al
       (2008)

• ‘yes/no’ performs similarly in tel & web modes
   • Smyth et al (2008)
Our research



• Replicate & extend Smyth et al (2006 & 2008)
   • Face-to-face mode as well as telephone and web
   • UK general population with internet access
   • Cognitive interviews as well as experimental data
Hypotheses

• Higher % of items chosen in tel mode with ‘y/n’
  than in F2F & web with ‘mark all’
• If using ‘y/n’ in all 3 modes, no differences
• If using ‘mark all’ in F2F & web, no differences
• Deeper processing with ‘y/n’
   • ‘y/n’ takes longer to complete than ‘mark all’
   • more primacy with ‘mark all’ respondents with below
     average completion times
   • ‘mark all’ respondents with at least average completion
     times select as many items as ‘y/n’ respondents
• Differences greater for difficult questions
Methods


• 2 split ballot experiments
   • Easy attitude question about neighbourhood with 8 items
   • Difficult attitude question about poverty with 8 items

• Mean number of endorsements for each format in
  each mode
• Response latencies for each format in each mode
• OLS regression
• Control variables for differential non-response
2 split ballot experiments

(1) Neighbourhood                   (2) Reducing poverty
                                    This card shows a number of different
What are the things that you like   ways for reducing poverty. In your
about your neighbourhood?           opinion, which of the following would
                                    be effective in reducing poverty?
 SHOW CARD
o Its community spirit              SHOWCARD
o It feels safe                     o Increasing pensions
o The neighbours                    o Investing in education for children
o The character of its buildings    o Improving access to childcare
o Its cleanliness                   o Redistribution of wealth
o Its location                      o Increasing trade union rights
o Its quietness                     o Reducing discrimination
o Its transport facilities          o Increasing income support
                                    o Investing in job creation
                                    o None of these
Results

• Hypotheses supported:
  • higher % of items selected in tel with ‘y/n’ than F2F &
    web with ‘mark all’
  • ‘y/n’ takes longer to complete than ‘mark all’
  • no differences when using ‘mark all’ in F2F & web
• But:
  • when using ‘y/n’ in all modes, differences observed
    between tel (& F2F) and web
  • differences were greater for the difficult question
  • no primacy among ‘mark all’ respondents with less than
    average completion times
  • ‘mark all’ respondents with at least average completion
    times still selected fewer items
Differences with Smyth et al      (2006 & 2008)



• Differences in study population
   • Smyth et al: students at USA university
   • This study: general population who used internet

• Differences in question design
   • Smyth et al: 9-15 items
   • This study: 8 items for both questions
Full response list in 1 step
          versus
  Branching into 2 steps
Example of scalar question

Fully labelled scale                   Branched question

Please tell me whether you consider    Please tell me whether you consider
your local shopping facilities to be   your local shopping facilities to be
…….                                    …….
 o extremely poor,                     o poor or
 o very poor,                          o good?
 o poor,
 o good,                               Would this be …….
 o very good, or                       o (poor/good),
 o extremely good?                     o very (poor/good), or
                                       o extremely (poor/good)?
Full scale vs branching

• Branching makes the task easier for respondents
  • Decomposition principle
  • Armstrong et al (1975), Groves & Kahn (1979), Krosnick &
    Berent (1993), Yu et al (2003), Malhotra et al (2008)


• Branching produces more extreme responses
  • Reduces reluctance to select extreme responses
  • Nicolaas et al (2000), Yu et al (2003)
Hypotheses



• More extreme responses in tel mode with
  branching than in F2F & web with no branching
   • This effect is more prevalent for attitudinal than factual
     questions

• Within each mode, more extreme responses with
  branching
   • This effect is more prevalent for attitudinal than factual
     questions

• If using branching in all 3 modes, no differences
Methods

• 4 split ballot experiments
   •   Easy attitude question about shopping facilities
   •   Difficult attitude question about standard of living
   •   Easy factual question about rent/mortgage
   •   Difficult factual question about grocery shopping

• Proportion of extreme & non-extreme responses
   • Highest versus the lowest category
   • Two highest versus the two lowest categories

• Logistic regression
• Control variables for differential non-response
4 split ballot experiments


(1) Please tell me          (2) Thinking back to the
whether you consider        last general election,
your local shopping         would you say that the
facilities to be ...        standard of living has ...
READ OUT                      SHOW CARD
oextremely poor,              oincreased by a large amount,
o very poor,                  o increased by a medium amount,
o poor,                       o increased by a small amount,
o good,                       o stayed the same,
o very good or                o decreased by a small amount,
o extremely good?             o decreased by a medium amount, or
                              o decreased by a large amount?
4 split ballot experiments (continued)

(3) How much did your         (4) How much did your
household spend last month    household spend last month
in rent or mortgage for the   on grocery shopping?
accommodation you live in?

 READ OUT                     READ OUT
 oless than £100              o less than £100
 o £100 - £199                o £100 - £199
 o £200 - £299                o £200 - £299
 o £300 - £399                o £300 - £399
 o £400 - £499                o £400 - £499
 o or £500 or more?           o or £500 or more?
Results


• Hypotheses supported or partly supported:
  • more extreme responses in tel with branching than F2F
    & web with no branching
  • difference was greater for attitudinal than factual
    questions
  • within each mode, branching produced more extreme
    responses but only for attitudinal questions
• Hypotheses not supported:
  • within each mode no branching seemed to produce
    more extreme responses than branching for factual
    questions
  • many differences observed across modes when using
    branched format in all 3 modes
Cognitive interviews
Mode effects with Y/N (CAPI vs CAWI) (1)


Subtleties going on that could affect aggregate
mode comparisons.
1. Instances of clear and possible satisficing
   • Most of these were in the ‘yes’ category
   • Most of these were in CAWI
2. Instances of Rs in the middle ground between
    ‘yes’ and ‘no’ (e.g., qualified their answer or said
    it depends)
   • Almost all chose ‘yes’ rather than ‘no’
   • More in CAPI than CAWI
Mode effects with Y/N (CAPI vs CAWI) (2)

3. Of 6 questions, 2 were more sensitive (increasing
    income support benefit and redistribution of wealth),
   • slightly more ‘yes’ answers in CAPI than in CAWI
Finding (1) –
• Opposite to quant findings, but not surprising - more
   satisificing in CAWI on other hypotheses
Findings (2) and (3) –
• In the direction of the quantitative findings

Resultant quantitative effects would depend on
   prevalence of respondents in these different
   categories
Branching versus No Branching (1)

• Needed direct evidence rather than
  aggregate evidence
• 12 Rs asked the 2 attitude questions in
  branched format as part of the survey
  questions
• Much later in cognitive interviews, R given
  a showcard with ‘non-branched’ format of
  same question
• Interviewer was to probe for any
  inconsistencies
Branching versus No Branching (2)

• Some Rs were inconsistent
• 2 themes:
   • Vagueness of the answer categories
      • EXAMPLE: “To me, extremely good and
        very good are the same thing”
   • Confusion in scope of question
      • EXAMPLE: Were ‘local shopping facilities’
        for food shopping or clothes shopping?
• Although not explaining the branching effect, per
  se, findings suggest how variable respondents’
  answers can be.
Conclusions
Conclusions


• ‘Y/N for each’ and branching formats are not
  necessarily functionally equivalent across all
  three modes
• Optimising question format for each mode should
  not be done without empirical evidence that this
  will produce comparable data across modes
Thank you
www.natcen.ac.uk

More Related Content

Viewers also liked

Ahip webinar 12 5 (3)
Ahip webinar 12 5 (3)Ahip webinar 12 5 (3)
Ahip webinar 12 5 (3)Julie Doyle
 
Rss Oct 2011 Mixed Modes Pres6
Rss Oct 2011 Mixed Modes Pres6Rss Oct 2011 Mixed Modes Pres6
Rss Oct 2011 Mixed Modes Pres6GerryNicolaas
 
Kaylas beowulf powerpoint
Kaylas beowulf powerpointKaylas beowulf powerpoint
Kaylas beowulf powerpointlovey_kayla94
 
Hiphop evaluacion
Hiphop evaluacionHiphop evaluacion
Hiphop evaluaciondp10306
 

Viewers also liked (6)

Ahip webinar 12 5 (3)
Ahip webinar 12 5 (3)Ahip webinar 12 5 (3)
Ahip webinar 12 5 (3)
 
Rss Oct 2011 Mixed Modes Pres6
Rss Oct 2011 Mixed Modes Pres6Rss Oct 2011 Mixed Modes Pres6
Rss Oct 2011 Mixed Modes Pres6
 
Kaylas beowulf powerpoint
Kaylas beowulf powerpointKaylas beowulf powerpoint
Kaylas beowulf powerpoint
 
Human and Nature in Harmony
Human and Nature in HarmonyHuman and Nature in Harmony
Human and Nature in Harmony
 
Instrumentos de evaluación (tema 4)
Instrumentos de evaluación (tema 4)Instrumentos de evaluación (tema 4)
Instrumentos de evaluación (tema 4)
 
Hiphop evaluacion
Hiphop evaluacionHiphop evaluacion
Hiphop evaluacion
 

Similar to Rss Oct 2011 Mixed Modes Pres5

Collection of data
Collection of dataCollection of data
Collection of dataBaiju KT
 
Rss Oct 2011 Mixed Modes Pres4
Rss Oct 2011 Mixed Modes Pres4Rss Oct 2011 Mixed Modes Pres4
Rss Oct 2011 Mixed Modes Pres4GerryNicolaas
 
How to design effective online surveys
How to design effective online surveysHow to design effective online surveys
How to design effective online surveysUserZoom
 
Open Inclusion Research London a11y Meetup May 2018
Open Inclusion Research London a11y Meetup May 2018Open Inclusion Research London a11y Meetup May 2018
Open Inclusion Research London a11y Meetup May 2018Christine Hemphill
 
RSS 2012 How to Write a Health Survey
RSS 2012 How to Write a Health SurveyRSS 2012 How to Write a Health Survey
RSS 2012 How to Write a Health SurveyWesam Abuznadah
 
The latte levy; Why environmental policy requires theory in design and the pu...
The latte levy; Why environmental policy requires theory in design and the pu...The latte levy; Why environmental policy requires theory in design and the pu...
The latte levy; Why environmental policy requires theory in design and the pu...Peter King
 
SurveyB U S 1 1 5 W I N T E R 2 0 1 7Survey — the.docx
SurveyB U S  1 1 5  W I N T E R  2 0 1 7Survey — the.docxSurveyB U S  1 1 5  W I N T E R  2 0 1 7Survey — the.docx
SurveyB U S 1 1 5 W I N T E R 2 0 1 7Survey — the.docxssuserf9c51d
 
2 lecture 1 course introduction
2   lecture 1 course introduction2   lecture 1 course introduction
2 lecture 1 course introductionJason Edington
 
anchoring-heuristic Decision Making
anchoring-heuristic Decision Makinganchoring-heuristic Decision Making
anchoring-heuristic Decision MakingÖzkan Özer
 
3 survey, questionaire, graphic techniques
3 survey, questionaire, graphic techniques3 survey, questionaire, graphic techniques
3 survey, questionaire, graphic techniquesPenny Jiang
 
Collecting and reccording data si
Collecting and reccording data siCollecting and reccording data si
Collecting and reccording data sishehlaijaz
 
OFHS_Tutorial_STATA.ppt
OFHS_Tutorial_STATA.pptOFHS_Tutorial_STATA.ppt
OFHS_Tutorial_STATA.pptssuser3840bc
 
OFHS_Tutorial_STATA (1).ppt
OFHS_Tutorial_STATA (1).pptOFHS_Tutorial_STATA (1).ppt
OFHS_Tutorial_STATA (1).pptMohammed35151
 

Similar to Rss Oct 2011 Mixed Modes Pres5 (20)

Collection of data
Collection of dataCollection of data
Collection of data
 
The Student Post Exchange Survey
The Student Post Exchange SurveyThe Student Post Exchange Survey
The Student Post Exchange Survey
 
Rss Oct 2011 Mixed Modes Pres4
Rss Oct 2011 Mixed Modes Pres4Rss Oct 2011 Mixed Modes Pres4
Rss Oct 2011 Mixed Modes Pres4
 
Workshop session 6 - Advances in weighting
Workshop session 6 - Advances in weightingWorkshop session 6 - Advances in weighting
Workshop session 6 - Advances in weighting
 
Student post exchange survey
Student post exchange surveyStudent post exchange survey
Student post exchange survey
 
How to design effective online surveys
How to design effective online surveysHow to design effective online surveys
How to design effective online surveys
 
Open Inclusion Research London a11y Meetup May 2018
Open Inclusion Research London a11y Meetup May 2018Open Inclusion Research London a11y Meetup May 2018
Open Inclusion Research London a11y Meetup May 2018
 
RSS 2012 How to Write a Health Survey
RSS 2012 How to Write a Health SurveyRSS 2012 How to Write a Health Survey
RSS 2012 How to Write a Health Survey
 
The latte levy; Why environmental policy requires theory in design and the pu...
The latte levy; Why environmental policy requires theory in design and the pu...The latte levy; Why environmental policy requires theory in design and the pu...
The latte levy; Why environmental policy requires theory in design and the pu...
 
PR 2 Chapter 3.pptx
PR 2 Chapter 3.pptxPR 2 Chapter 3.pptx
PR 2 Chapter 3.pptx
 
SurveyB U S 1 1 5 W I N T E R 2 0 1 7Survey — the.docx
SurveyB U S  1 1 5  W I N T E R  2 0 1 7Survey — the.docxSurveyB U S  1 1 5  W I N T E R  2 0 1 7Survey — the.docx
SurveyB U S 1 1 5 W I N T E R 2 0 1 7Survey — the.docx
 
NH back to school Address 2012
NH back to school Address 2012NH back to school Address 2012
NH back to school Address 2012
 
2 lecture 1 course introduction
2   lecture 1 course introduction2   lecture 1 course introduction
2 lecture 1 course introduction
 
anchoring-heuristic Decision Making
anchoring-heuristic Decision Makinganchoring-heuristic Decision Making
anchoring-heuristic Decision Making
 
3 survey, questionaire, graphic techniques
3 survey, questionaire, graphic techniques3 survey, questionaire, graphic techniques
3 survey, questionaire, graphic techniques
 
Collecting and reccording data si
Collecting and reccording data siCollecting and reccording data si
Collecting and reccording data si
 
surveys non experimental
surveys non experimentalsurveys non experimental
surveys non experimental
 
OFHS_Tutorial_STATA.ppt
OFHS_Tutorial_STATA.pptOFHS_Tutorial_STATA.ppt
OFHS_Tutorial_STATA.ppt
 
OFHS_Tutorial_STATA (1).ppt
OFHS_Tutorial_STATA (1).pptOFHS_Tutorial_STATA (1).ppt
OFHS_Tutorial_STATA (1).ppt
 
Longitudunal studies for policy impact
Longitudunal studies for policy impactLongitudunal studies for policy impact
Longitudunal studies for policy impact
 

Rss Oct 2011 Mixed Modes Pres5

  • 1. Is it a good idea to optimise question format for mode of data collection? Gerry Nicolaas National Centre for Social Research Pamela Campanelli Independent Survey Methods Consultant Steven Hope University College London Annette Jäckle University of Essex Peter Lynn University of Essex
  • 2. Overview • Background • Research question • ‘Mark all that apply’ vs ‘yes/no for each item’ • Literature, Hypotheses, Methods, Results • Branched questions vs non-branching • Literature, Hypotheses, Methods, Results • Cognitive interview results • Methods, Results • Conclusion
  • 3. Background • Face-to-face interviewing is dominant mode in UK • Interviewer presence • Aural stimuli & Visual stimuli (e.g. show cards) • Many questions designed to be ‘optimal’ for face-to- face interviews • abundant use of show cards, long response lists, ‘mark all that apply’, fully labelled scales, etc • Push towards cheaper data collection modes and mixed modes • Some questions used in face-to-face interviews need to be adapted for other modes
  • 4. Research question • Is it a good idea to ‘optimise’ question format for mode of data collection? • Increasing risk of differences in measurement across modes? • 2 question formats that tend to be changed for telephone interviews • Mark all that apply • Scales
  • 5. ‘Mark all that apply’ versus ‘yes/no for each item’
  • 6. Example of ‘mark all that apply’ F2F & Self-Completion Telephone interview This card shows a number of I am now going to ask you a different ways for reducing number of questions about poverty. In your opinion, which of different methods for reducing the following would be effective in poverty. In your opinion, which of reducing poverty? the following would be effective? SHOW CARD / MARK ALL Would increasing pensions reduce o Increasing pensions poverty? o Investing in education for children o Yes o Improving access to childcare o No o Redistribution of wealth o Increasing trade union rights Would investing in education for o Reducing discrimination children reduce poverty? o Increasing income support o Yes o Investing in job creation o No o None of these etc……….
  • 7. ‘mark all’ vs ‘yes/no for each’ • Difficult to interpret absence of being marked with ‘mark all’ • Does not apply or item non-response? • Sudman & Bradburn (1982) • Deeper processing of each item with ‘yes/no’ • More items selected with ‘yes/no’ • Respondents take more time to complete ‘yes/no’ • Primacy effects with ‘mark all’ • Rasinski et al (1994), Smyth et al (2006), Smyth et al (2008) • ‘yes/no’ performs similarly in tel & web modes • Smyth et al (2008)
  • 8. Our research • Replicate & extend Smyth et al (2006 & 2008) • Face-to-face mode as well as telephone and web • UK general population with internet access • Cognitive interviews as well as experimental data
  • 9. Hypotheses • Higher % of items chosen in tel mode with ‘y/n’ than in F2F & web with ‘mark all’ • If using ‘y/n’ in all 3 modes, no differences • If using ‘mark all’ in F2F & web, no differences • Deeper processing with ‘y/n’ • ‘y/n’ takes longer to complete than ‘mark all’ • more primacy with ‘mark all’ respondents with below average completion times • ‘mark all’ respondents with at least average completion times select as many items as ‘y/n’ respondents • Differences greater for difficult questions
  • 10. Methods • 2 split ballot experiments • Easy attitude question about neighbourhood with 8 items • Difficult attitude question about poverty with 8 items • Mean number of endorsements for each format in each mode • Response latencies for each format in each mode • OLS regression • Control variables for differential non-response
  • 11. 2 split ballot experiments (1) Neighbourhood (2) Reducing poverty This card shows a number of different What are the things that you like ways for reducing poverty. In your about your neighbourhood? opinion, which of the following would be effective in reducing poverty? SHOW CARD o Its community spirit SHOWCARD o It feels safe o Increasing pensions o The neighbours o Investing in education for children o The character of its buildings o Improving access to childcare o Its cleanliness o Redistribution of wealth o Its location o Increasing trade union rights o Its quietness o Reducing discrimination o Its transport facilities o Increasing income support o Investing in job creation o None of these
  • 12. Results • Hypotheses supported: • higher % of items selected in tel with ‘y/n’ than F2F & web with ‘mark all’ • ‘y/n’ takes longer to complete than ‘mark all’ • no differences when using ‘mark all’ in F2F & web • But: • when using ‘y/n’ in all modes, differences observed between tel (& F2F) and web • differences were greater for the difficult question • no primacy among ‘mark all’ respondents with less than average completion times • ‘mark all’ respondents with at least average completion times still selected fewer items
  • 13. Differences with Smyth et al (2006 & 2008) • Differences in study population • Smyth et al: students at USA university • This study: general population who used internet • Differences in question design • Smyth et al: 9-15 items • This study: 8 items for both questions
  • 14. Full response list in 1 step versus Branching into 2 steps
  • 15. Example of scalar question Fully labelled scale Branched question Please tell me whether you consider Please tell me whether you consider your local shopping facilities to be your local shopping facilities to be ……. ……. o extremely poor, o poor or o very poor, o good? o poor, o good, Would this be ……. o very good, or o (poor/good), o extremely good? o very (poor/good), or o extremely (poor/good)?
  • 16. Full scale vs branching • Branching makes the task easier for respondents • Decomposition principle • Armstrong et al (1975), Groves & Kahn (1979), Krosnick & Berent (1993), Yu et al (2003), Malhotra et al (2008) • Branching produces more extreme responses • Reduces reluctance to select extreme responses • Nicolaas et al (2000), Yu et al (2003)
  • 17. Hypotheses • More extreme responses in tel mode with branching than in F2F & web with no branching • This effect is more prevalent for attitudinal than factual questions • Within each mode, more extreme responses with branching • This effect is more prevalent for attitudinal than factual questions • If using branching in all 3 modes, no differences
  • 18. Methods • 4 split ballot experiments • Easy attitude question about shopping facilities • Difficult attitude question about standard of living • Easy factual question about rent/mortgage • Difficult factual question about grocery shopping • Proportion of extreme & non-extreme responses • Highest versus the lowest category • Two highest versus the two lowest categories • Logistic regression • Control variables for differential non-response
  • 19. 4 split ballot experiments (1) Please tell me (2) Thinking back to the whether you consider last general election, your local shopping would you say that the facilities to be ... standard of living has ... READ OUT SHOW CARD oextremely poor, oincreased by a large amount, o very poor, o increased by a medium amount, o poor, o increased by a small amount, o good, o stayed the same, o very good or o decreased by a small amount, o extremely good? o decreased by a medium amount, or o decreased by a large amount?
  • 20. 4 split ballot experiments (continued) (3) How much did your (4) How much did your household spend last month household spend last month in rent or mortgage for the on grocery shopping? accommodation you live in? READ OUT READ OUT oless than £100 o less than £100 o £100 - £199 o £100 - £199 o £200 - £299 o £200 - £299 o £300 - £399 o £300 - £399 o £400 - £499 o £400 - £499 o or £500 or more? o or £500 or more?
  • 21. Results • Hypotheses supported or partly supported: • more extreme responses in tel with branching than F2F & web with no branching • difference was greater for attitudinal than factual questions • within each mode, branching produced more extreme responses but only for attitudinal questions • Hypotheses not supported: • within each mode no branching seemed to produce more extreme responses than branching for factual questions • many differences observed across modes when using branched format in all 3 modes
  • 23. Mode effects with Y/N (CAPI vs CAWI) (1) Subtleties going on that could affect aggregate mode comparisons. 1. Instances of clear and possible satisficing • Most of these were in the ‘yes’ category • Most of these were in CAWI 2. Instances of Rs in the middle ground between ‘yes’ and ‘no’ (e.g., qualified their answer or said it depends) • Almost all chose ‘yes’ rather than ‘no’ • More in CAPI than CAWI
  • 24. Mode effects with Y/N (CAPI vs CAWI) (2) 3. Of 6 questions, 2 were more sensitive (increasing income support benefit and redistribution of wealth), • slightly more ‘yes’ answers in CAPI than in CAWI Finding (1) – • Opposite to quant findings, but not surprising - more satisificing in CAWI on other hypotheses Findings (2) and (3) – • In the direction of the quantitative findings Resultant quantitative effects would depend on prevalence of respondents in these different categories
  • 25. Branching versus No Branching (1) • Needed direct evidence rather than aggregate evidence • 12 Rs asked the 2 attitude questions in branched format as part of the survey questions • Much later in cognitive interviews, R given a showcard with ‘non-branched’ format of same question • Interviewer was to probe for any inconsistencies
  • 26. Branching versus No Branching (2) • Some Rs were inconsistent • 2 themes: • Vagueness of the answer categories • EXAMPLE: “To me, extremely good and very good are the same thing” • Confusion in scope of question • EXAMPLE: Were ‘local shopping facilities’ for food shopping or clothes shopping? • Although not explaining the branching effect, per se, findings suggest how variable respondents’ answers can be.
  • 28. Conclusions • ‘Y/N for each’ and branching formats are not necessarily functionally equivalent across all three modes • Optimising question format for each mode should not be done without empirical evidence that this will produce comparable data across modes