Paper presented at the Methodology of Longitudinal Surveys International Conference, 12-14 July 2006, University of Essex, Colchester, UK

1. Seriously mixed methods
Do they risk non-response and attrition?
Ben Anderson
Chimera, University of Essex

2. The Menu
• Why bother?
• The background
• The panel and its methods
• Who dropped out (and why)
• What can we learn?
www.essex.ac.uk/chimera

3. Multiple methods - why bother?
• Data Triangulation:
• Different data on the same individuals
• Different instruments and methods (qual, quant,
administrative)
• Cross-confirmation and validation
• Respondents lie, they forget and they don’t
care
• Multiple methods can unravel some of this
• Different views - different insights
• Patterns (what?) and explanations (why?)
www.essex.ac.uk/chimera

4. Other reasons
• Interaction of research modes (and
researchers!)
• Leads to insights & innovation
• Multiple methods = 'real life' methods
• Increasingly valued in policy & evaluation
research
• ‘rounded view’
www.essex.ac.uk/chimera

5. But
• Such methods may
• Increase respondent burden
• Increase fears of privacy and surveillance
• Or conversely
• Develop stronger relationships between
researchers and respondents
• Increase respondent ‘attachment’
www.essex.ac.uk/chimera

6. An example: BT’s Digital Living project
Quantitative
Phone call records
PC/Internet usage logs
Surveys
Time-use diaries
Interviews
Shadowing & Observation
Digital Ethnography Rich contextual
Qualitative picture
www.essex.ac.uk/chimera

7. GB Longitudinal Panel
Dec 1998 Dec 1999 Dec 2000
• GB surveys (2500 individuals in 999 hh)
• Call record capture (635 of 999 hh)
• Internet logs (16 of 999 hh)
• ‘Long conversations’ (37 of 999 hh)
Wave 1 Wave 2 Wave 3
• Qualified random sample (clustered)
• Wave 1 interviews = CAPI
• Wave 2 & 3 = CATI
www.essex.ac.uk/chimera

8. Wave 1 process
• Conduct face to face survey (HoL)
up to 6 months
• Leave time-use diary
• Obtain permission to collect call records
• Obtain permission to re-contact for next survey and
ethnography
• Implement call record capture
• Decide ethnographic sample frame (ICT rich/poor;
income rich/poor)
• Select households from eligible pool (requires survey data)
• Approach households for interview (via survey agency)
• Interview and arrange re-interviews/shadowing etc
• Decide logging sample frame (anyone with Win95!)
• Select households from eligible pool (requires survey data)
• Approach households
• Send disk (self-installer)
www.essex.ac.uk/chimera

9. Added complexities...
• Wave 1 bias
• 100% of households to have a telephone
• 50% to have a personal computer
• Boost sample at wave 2
• Original address file, random selection
• To maintain sample size
• CATI
• Overall a rare if not unique beast!
www.essex.ac.uk/chimera

10. Wave 2 & 3 process
• Attempt re-contact
up to 3 months
• Conduct CATI survey
• Post out time-use diary
• Check permission to collect call records
• Obtain permission to re-contact for next survey and follow-
up interviews
• Boost sample (wave 2 only)
• recruit & interview as Wave 1
www.essex.ac.uk/chimera

11. Response rates (individuals)
• Cross- sectional Undefined
Survey plus diary
Wave 1
1093 42%
Wave 2
6
649 25%
Wave 3
10
723 30%
• (unweighted) Survey only
Non-response
668 26%
273 10%
918 36%
391 15%
840 35%
321 13%
Children's diary 163 82 73
No children's diary 125 220 208
Child under 9 286 289 231
Total sample size 2608 2555 2406
Interviews Diaries
• Longitudinal (Always a child)
Never
697
462 13%
697
1415 39%
Wave 1 only 511 14% 480 13%
• (unweighted) Wave 2 only 136 4% 106 3%
Wave 3 only 197 5% 214 6%
Waves 1 and 2 224 6% 172 5%
Waves 2 and 3 365 10% 68 2%
Waves 1 and 3 159 4% 138 4%
Waves 1, 2 and 3 842 23% 303 8%
www.essex.ac.uk/chimera

12. What do we want to know?
• Did the three experimental ‘treatments’
cause non-response?
• To keep it simple:
• Consider w1 to w2 and w1 to w3 effects only
• Ignore boost sample
• Focus on
– refusal and non-contact in responding households
(excludes movers)
– Non-contact (non-responding households)
– Attrition
www.essex.ac.uk/chimera

13. Pathways
W1 W2 W3
79% Interviewed in all waves
48%
W1 interviewees
61%
12%
12% ‘Non-response’ w3
‘Non-response’ w2
35% 25%
72% Attrition after w1
www.essex.ac.uk/chimera

14. Wave 1 to wave 2 effects
• Comparison of response rates
No call Yes, call Difference Difference
records records (call Difference (instrumen
Wave 2 outcome % % Total % records) (qual) tation)
Interview 39.43 48.01 43.83 8.58 -0.44 10.43
Refusal 14.06 15.36 14.72 1.3 -9.10 -10.97
No contact in a responding
hh 2.83 1.35 2.07 -1.48 1.65 -2.26
No hh response 23.96 15.36 19.56 -8.6 -11.47 -7.41
Other 19.72 19.93 19.82 0.21 19.36 10.21
N 1273 1335 52 51
Chi sq 43.26*** 16.96** 10.71*
• % of w1 interviewees
www.essex.ac.uk/chimera

15. Wave 1 to wave 3 effects
• Comparison of response rates
Difference Difference
No call Yes, call (call Difference (instrument
Wave 3 outcome records % records % Total % records) (qual) ation)
Interview 77.89 78.59 78.28 0.70 -6.42 -4.83
Refusal 5.58 6.09 5.87 0.51 -1.79 -2.53
No contact in a responding
hh 2.99 1.88 2.36 -1.11 1.71 4.49
No hh response 11.75 10.78 11.21 -0.97 8.84 1.70
Other 1.79 2.66 2.28 0.87 -2.33 1.17
Total 502 640 24 29
Chi sq 2.778 2.61 2.94
• % of w1 and w2 interviewees
www.essex.ac.uk/chimera

16. Comparison
• Wave 1-2
• Wave 1-3
www.essex.ac.uk/chimera

17. Multivariate analysis
• Logistic approach
– P(x) at t = control variables/known effects +
treatments
– Where X is
• Refusal at t (responding hh)
• Non-contact at t (responding hh)
• Non-contact at t (non-responding hh)
• Attrition
www.essex.ac.uk/chimera

18. Known effects
• Based on Lynn et al (2005)
Refuse Non-contact In HoL?
Age Elderly Elderly & Young Y
Income Lower Higher and/or employed Y
Gender Men Y
Education Less Y
Composition singles singles Y
Culture Ethnic minorities Y
Mobility High mobility High mobility N
Location Urban Urban N
• In addition:
– Technophobia (‘resonance’)
– MRS code (AB, C1, C2, D,E) as proxy for wealth
www.essex.ac.uk/chimera

19. W2 results
Variable w2 refusal w2 non-contact w2 non-contact
(ind) (hh)
Age -0.017* -0.062** -0.043***
MRS Code 0.257** 0.158 0.225**
Gender (female) -0.582** -1.428* -0.258*
Qualification 0.095 -0.208 -0.048
level
Single person -0.821 0.399
Ethnic minority -0.505 0.125
Technophobia 0.028 0.164 0.049
Call records 0.079 -0.735 -0.351
Qualitative -0.875 1.831 -0.793
Internet logging -0.678 -1.163
Constant -1.961** -1.527 -0.565
Pseudo r sq 0.044 0.128 0.08
Chi sq 38.56854 20.49012 70.00324
N 1172 881 1243
• logit, cluster (household identifier) [stata], values = b
www.essex.ac.uk/chimera

20. W3 results
w3 w3 non-contact w3 non-contact
refusal (ind) (hh)
Age -0.02 -0.034 -0.026**
MRS Code -0.06 -0.187 -0.181
Gender -0.431 0.215 -0.149
(female)
Qualification 0.052 0.14 -0.016
level
Single person -0.175 0.206
Ethnic minority 0.429 1.810** 0.459
Technophobia 0.032 -0.162 -0.039
Call records 0.445 -0.39 -0.001
Qualitative -0.125 0.782 0.936
Internet logging -0.243 1.234 0.326
Constant -1.737* -0.778 0.298
Pseudo r sq 0.022 0.106 0.035
Chi sq 12.18588 34.6003 19.14218
N 880 747 932
• logit, cluster (household identifier) [stata], values = b
www.essex.ac.uk/chimera

21. Comparison
• Statistically significant results only
• logit, cluster (household identifier) [stata]
www.essex.ac.uk/chimera

22. Attrition
Variable b
Age -0.052***
MRS Code 0.175*
Gender (female) -0.446***
Qualification level 0.004
Single person 0.005
Ethnic minority -0.122
Call records -0.313
Qualitative 0.055
Internet logging -1.566
Technophobia 0.064*
Region (North)
yorkshire & humberside 1.310**
east midlands 0.357
east anglia 0.866
south east (excl. london) 1.089*
south west 0.893 • Added region
west midlands 0.516
north west 0.713
wales 0.802
scotland 0.656
• Call records variable ‘nearly’
greater london 1.322** significant (p = 0.066)
Constant -0.403
Pseudo r sq 0.12
Chi sq 110.2714
N 1190
• logit, cluster (household identifier) [stata]
www.essex.ac.uk/chimera

23. Conclusions I
• Multi-method projects give you ‘better’ data
• And ‘better’ results (see elsewhere)
• But
• They are resource hungry (researcher and respondent
time/load)
• They are complex to manage and analyse
• You have to be multi-disciplinary/multi-skilled
• All the usual qual/quant bickering takes place
• All of which are good reasons to do them
www.essex.ac.uk/chimera

24. Conclusions II
• Disappointingly:
• Qualitative interviews did not help prevent non-response
or attrition
• BUT encouragingly
• None of the ‘treatments’ were associated with non-
response or attrition
• So overall we should do this more often!
www.essex.ac.uk/chimera

25. Get the data
• All 3 waves of the survey
• UK Data Archive SN = 4607
• Free to UK Data Archive subscribers for non-commercial
research
• Held at Chimera (may be in UKDA
eventually):
• Qualitative transcripts
• Call records (disclosure issues)
• Internet usage logs
www.essex.ac.uk/chimera

26. Thank you
• benander@essex.ac.uk
www.essex.ac.uk/chimera