Web panel surveys

1. Quality of Internet Surveys: what can we
learn from election polling?
Patrick Sturgis
Web Panel Surveys, Methods and Experiences
Statistics Sweden, 2 November 2016

2. What I’ll talk about
• The 2015 polling disaster
• History of polling (in)accuracy (GB)
• The 2015 Inquiry
– Hypotheses
– Evidence
– Conclusions
• Herding (if time)
• EU referendum polls
• How random probability surveys fared
2

3. 3

4. The final polls
4
Pollster Mode Fieldwork n Con Lab Lib UKIP Green Other
Populus O 5–6 May 3,917 34 34 9 13 5 6
Ipsos-MORI P 5–6 May 1,186 36 35 8 11 5 5
YouGov
O
4–6 May
10,30
7
34 34 10 12 4 6
ComRes P 5–6 May 1,007 35 34 9 12 4 6
Survation O 4–6 May 4,088 31 31 10 16 5 7
ICM P 3–6 May 2,023 34 35 9 11 4 7
Panelbase O 1–6 May 3,019 31 33 8 16 5 7
Opinium O 4–5 May 2,960 35 34 8 12 6 5
TNS UK O 30/4–4/5 1,185 33 32 8 14 6 6
Ashcroft* P 5–6 May 3,028 33 33 10 11 6 8
BMG* O 3–5 May 1,009 34 34 10 12 4 6
SurveyMonkey*
O
30/4-6/5
18,13
1
34 28 7 13 8 9
Result 37.8 31.2 8.1 12.9 3.8 6.3

5. Election Result v Average of final Polls (GB)
34
33
13
8
5
7
37,8
31,2
12,9
8,1
3,8
6,3
0
5
10
15
20
25
30
35
40
Tory Labour UKIP Lib Dem Green Other
V
o
t
e
S
h
a
r
e
%
Party

6. “Had the forecasts been different, then the
nightly news bulletins would surely have
concentrated rather more on the vast
spending cuts to come, and rather less on the
potential role of Scottish nationalists in a hung
parliament. That might have influenced the
result.”
Editorial 14/5/15

8. 2-month moving average poll vote intention estimates 2010-2015
8

9. Some history
9

10. Error on Conservative vote share
10

11. Error on Labour vote share
11

12. Error on Con/Lab lead
12

13. Frequency of GB Polls 1940-2015
0
50
100
150
1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015
13
N of election polls
1945-2010 = 3,500
N of election polls
2010-2015 = 1,942

14. Methodology of the polls
• All polls were done using phone or online methods
1. Non-random recruitment + match sample to population totals
(e.g. age, sex, region, social grade, past vote)
2. Allocation of Don’t Knows/refusals
3. Apply turnout weight
14

15. Quota sampling methodology
For quota sampling to ‘work’, following conditions should by met:
1. Within levels of the quota/weighting variables, sample should
have the same vote intention as same group in population
2. Assigned probabilities of turnout should be accurate
3. Respondents’ stated vote intentions should agree with how
they actually voted
4. Treatment of DK/refusals should be accurate

16. Evidence Considered
• Three polls from each BPC pollster
– 1st poll of short campaign
– Penultimate poll
– Final poll
• Plus re-contact surveys, if undertaken
• Contemporaneous probability surveys
– British Election Study
– British Social Attitudes survey
• All published estimates replicated using micro-data
16

17. Published estimates: Con-Lab difference
17

18. Whodunnit?

19. The suspects
Late swing

20. The suspects
Late swing Sampling/weighting

21. The suspects
Late swing Sampling/weighting Turnout misreporting

22. The suspects
Late swing Sampling/weighting Turnout misreporting
Don’t Know/refusal

23. The suspects
Don’t Know/refusal
Late swing Sampling/weighting Turnout misreporting
Question wording

24. The suspects
Don’t Know/refusal
Late swing Sampling/weighting Turnout misreporting
Question wording Registration/postal

25. Final polls: Con-Lab after different weightings
25

26. Unlikely to have had an effect
• Postal voting
• Voter registration
• Overseas voters
• Question wording/framing
• Turnout weighting
• Deliberate misreporting
26

27. Late swing
• Main evidence from post-election re-contact polls, where
respondents of pre-election polls interviewed after
election
• Compare reported vote pre and post election
• No strong evidence of late swing to Conservatives

28. Turnout weighting
Different types of evidence on the effects of turnout weighting:
• Pre-election vote intention using known voters in re-contact polls
• Modelling validated turnout by party*ltv
• Sensitivity of estimates to specifications of turnout probabilities
None of these makes a notable difference
28

29. Reported vote before and after election (known voters)

30. Difference in Con lead phone-online2011 2012 2013 2014 2015 2016
−10
Date
Online
Telephone
2011 2012 2013 2014 2015 2016
−4−2024
Mode of Data Collection Difference
Date
Phone−Online(Con−LabMargin)
●
Final
Polls

31. Unrepresentative samples
31

32. Final polls vs. Post-election surveys
32

33. Reported vote before and after election (known voters)
33

34. Conservative lead by age group, polls v BES/BSA
34

35. Conservative lead by GB region, polls v election result
35

36. Age among those aged 65+ (three polls)
36

37. self-reported 2010 turnout by age band (polls v BES)
37

38. Early-call respondents in BES and BSA

39. EU Referendum polls
44

40. The final EU Ref polls
Fieldwork Sample Remain Leave MAE
ORB 14–19 June 877 54 46 5.9
Survation 20 June 1003 51 49 2.9
ComRes 17-22 June 1032 54 46 5.9
Opinium 20-22 June 3011 49 51 0.9
YouGov 20-23 June 3766 51 49 2.9
Ipsos MORI 21-22 June 1592 52 48 3.9
Populus 21-22 June 4740 55 45 6.9
TNS* 16-22 June 2320 48.8 51.2 0.7
Result 48 52
Average MAE 3.8
*TNS did not pre-announce as their final poll, so were not included in the British Polling Council list of final
polls (where the average error is 4.3).

41. -15
-10
-5
0
5
10
15
14-April 28-April 12-May 26-May 9-June 23-June
Online Telephone
Online vs. phone

42. Random probability estimates
Survey Mode/fieldwork RR Remain % Leave %
ICM Face-to-Face 1-19 June 60% 54% 46%
NatCen Online/phone 16 May -12 June 19% 53% 47%
??? Face-to-Face 27 May-19 June ~60% 55% 45%
47