The document discusses a study that used a "wisdom of crowds" approach to predict the results of the 2009 German federal election, with over 2,000 participants providing their predictions of the percentage of votes each party would receive; it finds the crowd was reasonably accurate overall but overestimated some parties like SPD and underestimated others like FDP; it also analyzes how variables like gender, age, education level, and political interest affected prediction accuracy.

1. Is the Crowd wise enough to predict the Election
Results for the German “Bundestag”?
Thomas Donath, NORDLIGHT research GmbH
Christoph Irmer, ODC Services GmbH

2. Agenda
01 Check In
02 Research Question
03 Methods & Data
04 Results
05 Summary / Outlook
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 2

3. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 3

4. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 4

5. What does it take to produce smarter
outcomes with a crowds “collective
intelligence”?
01 Diversity of opinion
02 Independence of opinion
03 Decentralization
04 Aggregation
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 5

6. The Research Question
Could a „Wisdom of Crowds“ approach to
election prognosis potentially compete
with or even outperform
traditional means of prognosis
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 6

7. Sample
2,323 valid respondents (X,XXX invitations -> XX% valid response rate)
 18 years and older
 widely spread demographics
 no other screening or weighting
 recruited through the ODC Services GmbH
online access panel “opinion-people”
Data filtering: respondents were only excluded if
 sum of % >100.0
 sum of % <100.0
 % for a single party = 100.0
Incentive
 for the smallest summed absolute deviation in %
 1. price: 300 €, 2. and 3. price 100 € each
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 7

8. Procedure for Data Collection
 Instruction (translated from German)
 1. We welcome each individual
participation. We only ask you to
submit a serious estimate.
 2. Please submit your personal
prognosis. To do so, please think
about how the percents will be
distributed across the parties.
 3. Please submit a prognosis as
precise as possible and make use of visual Feedback:
the position after the decimal point. sum is displayed green if equal
to 100.0
 How many percent of votes will the
following parties receive in the
Bundestagswahl 2009 on next
Sunday? Please do not enter the result
you wish for, but your estimate what
the official final result will be.
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 8

9. Wisdom of Crowd Result
CDU / CSU
 Distribution of estimate
CDU / CSU
400
 survey estimate, mean = 35.0%
reading help: approx. 300  election result: 33.8%
individuals estimated the CDU
/ CDU to obtain approx. 34%   +1.2
(34.0% - 34.9%)
300  CDU / CSU overestimated
 standard deviation = 5.5
 95% confidence interval: 34.7 –
200 35.2
 kurtosis = 6.4  leptokurtic,
„supergaußförmig“
100  skewness = -0.1  very low
skewness, symmetrical distribution
 n=2,323
10% 20% 30% 40% 50% 60% 70% 80% 90%
CDU / CSU
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 9

10. Wisdom of Crowd Result
SPD
 Distribution of estimate
SPD
400
 survey estimate, mean = 27.4%
 election result: 23.0%
  +4.4
300  SPD strongly overestimated
 standard deviation = 5.3
 95% confidence interval: 27.2 –
200 27.6
 kurtosis = 2.0  slightly
leptokurtic, „supergaußförmig“
100  skewness = 0.3  low skewness,
fairly symmetrical distribution
 n=2,323
10% 20% 30% 40% 50% 60% 70% 80% 90%
SPD
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 10

11. Wisdom of Crowd Result
FDP
 Distribution of estimate
FDP
400
 survey estimate, mean = 12.3%
 election result: 14.6%
  -2.3
300  FDP strongly underestimated
 standard deviation = 3.8
 95% confidence interval: 12.1 –
200 12.5
 kurtosis = 8.0  leptokurtic,
„supergaußförmig“
100  skewness = 1.3  slightly right-
skewed
 n=2,323
10% 20% 30% 40% 50% 60% 70% 80% 90%
FDP
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 11

12. Wisdom of Crowd Result
Die Linke
 Distribution of estimate
Die Linke
400
 survey estimate, mean = 10.5%
 election result: 11.9%
  -1.4
300  Die Linke underestimated
 standard deviation = 4.9
 95% confidence interval: 10.3 –
200 10.7
 kurtosis = 9.0  leptokurtic,
„supergaußförmig“
100  skewness = 1.9  slightly right-
skewed
 n=2,323
10% 20% 30% 40% 50% 60% 70% 80% 90%
Die Linke
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 12

13. Wisdom of Crowd Result
B90 – Die Grünen
 Distribution of estimate
Bündnis 90 - Die Grünen
400
 survey estimate, mean = 10.0%
 election result: 10.7%
  -0.7
300  B90 / Die Grünen
underestimated
 standard deviation = 3.3
200  95% confidence interval: 9.9 – 10.2
 kurtosis = 2.9  leptokurtic,
„supergaußförmig“
100  skewness = 0.3  very low
skewness, fairly symmetrical
distribution
 n=2,323
10% 20% 30% 40% 50% 60% 70% 80% 90%
Bündnis 90 - Die Grünen
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 13

14. Wisdom of Crowd Result
Andere
 Distribution of estimate
Andere (other parties)
400
 survey estimate, mean = 4.8%
 election result: 6.0%
  -1.2
300  “Andere” underestimated
 standard deviation = 3.9
 95% confidence interval: 4.6 – 4.9
200  kurtosis = 33.6  strongly
leptokurtic, „supergaußförmig“
 skewness = 4.1  right-skewed
100  n=2,323
10% 20% 30% 40% 50% 60% 70% 80% 90%
Andere (other parties)
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 14

15. Comparison with Polls
35,0
36,0
33,0
CDU / CSU 35,0
35,0
35,0
33,8
27,4
25,0
25,0
SPD 24,0
25,0 larger deviation from
26,0
23,0 final result and polls
12,3
13,0
14,0 larger deviation from
FDP 13,5
13,0
14,0 final result and polls
14,6
10,5
sum of absolute
11,0
12,0
%-points difference
Die Linke 11,5
12,0
from final result
11,0
11,9 18.09.09 - 23.09.09 Wisdom of Crowds 11.1%-points
10,0 20.09.09 Forschungsgruppe Wahlen (FGW) 8.4%-points
10,0
Bündnis 90 – 10,0
11,0 25.09.09 Forsa 4.2%-points
Die Grünen 11,0
10,0
10,7 22.09.09 IfD Allensbach 5.0%-points
4,8
5,0 18.09.09 Emnid 7.2%-points
6,0
Andere 4,0
5,0
17.09.09 Infratest dimap 8.4%-points
4,0
6,0 27.09.09 Final Result for Election
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 15

16. Moderator Variables I
sum %-point
deviation (group estimate n
vs. final result)
sex Mann-Whitney-U-Test
male 9,7% (1237) Z -11,9
female 12,7% (1073) p 0,000
age Kruskal-Wallis-Test
18 - 29 years 9,4% (441) Chi-Quadrat 72,0
30 - 39 years 11,6% (418) p 0,000
40 - 49 years 12,3% (486)
50 - 59 years 11,1% (412)
60 - 69 years 11,1% (371)
70 years and older 11,6% (142)
graduation Kruskal-Wallis-Test
Hauptschulabschluss (Volksschulabschluss) 12,2% (499) Chi-Quadrat 91,5
Realschule (Mittlere Reife) 11,6% (676) p 0,000
Abschluss der Polytechnischen Oberschule 10,8% (127)
Fachhochschulreife 10,5% (228)
Abitur (allgemeine oder fachgebundene Hochschulreife) 10,3% (686)
How often do you inform yourself about the political
Kruskal-Wallis-Test
occurances?
to be honest, not at all 14,2% (171) Chi-Quadrat 235,1
less than once per month 12,1% (189) p 0,000
once per month 16,0% (162)
once per week 11,7% (465)
several times per week 9,6% (1212)
 Virtually all socio- and psychographic moderator variables influence the precision of Wisdom
of Crowds estimates.
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 16

17. Moderator Variables II
sum %-point
deviation (group estimate n
vs. final result)
How interested are you in politics? Kruskal-Wallis-Test
to be honest, I am not interested in politics at all 14,4% (202) Chi-Quadrat 247,7
I am interested a tiny bit in politics. 12,9% (405) p 0,000
I am interested somewhat in politics. 11,6% (915)
I am strongly interested in politics. 9,5% (595)
I am very strongly interested in politics. 8,8% (191)
Which statement fits you personally the most? Kruskal-Wallis-Test
I have already voted via letter vote. 10,3% (381) Chi-Quadrat 75,1
I am going to vote and know exactly which party I will vote
10,8% (1167) p 0,000
for with my Zweitstimme (Landesliste).
I am going to vote, but I do not yet know exactly which
11,2% (405)
party I will vote for with my Zweitstimme (Landesliste).
I do not know yet whether I will vote, but I would know
exactly which party I would vote for with my Zweitstimme 13,2% (94)
(Landesliste).
I do not know yet whether I will vote and also I would not
know exactly which party I would vote for with my 12,3% (115)
Zweitstimme (Landesliste).
I am definitely not going to vote. 13,0% (105)
Have you seen prognoses for the results of the
Kruskal-Wallis-Test
Bundestagswahl 2009 within the last weeks?
yes, and I can remember (roughly) the results 10,1% (597) Chi-Quadrat 221,9
yes, but I cannot remember exactly the results 11,0% (961) p 0,000
no, I have not seen any prognoses 12,8% (700)
other sign. moderator variable: state (Bundesland), household size, marital status, employment status, type of work, income
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 17

18. Correlation with own Choice
personal estimate for final result
Korrelati onen
Bündnis 90 -
CDU / CSU SPD FDP Die Linke Die Grünen Andere
Kendall-Tau-b Sonntagsf rage: CDU / Korrelationskoef f izient ,221** -,072** ,094** -,087** -,058** -,051**
CSU Sig. (2-seitig) ,000 ,000 ,000 ,000 ,001 ,003
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: SPD Korrelationskoef f izient -,066** ,166** -,107** -,023 ,043* -,035*
own choice for vote
Sig. (2-seitig) ,000 ,000 ,000 ,181 ,013 ,044
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: FDP Korrelationskoef f izient ,018 -,081** ,164** -,028 ,008 -,020
Sig. (2-seitig) ,297 ,000 ,000 ,099 ,645 ,251
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: DIE Korrelationskoef f izient -,125** -,075** -,067** ,236** -,012 ,003
LINKE Sig. (2-seitig) ,000 ,000 ,000 ,000 ,488 ,867
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: BÜNDNIS Korrelationskoef f izient -,048** ,036* -,044** -,027 ,145** -,029
90/DIE GRÜNEN Sig. (2-seitig) ,005 ,034 ,010 ,107 ,000 ,092
N 2323 2323 2323 2323 2323 2323
Sonntagsf rage: Andere Korrelationskoef f izient -,085** -,019 -,033 ,027 -,036* ,127**
Sig. (2-seitig) ,000 ,270 ,053 ,120 ,037 ,000
N 2323 2323 2323 2323 2323 2323
**. Die Korrelation ist auf dem 0,01 Niv eau signif ikant (zweiseitig).
*. Die Korrelation ist auf dem 0,05 Niv eau signif ikant (zweiseitig).
 There is an opinion bias: Respondents tend to estimate the final result higher for those parties they
would also vote for.
 The effect is on average 2,7%-points.
 I.e. people estimate the final result for parties they would vote for 2,7%-points higher compared
to people who would vote for other parties.
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 18

19. Wisdom of Crowds vs. Unweighted, non-
represantative Poll
28,1
CDU / CSU 35,0
33,8
22,1
SPD 27,4
23,0
13,5
FDP 12,3
14,6
14,6
DIE LINKE 10,5
11,9 sum of absolute
BÜNDNIS 90/DIE 10,0 %-points deviation
10,0 from final result
GRÜNEN 10,7
11,7
Sonntagsfrage (Votes for the Panel) 16.8%-points
Andere 4,8 Wisdom of Crowds (Estimate of the Panel) 11.1%-points
6,0 Final Result for Election
 The Wisdom of Crowds method yields a better estimate for the outcome of the election than a non-
weighted, non-representative poll (Sonntagsfrage) from the same panel would have yielded.
 The “subjective” opinion bias – overestimating the preferred parties outcome – is clearly weaker
than the “objective” reasoning (see esp. CDU/CSU, SPD, Andere).
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 19

20. Effect of seen Prognoses on Precision
 People with (self-claimed) awareness or knowledge of recent prognoses yield better estimates.
 However, it can be shown that Wisdom of Crowds also works in absence of such knowledge.
 Political interest in itself – even without such knowledge – increases the precision.
 Result: People who claim to have seen recent prognoses submit better estimates.
 Hypothesis: Without representative polls in media, even people interested in politics would be unable to estimate the outcome precisely.
 However, there is a confoundation of seeing prognoses with general interest, the latter also being a strong moderator variable for estimate precision. (Kendall Tau-
b=0.390, p=0.000, n=2,244, fairly linear function for the used scales)
 Partial Correlation (functions of inter-correlation for used scales are all fairly linear)
 having seen prognoses with estimate precision (controlled for interest): r=0.172, p=0.000, n=2,241, r2=2.9%
 interest with estimate precision (controlled for having seen prognoses): r=0.220, p=0.000, n=2,241, r 2=4.9%
 Group 1: no prognoses seen, slight to high interest in politics difference = 11.3 %-points
 Group 2: no prognoses seen, low interest difference = 14.7 %-points
 (prognoses seen and roughly remembered difference = 10.1 %-points)
 n=389; n=307
 Mann-Whitney-U-Test: Z=-6.2, p=0.000
 Hypothesis can be rejected, based on self-report of participants: even without seeing recent prognoses, participants who are interested in politics provide good
estimates.
 Of course there is no reasonable way to truly eliminate seeing or benefiting from past prognoses seen by participants. However, this is not a requirement of the
Wisdom of Crowds approach. It actually requires to some extend informed participants.
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 20

21. Summary of Results
 Wisdom of Crowds outperforms unweighted, non-representative poll result of the same
sample.
 In this single study however, Wisdom of Crowds is inferior compared to representative
polls.
 The self-reported knowledge of recent prognoses drastically increases the quality of the
prediction. But also people interested in politics without that knowledge yield fairly
accurate predictions drastically outperforming participants without interest.
 Generally prediction quality is influenced massively by moderator variables.
 This fact is somewhat contrary to the claim by Surowiecki that the individual knowledge of
people hardly matters for the precision.
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 21

22. Outlook
 Wisdom of Crowds might not serve to compete with existing polling methods (yet!)
 but…
 imagine it would not have been election polls but e.g. Product-Tests, Ad-Tests, … :
35
CDU/CSU
Product C 33,8
27,4
SPD
Product D 23
FDP
Product A 12,3
14,6 Wisdom Of Crowds
Linke Final Result
Product F 10,5
11,9
B90 / Grüne
Product E 10
10,7
Andere
Product B 4,8
6
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 22

23. THANK YOU VERY MUCH!
Thomas Donath Christoph Irmer
NORDLIGHT research GmbH ODC Services GmbH
t.donath@nordlight-research.com ci@odc-services.com
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 23