What’s really going on inside panels?
And does it matter?
________
NORSTAT SUMMER LOUNGE
June 2014
Dan Kvistbo

SORT OF AN AGENDA
 HOW FAR WE ARE WHERE WE ARE GOING 

THE BEGINNING

BARRIERS ALL BUT GONE
1999-2001
_____
• Toluna
• GMI
• Ciao!
• Lightspeed
• Zapera
• Research Now
• etc.
_____

LOW COST | HIGH SPEED
LightSPEED | FLYresearch | ResearchNOW | SPEEDfacts etc.

FOCUS ON BENEFITS | DATA ASSUMED REP2
RESEARCH BUYERS IN LINE TO MOVE STUDIES ONLINE

CASE IN POINT | P&G
P&G | 1999: 15%  2000: 40%  2001: 50%
”To save money and expedite
market research surveys”...
_____
”[...] because surveys are
completed 75% faster over the
Web-and at half the cost of
conventional methods [...]”

TROUBLED WATERS | 2003‘ish - 2006 ()

CULMINATION | SEP 2006 RESEARCH INDUSTRY SUMMIT
“Two surveys a week apart yielded
different recommendations…
_____
I never thought I was trading data
quality for cost savings”
A WOMAN WITH A BUDGET
Kim Dedeker, Procter
& Gamble's VP-Global
Consumer & Market
Knowledge, 2006
(Now Chairman, the
Americas at Kantar)

WHAT HAD GONE WRONG?

PART (I) OF THE TALE
“...clients, have put many
pressures on the research
suppliers […]”
_____
”We’ve focused on cost and
speed and an unintended
consequence was erosion in
quality. Now, we’re starting to feel
the effects”…
Kim Dedeker

PART (II) OF THE TALE
• Heavy survey taking
• Panel conditioning
• Declining cooperation
• Increase in demand
• Persistent mode differences
• Long and complex questionnaires
• Multi panel membership
• The 0.25% doing 30+%
• ‘Yes we can’
• Etc.
_____
(Source: Reg Baker, Market Strategies International)

MR INDUSTRY RESPONSE?
”It’s like the whole in the ozone
layer. Everyone knows it’s a
growing problem.”
_____
”But they just ignore it and go on to
the next project.”…
(S. Morwood, VP Worldwide Market Research for
IBM. AdAge oct. 2006)

TRUTH BE TOLD
ESOMAR 25
2005 – 2006 – 2007 – 2008 – 2009 – 2010 – 2011 – 2012 – 2013 – 2014 –
The Grand Mean Project
ARF ORQC
”Million dollar” FoQ project
ISO 26362
AAPOR Report on Online Panels
FoQ 2 (Reduce the Risk)
Report on the AAPOR task force on non-probability sampling
ESOMAR/GRBN (draft) Guideline For Online Sample Quality
*(Consultation draft)
<
26 28
 HALFWAY THROUGH! 

BUT IN ESSENCE
“75 years of basing quantitative market research
on a paradigm of projecting the results of
samples onto populations via assumptions of
random probability sampling, has been replaced
by a willingness to use online access panels,
which do not approximate to this model.”
_____
- Ray Poynter
(Handbook of Online and Social Media Research, 2010)

AND DESPITE THE EFFORTS
Lemon market | Conditions:
_____
• Asymmetry of information (buyer vs. seller)
• Incentive for the seller to pass off a low-quality
product as a higher-quality one
• Sellers with a bettter product have no way of
demonstrating this credibly to buyers
• Deficiency of effective quality assurances/guarantees
_____
http://en.wikipedia.org/wiki/The_Market_for_Lemons (Paraphrased)
Ref: The Market for "Lemons": Quality Uncertainty and the Market Mechanism
- George A. Akerlof
The Quarterly Journal of Economics, Vol. 84, No. 3. (Aug., 1970), pp. 488-500

BLACK BOX
White-box testing
_____
a method of testing [software] that tests
internal structures or workings [of an
application], as opposed to [its] functionality.
(http://en.wikipedia.org/wiki/White-box_testing)

SO... WHAT IS GOING ON INSIDE?
 CLOSER...

DIRTY LITTLE SECRETS OF ONLINE PANELS

ISSUE #1 CONDUCT
_____
N = 1000, whatever it takes

ISSUE #2 RECRUITMENT

ISSUE #3 RESPONSE RATES
Human Interaction?
Trust that the survey is legitimate?
Time involvement with each respondent?
Attention given to each respondent?
Respondent control over access?
Respondent control over whether to respond?







CONSEQUENCE
The concept of TSE
gets a whole new
meaning…
• “Total” sampling error
• “Total” coverage error
• “Total” non-response error
• Etc.

IS IT EITHER OR?
A Data
Q sample
A convenience
sample of a
convenience
sample
Random
probability
sample
Convenience
sample
A quota sample
of a random
sample
Yes, you are
”...trading data
quality for cost
savings...”

TO CONCLUDE
“If you pay peanuts for your sample, chances are
you're getting monkeys”
(Gordon Morris Global Insights Manager at Sony Ericsson)
”I don’t want it done scientifically, I just want it
done right!”
(Client, Ron Sellers)
“Sometimes close counts in more than
horseshoes”
(Reg Baker)

LOOKING FORWARD
A few words on the future of online research
1. It’s (increasingly) mobile
2. It moves beyond self-reported attitudes,
emotions and behaviours to merge survey data and
behavioral data
3. Success depends as much on buyers
acknowledging their responsibility (ask better
questions and adapt to the realm of mobile!) as on
sample providers to deliver the appropriate sample
& platform

THANK YOU
Or google
...there’s only one of me
_____
DAN KVISTBO
dan.kvistbo@norstat.dk
+45 27772016
@Kvistbo

Questions?