Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
What’s really going on inside panels?
And does it matter?
________
NORSTAT SUMMER LOUNGE
June 2014
Dan Kvistbo
SORT OF AN AGENDA
 HOW FAR WE ARE WHERE WE ARE GOING 
THE BEGINNING
BARRIERS ALL BUT GONE
1999-2001
_____
• Toluna
• GMI
• Ciao!
• Lightspeed
• Zapera
• Research Now
• etc.
_____
LOW COST | HIGH SPEED
LightSPEED | FLYresearch | ResearchNOW | SPEEDfacts etc.
FOCUS ON BENEFITS | DATA ASSUMED REP2
RESEARCH BUYERS IN LINE TO MOVE STUDIES ONLINE
CASE IN POINT | P&G
P&G | 1999: 15%  2000: 40%  2001: 50%
”To save money and expedite
market research surveys”...
_____
...
TROUBLED WATERS | 2003‘ish - 2006 ()
CULMINATION | SEP 2006 RESEARCH INDUSTRY SUMMIT
“Two surveys a week apart yielded
different recommendations…
_____
I never...
WHAT HAD GONE WRONG?
PART (I) OF THE TALE
“...clients, have put many
pressures on the research
suppliers […]”
_____
”We’ve focused on cost and
...
PART (II) OF THE TALE
• Heavy survey taking
• Panel conditioning
• Declining cooperation
• Increase in demand
• Persistent...
MR INDUSTRY RESPONSE?
”It’s like the whole in the ozone
layer. Everyone knows it’s a
growing problem.”
_____
”But they jus...
TRUTH BE TOLD
ESOMAR 25
2005 – 2006 – 2007 – 2008 – 2009 – 2010 – 2011 – 2012 – 2013 – 2014 –
The Grand Mean Project
ARF O...
BUT IN ESSENCE
“75 years of basing quantitative market research
on a paradigm of projecting the results of
samples onto po...
AND DESPITE THE EFFORTS
Lemon market | Conditions:
_____
• Asymmetry of information (buyer vs. seller)
• Incentive for the...
BLACK BOX
White-box testing
_____
a method of testing [software] that tests
internal structures or workings [of an
applica...
SO... WHAT IS GOING ON INSIDE?
 CLOSER...
DIRTY LITTLE SECRETS OF ONLINE PANELS
ISSUE #1 CONDUCT
_____
N = 1000, whatever it takes
ISSUE #2 RECRUITMENT
ISSUE #3 RESPONSE RATES
Human Interaction?
Trust that the survey is legitimate?
Time involvement with each respondent?
Att...
CONSEQUENCE
The concept of TSE
gets a whole new
meaning…
• “Total” sampling error
• “Total” coverage error
• “Total” non-r...
IS IT EITHER OR?
A Data
Q sample
A convenience
sample of a
convenience
sample
Random
probability
sample
Convenience
sample...
TO CONCLUDE
“If you pay peanuts for your sample, chances are
you're getting monkeys”
(Gordon Morris Global Insights Manage...
LOOKING FORWARD
A few words on the future of online research
1. It’s (increasingly) mobile
2. It moves beyond self-reporte...
THANK YOU
Or google
...there’s only one of me
_____
DAN KVISTBO
dan.kvistbo@norstat.dk
+45 27772016
@Kvistbo
Questions?
Nächste SlideShare
Wird geladen in …5
×

Whats really going on inside panels and does it matter

1.164 Aufrufe

Veröffentlicht am

Veröffentlicht in: Internet, Business, Technologie
  • Als Erste(r) kommentieren

Whats really going on inside panels and does it matter

  1. 1. What’s really going on inside panels? And does it matter? ________ NORSTAT SUMMER LOUNGE June 2014 Dan Kvistbo
  2. 2. SORT OF AN AGENDA  HOW FAR WE ARE WHERE WE ARE GOING 
  3. 3. THE BEGINNING
  4. 4. BARRIERS ALL BUT GONE 1999-2001 _____ • Toluna • GMI • Ciao! • Lightspeed • Zapera • Research Now • etc. _____
  5. 5. LOW COST | HIGH SPEED LightSPEED | FLYresearch | ResearchNOW | SPEEDfacts etc.
  6. 6. FOCUS ON BENEFITS | DATA ASSUMED REP2 RESEARCH BUYERS IN LINE TO MOVE STUDIES ONLINE
  7. 7. CASE IN POINT | P&G P&G | 1999: 15%  2000: 40%  2001: 50% ”To save money and expedite market research surveys”... _____ ”[...] because surveys are completed 75% faster over the Web-and at half the cost of conventional methods [...]”
  8. 8. TROUBLED WATERS | 2003‘ish - 2006 ()
  9. 9. CULMINATION | SEP 2006 RESEARCH INDUSTRY SUMMIT “Two surveys a week apart yielded different recommendations… _____ I never thought I was trading data quality for cost savings” A WOMAN WITH A BUDGET Kim Dedeker, Procter & Gamble's VP-Global Consumer & Market Knowledge, 2006 (Now Chairman, the Americas at Kantar)
  10. 10. WHAT HAD GONE WRONG?
  11. 11. PART (I) OF THE TALE “...clients, have put many pressures on the research suppliers […]” _____ ”We’ve focused on cost and speed and an unintended consequence was erosion in quality. Now, we’re starting to feel the effects”… Kim Dedeker
  12. 12. PART (II) OF THE TALE • Heavy survey taking • Panel conditioning • Declining cooperation • Increase in demand • Persistent mode differences • Long and complex questionnaires • Multi panel membership • The 0.25% doing 30+% • ‘Yes we can’ • Etc. _____ (Source: Reg Baker, Market Strategies International)
  13. 13. MR INDUSTRY RESPONSE? ”It’s like the whole in the ozone layer. Everyone knows it’s a growing problem.” _____ ”But they just ignore it and go on to the next project.”… (S. Morwood, VP Worldwide Market Research for IBM. AdAge oct. 2006)
  14. 14. TRUTH BE TOLD ESOMAR 25 2005 – 2006 – 2007 – 2008 – 2009 – 2010 – 2011 – 2012 – 2013 – 2014 – The Grand Mean Project ARF ORQC ”Million dollar” FoQ project ISO 26362 AAPOR Report on Online Panels FoQ 2 (Reduce the Risk) Report on the AAPOR task force on non-probability sampling ESOMAR/GRBN (draft) Guideline For Online Sample Quality *(Consultation draft) < 26 28  HALFWAY THROUGH! 
  15. 15. BUT IN ESSENCE “75 years of basing quantitative market research on a paradigm of projecting the results of samples onto populations via assumptions of random probability sampling, has been replaced by a willingness to use online access panels, which do not approximate to this model.” _____ - Ray Poynter (Handbook of Online and Social Media Research, 2010)
  16. 16. AND DESPITE THE EFFORTS Lemon market | Conditions: _____ • Asymmetry of information (buyer vs. seller) • Incentive for the seller to pass off a low-quality product as a higher-quality one • Sellers with a bettter product have no way of demonstrating this credibly to buyers • Deficiency of effective quality assurances/guarantees _____ http://en.wikipedia.org/wiki/The_Market_for_Lemons (Paraphrased) Ref: The Market for "Lemons": Quality Uncertainty and the Market Mechanism - George A. Akerlof The Quarterly Journal of Economics, Vol. 84, No. 3. (Aug., 1970), pp. 488-500
  17. 17. BLACK BOX White-box testing _____ a method of testing [software] that tests internal structures or workings [of an application], as opposed to [its] functionality. (http://en.wikipedia.org/wiki/White-box_testing)
  18. 18. SO... WHAT IS GOING ON INSIDE?  CLOSER...
  19. 19. DIRTY LITTLE SECRETS OF ONLINE PANELS
  20. 20. ISSUE #1 CONDUCT _____ N = 1000, whatever it takes
  21. 21. ISSUE #2 RECRUITMENT
  22. 22. ISSUE #3 RESPONSE RATES Human Interaction? Trust that the survey is legitimate? Time involvement with each respondent? Attention given to each respondent? Respondent control over access? Respondent control over whether to respond?      
  23. 23. CONSEQUENCE The concept of TSE gets a whole new meaning… • “Total” sampling error • “Total” coverage error • “Total” non-response error • Etc.
  24. 24. IS IT EITHER OR? A Data Q sample A convenience sample of a convenience sample Random probability sample Convenience sample A quota sample of a random sample Yes, you are ”...trading data quality for cost savings...”
  25. 25. TO CONCLUDE “If you pay peanuts for your sample, chances are you're getting monkeys” (Gordon Morris Global Insights Manager at Sony Ericsson) ”I don’t want it done scientifically, I just want it done right!” (Client, Ron Sellers) “Sometimes close counts in more than horseshoes” (Reg Baker)
  26. 26. LOOKING FORWARD A few words on the future of online research 1. It’s (increasingly) mobile 2. It moves beyond self-reported attitudes, emotions and behaviours to merge survey data and behavioral data 3. Success depends as much on buyers acknowledging their responsibility (ask better questions and adapt to the realm of mobile!) as on sample providers to deliver the appropriate sample & platform
  27. 27. THANK YOU Or google ...there’s only one of me _____ DAN KVISTBO dan.kvistbo@norstat.dk +45 27772016 @Kvistbo
  28. 28. Questions?

×