At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Mixing the Right Sample Ingredients. The presentation was given by Jackie Lorch, VP of Global Knowledge Management for Survey Sampling International. The presentation discussed which factors to blend and emphasized the importance of implementing multi-source testing.

2. Mixing the Right Sample
Ingredients
A New Source Recipe
Jackie Lorch
VP, Global Knowledge Management
Survey Sampling International

3. What is the Problem
We Are Trying to Solve?
1. A systematic problem with online access panels

4. What is the Problem
We Are Trying to Solve?
2. A scarce resource

5. Many Reasons People Are Not in Panels
Why have you not joined an Why do you no longer belong to an
online research panel?1 online research panel?2
I don’t want any more emails 7% Too many surveys 4%
I think they are scams 6% Not enough surveys 27%
It’s a waste of time 3% I never qualified for a survey 44%
Not enough time to take surveys 11% Poor respondent experience 4%
Never heard of an online research panel 20% Not enough time to take surveys 17%
Don’t want to commit to a panel 22% Didn’t want to commit to a panel anymore 8%
Never had any opportunities to join a panel 37% Other 8%
Other 6%
1. Asked of people online who have never joined a research panel
2. Asked of people online, on a panel previously, not currently
Multiple answers allowed

6. What is the Problem
We Are Trying to Solve?

7. What is the Problem
We Are Trying to Solve?

8. A Better Breed of Online Sample

9. Real Differences in Behavior
Own an MP3 player

10. Why Not Use Socio-Demographics?
• Demographics may not be the most helpful
or relevant stabilizers
• Demographic quotas work if the stratification
chosen is relevant to questionnaire topic
• A product may be liked across all demographics

11. Which are the right quotas?
Sample 1 Sample 2
Balanced on: Balanced on:
Age Ethnicity
Gender
Region

12. From Sources to People
• Using thousands of sources
• Metric must be at “person” level

13. Beyond Socio-Demographics
With effective stabilization variables, no need to
maintain same blend of sources project-to-project

14. Factors for the Blend
Broad set of factors defining groups of people,
spanning multiple disciplines
Personality traits Need for cognition (Cacioppo et al)
Music preferences (Rentfrow) Neurographics measures
Cognitive ability Propensity factors e.g., to participate,
(Kahneman and Frederick) share, risk averse, attitude to privacy
Geographic / personality alignment Chronotypes, “lark” or “owl”
(Gosling and Rentfrow) (Roenneberg)
Social Values. Schwarz has international Disruption/orienting reflex measure,
benchmarks habituation disruption

15. Factors for the Blend
• Questions created around 162 variables
• Tested factor performance against questions on
technology, hobbies, interests, brand preference,
loyalty, ad awareness
• Variables tested iteratively on power to
“move the needle” on dependent variables
• 14 variables identified as explaining
more variance than socio-demographics
alone; from these created 8 clusters to control

16. Multi-Source Testing
• Broad set of factors defining groups of people,
spanning multiple disciplines
• Factors included such topics as attitudes to trust,
gender roles, online behavior, personality traits,
ideas and beliefs
• Results varied by source

17. Results Closer to “Truth”
On measures where benchmarks are available,
blend comes closer to it
Social Media
iPod/MP3 Smartphone (Facebook or
Ownership Ownership MySpace)
SSI SurveySpot panel 37 16 60
Benchmark 43 c.20-22 69
SSI Blend 43 20 68
Benchmark sources: Pew Research; comScore; Harris Interactive

18. Implementing the Blend

19. Taming the Stream

20. Lock and Control the Stream

21. Simulation Without Control
MP3 ownership – uncontrolled

22. Simulation Holding Clusters Constant
MP3 ownership – controlled

23. Practical Steps When Blending
• Know your sources
• Use a consistent blend for the entire project
• Use benchmarks and calibrate
• Understand blending techniques being used

24. In Summary
• Multi-sourcing is the future, offering superior sample
• We can keep multi-sourced samples consistent by
pre-profiling participants
• Standard socio-demographics aren’t enough
• A concise list of variables can
be used for a broad range of
research subjects
• Participant experience is key