What we learned about cheaters and repeaters in marketing research, and what we as industry might do about it.

1. 50 Shades of
Grey
What we learned about
cheaters and repeaters
(and thought starters about
what we might do about it!)
Laurie Tema-Lyn (laurie@practical-imagination.com)
Abby Leafe (abby@newleaferesearch.com)
Updated and presented to the QRCA Chicago Chapter
Friday, July 18, 2014

2. Quick poll
2
What percentage of your respondents for a given
study do you believe have been properly
recruited?
 All of them, of course!
 About 75%
 At least half, right?
 I don’t know, I just hope for the best!

3. Would you settle for 85%?
3
“I’d say 85% of the
time I’m legitimately
qualified to do a study.”

4. 4
What we did

5. Straight to the source
If you wanted to participate in focus groups,
where would you go to get started?
Hint: What website do you never want to see
your study show up on?
5

6. 6

7.  73 responses
 Average # of groups claimed in past 2 years
7
74 responses
Average number of
groups (or other
research events)
claimed in the past
two years:
18

8. Who we interviewed
8
8 consumer respondents
Plus two doctors
Mix of age, ethnicity, gender. Average # of groups annually: 10-20;
Average annual income from research: $1,000 - $3,000
Plus 1 outlier!

9. 9
Research frequency Estimated annual income
from research
Ruth 5 groups, 3-4 one-on-ones,
mock jury in the past year
$1,000
Debra 7-8 past year (mostly IDIs) $1,000
Tony 8-10 research events/yr $1,000
Lisa 1-2 per month $2,000 - $3,000
Catherine 10 in the past year including
groups, one-on-ones, blog
$2,000 - $3,000
Amy 20+ over 10 years Est. hundreds plus gift cards
Patty 20-24 per year inc groups,
taste testing, mock jury
$2,000
Henry 60-70 groups per year Est. $6,000- $12,000?
Dr. E 2+ groups per year $1,000 - $2,000
Dr. B A couple per year (not asked)

10. Industry Professionals
 Judy Langer, Langer Qualitative
 Jerry Power, SchlesingerAssociates
 Andrea Schrager, Consumer Centers
 Merrill Shugoll, Shugoll Research
 QRCA Philly and Pacific NW Chapter members
 NewYork & Philadelphia MRA chapter members
 NathalieCoste, Love Brands
10

11. Caveat
 This is qualitative in nature, based on the input of…
 10 research participants in one geographic market
 Market research industry professionals in the
Northeast , Northwest and Paris, France.
 It represents the people we chose to interview and –
in the case of consumers – those whom we found
from Craigslist.
11

12. How much do we care?
 Our take: cheaters matter more than
repeaters
 Cheaters may not always be aware that they
are cheating
 Repeating as gateway to cheating?
12

13. 13
How they get in

14. Quick poll
14
What do you think happens when
you send a final, approved screener
to the facility?
* These findings pertain largely to facility research using their database; we don’t know if our
findings are indicative of what we would find with studies that are recruited by independent
recruiters or recruiting firms, or those with client-provided sample.

15. How C/Rs learn about your project
 Consistently ‘surfing’ for research studies
 Craigslist
 Backpage.com
 Ebay Classified
 FindFocusGroups.com
 Registering with facilities via their website
 Thereby getting emailed about new projects
 Calling facilities to ‘see what they have coming up’
 Networking with others to find out where to sign up
15

16. 16
“Now you’re on their list so you don’t
have to go looking anymore.”

17. How are they screened?
 Many talked about taking “screeners” online,
as surveys, to see if they fit initial criteria
 Most receive a follow-up phone call to
confirm their answers from the initial online
screening
 It can be tricky to keep your answers straight!
Especially when it comes to numbers or
percentages. So some of the “cheating” may
actually be a function of faulty memory.
17

18. At some point, they talk to a person
 This is where things get interesting!
 Experienced respondents know how to answer
certain questions
 Recruiters help them along, too
18

19. 19
“You can eat Doritos for this one.”

20. 20
Merrill Shugoll,
Shugoll
Research
“The
biggest
cheating
factor for
most
doctors is
years in
practice
and how
old they
are.”

21. Quick poll
Which of the following questions, if any, have you
included on a screener in the past 12 months?
(Choose all that apply.)
 Do you, or does anyone in your household or
family, work in any of the following
occupations?
 Which of the following ranges includes your
income? Stop me when I get to the right one.
 How many focus groups have you participated in
over the past 6 months?
21

22. Security, my $%@&
 “Security” questions are by far the ones that respondents
‘fudge’ in order to get picked for a group, especially:
 Industry affiliation (self or family)
 Income
 Past participation
 In-person work requires more care because proof often
required (usually a driver’s license with name and age)
 Interesting discussions have been happening this year on the
Linked In NGMR forum about this topic
22

23. 23
“I’ve been
told to try
to say I’ve
never done
a focus
group…you
tend to get
more
studies that
way.”
“If you tell them you make $19-30 thousand, they
don’t pick you …so I hike up my salary.”

24. Why “cheat?”
 They think it doesn’t matter
 They’ve often been coached to avoid
mentioning how many groups they’ve done
 In fact, they often think that their experience
in a category or with a brand means they will
have more to say and be a better participant
24

25. 25
“As long as
I’m being
unbiased and
giving
opinions, I
should be able
to go every
week.”
“I wouldn’t do 5 research studies on the same topic.”

26.  Product usage questions were deemed
much more difficult to fudge
 It’s hard to know what they are looking for
 You can’t use your‘back’button with an
online screener
 But, still some try to guess the‘right’
answer!
26
So what aren’t they lying about?

27. About those recruiters…
 Respondents universally reported being
“coached” to provide a particular answer
 “I’m supposed to take the first answer.” (But they
come back and record the candidate’s second
response.)
27

28. 28
“Well, we’re looking for people who are NOT in sales or banking… giving
a hint…so what kind of work do you do? It’s not in sales, is it?”

29. 29
“I got a little
bit of unfair
information.
If I were
unscrupulous…”

30. Other things we heard
 Pairing up to take screeners together
 Using more than one email address
 The waiting room as the screening tip
exchange center
 From a facility: Differences by geography
30

31. 31
“Some ladies I’ve met say they do 2-3 a week! ...maybe they
were trying to brag, but they seemed to know what they
were talking about.”

32. 32
“There’s nothing that people won’t lie about and that’s
what we found out when we moved to New York.”
Jerry Power,
Schlesinger
Associates

33. 33
Profile of a Cheater/Repeater

34. Two types of C/R
34
Mr. All-About-the-Money
• Tends to be lower income
• Really needs the money
• May be under-employed or job hunting
• Less concern about bending the truth
Ms. Social Butterfly
• Higher income, often working
• Less concerned with money
• Less likely to bend the truth
• Loves to have her voice heard!
• Strong ego, thinks of herself as a leader

35. Why do they want to participate?
 Money is nice; easy way to make legitimate
money
 Chance to meet people, be social
 Kill a few hours
 Feed the ego, one-upmanship
 Learn about new products, ideas, be in the
know
35

36. They see themselves as…
 Smart, insightful
 Lots to say, particularly because they “know
the ropes”
 Doing a service
36

37. 37
“It’s a no-brainer. I can spend a little time talking
and make a couple of bucks.”

38. 38
When they get in your group

39. “It’s an acting gig”
 Respondents who get into a group
improperly might behave in a few different
ways.
39

40. They prepare
40
“I kind of made up for it by
listening to the radio station
for the past week.”

41. They perform
41
“I am usually the one that ends up leading the group.”

42. “I didn’t know too much about
that particular medical
condition…I figured I would
agree with the person who sat
next to me.”
They get stage fright or they go
to confession afterwards!
42

43. They promote
 They tell their friends (and other professional
colleagues)—
 ”I referred another pediatrician to sign up with
Inspired Opinions.” (Elizabeth, pediatrician)
 “Generational:” Parents may encourage kids
participation
43

44. 44
What do we do now?

45. Screener ideas:
Question assumptions about security
45
Andrea
Schrager,
Consumer
Centers
“Couch it in
a way that
you’re going
to get a
sense for
‘here or
elsewhere.’”

46. Tips:
 Use “here or elsewhere” language
 Avoid giving answer choices for occupation
46
Screener ideas:
Question assumptions about security

47. 47
Screener ideas: Add instructions
NOTE: PLEASE DO NOT POST THIS STUDY ON CRAIGSLIST OR ANY OTHER
SOCIAL MEDIA OUTLET. DO NOT EMAIL ANY PORTION OF THE SCREENER
WITHOUT WRITTEN PERMISSION FROM NEW LEAFE. IF AN ONLINE VERSION OF
THIS SCREENER IS GOING TO BE USED TO RECRUIT, NEW LEAFE MUST APPROVE
THE PROGRAMMED VERSION BEFORE IT IS LAUNCHED.

48. 48
Add instructions to your screener

49. Screener ideas: Role play answers
 Are questions too specific?
 Is it too long?
 Could you answer these questions?
49

50. Screener ideas:Verify, verify, verify
50
“Any single time that you can get proof of what
you’re looking for, you should get proof.”

51. 51
Debbie Fox,
Independent
Field Associate
“We tell the
respondent
they have
to log into
Survey
Monkey and
look at the
actual
packages.’”
Screener ideas:Verify, verify, verify

52.  Require proof of use, purchase
 Ask respondents to go online to identify
actual product used
 Ask potential respondents to use
smartphones to send in pictures
52
Screener ideas:Verify, verify, verify

53. 53
Andrea
Schrager,
Consumer
Centers
“Know that
there’s a team
associated.”
Recruiter ideas:
Vet the recruiter’s procedures

54.  How many people will be recruiting this?
(Team approach is preferred.)
 How do you plan to recruit?
 How do you develop your database?
 How are recruiters incentivized?
54
Recruiter ideas:
Vet the recruiter’s procedures

55. 55
Merrill Shugoll,
Shugoll
Research
“Please ask
them what
protocols
are being
used to
recruit your
work.”
Recruiter ideas:
Vet the recruiter’s procedures

56. Recruiter ideas:
Brief the recruiter
 Brief not just the project manager but the
people who will actually be screening your
project
 Make sure they understand the nuances
56

57. Tip: Add homework
57
“If you do have a homework assignment and you’re
requiring people to think about it…the people who don’t
know anything are going to fall out.”
Jerry Power,
Schlesinger
Associates

58. The last line of defense
58
“We rely on the front desk people…you can catch it that way.”
Jerry Power,
Schlesinger
Associates

59. Wisdom of our peers: Fall 2013Views
59

60. Far out ideas…
60
 Here’s an idea from the QRCA Philly chapter.
Candace
Souweine,
Independent
Marketing
Research
Consultant
“What if we
incent them
to find
cheaters,
repeaters,
whatever?”

61. Far out ideas…
 Facial recognition or lie detection technology
 Can we get rid of the rescreener?
 Recording & retaining the screening calls for
qualified respondents
 Education for respondents or recruiters
61

62. But that’s just a starting place…
62
Let’s hear more from you!

63. Brainstorming activity
 Get into teams of 3 or more
 What additional ideas can you suggest?
 We'll ask you to share some nuggets and post
the rest in the next Connections
63

64. Questions?
64

65. 65
Laurie Tema-Lyn
laurie@practical-imagination.com
908.237.2246
Abby Leafe
abby@newleaferesearch.com
215.497.4974