Forrester Paper: The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech

For B2B Market Research Professionals

November 20, 2009
The Online Panel Quality Wars: All Market
Researchers Benefit — Especially B2B Tech
by Brad Bortner
with Ellen Daley and Chétina Muteba

Executi v e S ummary
At the same time that online quantitative research has taken off, dubiousness abounds about the
representation of the online panel sample. Buyers ask: Are panels representative of the markets
researchers use them for? What can we do about the moral hazard (greatly increased for B2B tech) that
leads bad actors to take surveys inappropriately? A wide variety of approaches have been percolating
in the market ranging from “it’s not really a problem” to various comprehensive solutions. Buyers are
finally pushing back. Major buyers — such as Microsoft and Proctor & Gamble — are making very
specific panel quality demands for their research vendors, and many of the major full-service research
vendors and panel providers are taking note. The end result? Three years of equally competing solutions
with no clear direction for buyers. This means market research professionals must be very specific about
demanding high quality panel for all of your research needs, or risk having your findings undercut when
your internal customers ask: “Is this analysis really representative of our market?”

A Dodgy Sample is the unwelcome guest at the online market reseaRch party
Online, panel-based research has quickly gone from the poor stepchild of market research to the
dominant mode for quantitative research in the US and is now undergoing rapid growth in Europe.1
Why the rapid growth? It’s faster, cheaper, and delivers innovative new capabilities.2

The fly in the ointment of all of this growth is suspicion about how representative online panels really are.
This is not just a matter of maintaining a representative panel; it is a matter of ensuring that respondents
are who they say they are, answer accurately, and do not take a survey more than once — under multiple
aliases. While the Internet greatly simplified the job of recruiting and managing panels, it also opened
the door to potential problems based on the moral hazard, the so-called “professional survey taker”
problem. What is going on?

· Bad actors, in general, are incented to behave badly. Online surveys generally pay per survey.
Those respondents who are financially motivated may attempt to take many surveys to make as
much as possible. Some accomplish this goal by signing up for many different panels, perhaps under
different identities. Others may disguise themselves to appear more attractive to a survey, or sign
up for the same panel under multiple personas. The end result can be one person taking the same
survey multiple times (especially when panels are mixed), creating invalid responses by people
pretending to be what they are not. There are even cases of bad actors creating “survey bots” to
harvest fees by responding to large numbers of surveys.3 The end result is corrupted insights.

Headquarters
Forrester Research, Inc., 400 Technology Square, Cambridge, MA 02139 USA
Tel: +1 617.613.6000 • Fax: +1 617.613.5000 • www.forrester.com

The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech 2

· The bad actor and moral hazard problem is worse for B2B. The incentives are much larger for
B2B surveys — $2 for B2C versus $20 or more for B2B — this is even greater in B2B technology
areas because there are fewer qualified potential respondents and a great desire to survey
them by tech vendors, resulting in a greater potential of people claiming they have roles and
responsibilities they do not to harvest the incentives.4

Are these really problems? Yes.5 While vendors are making strong efforts to address the issue, it’s a
buyer-beware market at this early stage. Since many buyers can’t see the effects of bad quality, which
are often subtle and hard to detect, they haven’t been informed shoppers and chosen — or told their
full-service vendor to choose — panel X over panel Y.6

However, major research buyers have recently been quite vocal about the need to fix this issue and
are becoming savvier. In particular, large, sophisticated online research buyers’ demand for quality
is galvanizing research providers and giving a lift to third-party solutions for panel quality issues in
this market. In addition, industry groups are in the game as well. For example, ARF, ESOMAR and
others are trying to assess and improve this situation by providing best practice guidelines to panel
vendors, and ARF has just released its own assessment to size the issue. 7

A Broad Array Of Approaches Offer Solutions — None Dominate . . . Yet
Multiple solutions and approaches to the problem of panel quality have been developing over the
past few years. These include panel consolidation, widespread adoption of machine fingerprinting,
panels with unique approaches to quality, and more comprehensive automated solutions. While
up to now there has been little industry consensus as to which approach is best, each of these four
approaches for buyers has been helping align the market behind enhanced quality:

· Panel mergers allow use of ultra-large panels. As Forrester predicted would happen in 2008,
in part because of the scale needed to invest in panel quality, several major panel merges have
occurred, including Greenfield/Toluna and e-Rewards/Research Now.8 Really large panels
have an advantage in that they don’t have to worry about de-duping against panel properties
they don’t own, and can focus on cleaning a panel they wholly own. This helps, but does not
eliminate the need to resolve issues around identity verification and assessing the validity of
respondents’ answers.

· Machine fingerprinting is gaining wide acceptance. Machine fingerprinting involves using
multiple identifying marks on a computer to ensure that no computer signs up to take the same
survey more than once. Many major panels are now using it, such as e-Rewards, Toluna, and
SSI. This is certainly a good step to take; however, it does not resolve the issue that computers
generally don’t take surveys, people do. It is possible for surveys to be taken from multiple
computers, or to attempt to disguise machine fingerprints.9 Some of the major vendors in this
space include Peanut Lab’s Optimus and RelevantID.

November 20, 2009 © 2009, Forrester Research, Inc. Reproduction Prohibited


· The use of unique identity confirming practices. E-Rewards is a leader in using by invitation-
only panel creation as opposed to open recruiting on the Internet and qualifying people as they
ask to join the panel. Western Wats also uses a form of invitation-only recruit by using excess
phone capacity to recruit. LinkedIn’s unique approach is to recruit B2B panel respondents from
its highly profiled social network.10 The end result for these solutions should be panels whose
members are much more verifiable. None, however, provide a saleable solution, especially when
more samples are needed then they can provide.

· Solutions that are more scalable have been expanding. MarketTools’ TrueSample has been
advocating a three-part “unique, identifiable, and engaged” solution. This approach attempts to
go beyond pure machine fingerprinting. The first of the three parts compares the sample against
publically available databases to ensure that the respondent exists. They then use machine
fingerprinting. Finally, they assess survey-taking behavior to see how engaged the respondent
is. Their claim is that up to 30% of samples get scrubbed when they go through this process.
This approach is salable, deployable against multiple panel sources, and works with many major
online survey tools such as Confirmit and SPSS. Unfortunately, many have been resistant to
work with a company they view as a research competitor.

Why has consensus about which solution to back proven elusive? Two reasons: cost and competitive
forces. First, most panels have a disincentive to filter their sample for quality issues if it would reduce
their panel by up to 30%, and most buyers are not asking for it. Second, to the extent that solutions are
marketed by nominal competitors, major research firms are resistant to partnering with them.

Buyer Demand is Forcing Movement toward Sample Quality solution consensus
The rumbling by major research buyers has been growing for a few years, but is now growing to a
fever pitch. Recent industry sessions about panel quality at ARF and other associations have been
tumultuous and quite well attended.11 Recently, major sample buyers, who have a lot of clout in this
market, are starting to insist on specific solutions or at least specific, verified high quality sample
outcomes.

Major Buyers Are Demanding, And Providers Are Reacting
The pressure from big research buyers is driving several reactions in the market. Full-service vendors
are concurring, or offering to come up with their own solutions, and panel vendors are doing
likewise. Even the machine fingerprinting solutions are starting to have a frontrunner, as panels seek a
nonthreatening partner to answer buyer’s needs for cleaner samples. Where are the lines being drawn?

· Big buyers are forcible pushing for change, often to favor TrueSample. This includes companies
such as ARSgroup, General Mills, Microsoft, and Proctor & Gamble. After extensive internal
analysis, Microsoft has announced it will only do business with market research vendors that use



TrueSample. General Mills has taken a similar stand. P&G is not yet forcing all of its vendors to
use TrueSample, but its criteria are such that that would be an obvious solution for its vendors,
and Nielsen BASES has agreed to be a TrueSample certified vendor for P&G. ARSgroup has also
decided to only use TrueSample for its massive online sample needs. Given that these sample
buyers are quite large, their decisions could potentially start to coalesce the market. 12

· GfK Group, IPSOS, Kantar, and Synovate formed “The Consortium” to address panel
quality. When P&G, after significant analysis, approached its major vendors (all which have
major private online panels) and requested enhanced panel quality, it decided to team up to
come up with its own solution. The results of this activity remain to be seen. On one hand, these
are major full-service research houses that have very deep pockets and resources. On the other
hand, they are competitors with each other. It is even possible that they may elect to sign on
with one of the other panel quality initiatives. Only time will tell.

· Panel providers and some full-service vendors are choosing sides. Toluna/Greenfield has
committed to providing TrueSample-certified sample to clients that wish it. MRops, a major
B2B research operations house, has standardized on Optimus. The bottom line for these, and
other market intermediaries, is that they will ultimately adopt the measures that their clients
demand.

· RelevantID is emerging as a machine fingerprinting frontrunner for panel vendors. Peanut
Labs was really a first mover in the machine fingerprinting market. There is a sense that it is
being held back by panel companies’ perception that it is a competitor. RelevantID has made
significant progress among panel vendors in the past year since it provides a similar solution
and is not perceived as a competitor, while Peanut Lab’s strength is in different parts of the
research ecosystem.

The Next Three Years Will Have Equally Competitive Solutions
The future holds better, higher quality sample! At the moment, TrueSample seems to have the wind
at its back, but the final resolution in the market will depend on part on what The Consortium
comes up with. Can these competitors actually work together to build a better mousetrap, or will
they simply license an off-the-shelf solution such as TrueSample? Will more buyers demand such
a comprehensive solution, or will they be comfortable with panel vendors best efforts and machine
fingerprinting? While Forrester believes ultimately one solution will at last dominate, that won’t
happen for a while. Expect these solutions to equally compete for the next three years. What are the
possible scenarios after that time?

· TrueSample triumphant! More and more buyers follow their peers and insist that their entire
sample be TrueSample-certified so they take the discussion about panel quality off the table. At
a certain point, it becomes the industry standard.



· The Consortium strikes back! The major full-service market research vendors come up with
their own solution that is equal or superior to TrueSample, and this satisfies the requirements of
major research buyers. Once such a solution exists, they could either set it up as a business in its
own right or keep it as a competitive advantage.

· Panel vendors go it alone, with machine fingerprinting-plus. It is still possible that panel
vendors could assert that machine fingerprinting, combined with other internal quality
initiatives, is good enough. Since the measures of quality sample are often up to argument,
and many buyers can’t even decide if somewhat better sample actually drives better business
decisions, this is a possible outcome if major buyers do not coalesce around one specific
solution in their sample quality demands.

R ecommendations

this is not the time for complacency, demand good sample now!
Now that transparency is possible, it is critical that buyers demand high-quality sample both from
their panel providers (if they have in-sourced quantitative research) and the full-service vendors
they are using. This is especially true for B2B sample in the tech industry, which is a rare and
eagerly sought after commodity. Fundamentally, bad sample risks driving bad decisions, and that
is something which no market researcher professional wants to be responsible for.

· If you have a direct relationship with the panel provider, demand and test for quality.
Find out how they are weeding out professional survey takers, de-duping when they merge
panels, and ensuring that panelists are engaged. Be sure to trust, but verify, by actually
insisting that there are tests in your surveys to catch respondents who are giving irrational
answers.
· If you are dealing with a full-service research provider, it’s not the time to play trust me.
Insist that they provide all of the same quality approaches as described above. Have them
detail how much sample is being scrubbed. If the answer is “almost none,” be suspicious. If it
is a B2B survey in the technology space and they give that answer, find another vendor.
· Keep an eye open for the next wave. All of the solutions in play are focused on fixing
quality issues with online panels. As with many technology-driven advances, sometimes the
market moves on to new paradigms just as the old one gets perfected. Keep an eye open
to see if the market is moving on from panels to more of a river sample or other forms of
surveying that are more focused on replicating the random sample experiences that phone
can deliver offline in an online mode. Perhaps surprisingly, the CEO of one major panel
vendor believes that such is the future, and is driven by a combination of ubiquitous Web
access and the limit to the current populations who will agree to join panels.



Endnotes
1
Source: Inside Research, Issue 249, February 2009.
2
Initially, because using online sample and tools reduced costs by up to 75% and cut the time to results by
about one-third versus traditional phone and in-person quantitative data gathering modes. Additionally,
the online tools that enabled surveys of online panels greatly simplified fielding complex survey designs,
including skip patterns, complex analytics (such as conjoint), pictures, and multimedia. See the July 19,
2007, “Five Keys To Success With Online Panels” report and see the May 1, 2007, “Online Panels: Cheap
And Effective, But Buyer Beware” report
3
This type of activity is not uncommonly launched from certain developing nations.
4
The major drivers of B2B tech sample quality issues are that it is a very scarce commodity, with a very high
moral hazard of incenting bad actors. See the April 17, 2009, “Watch Out For The Landmine Of B2B Online
Research” report.
5
Add to this the innocent panel members who just happen to like to take a lot of surveys. They may well
also sign up for multiple panels and be accidently swept into the same survey when panels are combined to
reach large sample goals. If they are not de-duped properly, even these non-bad actors can end up having
their responses counted more than once.
6
The effects of bad actors on surveys can range from obvious illogical responses to an over-reporting of
product control or utilization (to qualify for the survey) and averaged responses, since respondents are
trying to intuit how they should answer if they actually had that role or used that product.
7
ARF embarked on a detailed analysis of single panel bad sample, and found that they could indentify 20%
corrupt sample in a typical panel. ESOMAR issued detailed panel management guidelines to try to abate
the issue. comScore continues to stand by its assessment that a very small percentage of respondents are
responsible for many of the responses for some panels. MarketTools has analysis that shows the problem is
closer to 30%. Admittedly, MarketTools and comScore can be claimed to be self-interested in their findings,
but there is a lot of industry activity for something that some panel providers would consider a nonissue.
8
We made the prediction that panels are likely to consolidate in 2009. See the December 2, 2008 “Predictions
2009: What Happens In Market Research?” report.
9
Many vendors of such solutions would argue that this is not that likely.
10
LinkedIn’s entry into the B2B (especially tech) sample space has drawn a lot of attention. Since people
do not join their community to take surveys, there should be far less moral hazard, and their sample is
potentially very well profiled given the wealth of information members provide when they join. The open
question about LinkedIn, however, is can it scale big enough to meet sample demand since providing B2B
sample is not its core business.
11
These have been notable for leaving participants unhappy in that industry players could not seem to agree
on an approach.



12
Decision facilitated by version 3.0 of its platform, which works with major survey platforms, such as
Confirmit, SPSS, and MarketTools, and can clean sample on the fly, even from non-TrueSample certified
panels.

Forrester Research, Inc. (Nasdaq: FORR) is an independent research company that provides pragmatic and forward-thinking advice to global leaders in business
and technology. Forrester works with professionals in 20 key roles at major companies providing proprietary research, customer insight, consulting, events, and
peer-to-peer executive programs. For more than 26 years, Forrester has been making IT, marketing, and technology industry leaders successful every day. For
more information, visit www.forrester.com.
© 2009, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinions
reflect judgment at the time and are subject to change. Forrester®, Technographics®, Forrester Wave, RoleView, TechRadar, and Total Economic Impact are
trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. To purchase reprints of this document, please email
clientsupport@forrester.com. For additional information, go to www.forrester.com. 55747

Forrester Paper: The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Empfohlen

Empfohlen (20)

Forrester Paper: The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech