Weitere ähnliche Inhalte
Ähnlich wie Workshop on public relations measurement (20)
Mehr von David Geddes (6)
Kürzlich hochgeladen (20)
Workshop on public relations measurement
- 2. • Smart consumer
• Quantitative survey methods
1. Telephone
2. Web
3. Mail and multi-modal
4. Face-to-face
• Sampling
• Case studies
• Questions … as they come
Copyright © 2010 by The Institute for Public Relations
2
2
- 3. American Association for Public Opinion Research web
site (www.aapor.org)
Copyright © 2010 by The Institute for Public Relations
3
- 4. What are your business and comms
objectives?
What do you want to explore,
discover, test, or document?
What do you know already?
Who are the right people to talk
with?
What are appropriate data collection
methods?
Copyright © 2010 by The Institute for Public Relations
4
- 5. • Types
– Random digit dial (RDD)
– List from sample provider or panel
– Company or client list (customers, employees,
industry analysts, donors, partners, etc.)
• Uses and advantages
• Limitations and weaknesses
Copyright © 2010 by The Institute for Public Relations
5
5
- 6. Cell phone only households
– Same or different?
– What to do?
– Ethical and legislative issues
– A trend to follow
Copyright © 2010 by The Institute for Public Relations
6
- 8. • Standard Survey: 36% response rate
– Calling done over five days
– Selected respondent from people at home at time of
call (no random selection)
– Five call-backs, one call-back to refusals
• Rigorous Survey: 60.6% response rate
–
–
–
–
–
Eight-week calling period
Random selection of respondent from list
Pre-notification letters with $2 incentive
Multiple attempts (including letters to refusals)
Multiple call-backs
Source: Scott Keeter et al, Consequences of Reducing Nonresponse in a National
Telephone Survey, Public Opinion2010 by The Institute for Public Relations
Quarterly 64:125-148 (2000)
Copyright ©
8
- 9. • Oversampling
– Example: Attitudes to location-based apps
– Example: National survey on water conservation
• Weighting
– Example: Ethnicity or Age
– Example: Survey with oversampling
Copyright © 2010 by The Institute for Public Relations
9
- 10. •
•
•
•
National random sample of 1,000 households
Offered by all major research firms
Fast
Low cost
– Cost per question (phone vs. omnibus)
– Demographics included
– Costing parameters
•
•
•
•
•
Targeted audiences
Low cost alternatives
Deliverables
Applications
Limitations
Copyright © 2010 by The Institute for Public Relations
10
- 11. • Advantages and uses
• Limitations
– See AAPOR site
Copyright © 2010 by The Institute for Public Relations
11
- 12. • Don Dillman et al, Internet, Mail, and Mixed-Mode
Surveys: The Tailored Design Method, (Wiley, 2008)
• TDM method:
1.
2.
3.
4.
5.
6.
•
Respondent-friendly questionnaire
Personalized correspondence
Token financial incentive ($1 or $2 prepaid)
Up to five phone contacts
Mail survey with stamped return envelopes
Phone again
Other ways to improve response rates
Copyright © 2010 by The Institute for Public Relations
12
- 13. – Examine different sampling techniques
– Strengths and shortcomings
– Cases and examples
– Tools to make decisions in practice
– Not a comprehensive textbook treatment
Copyright © 2010 by The Institute for Public Relations
13
- 14. • Formal definition of our targets
– Example: Caregivers of Type II diabetes patients
• Generalize or project?
– Example: Poll of Kansas City voters about rental
car tax
• Understand and minimize sampling error
– How far off might our result be if we interviewed
another group of individuals?
• Make tradeoffs
– Budget, time, other factors given objectives
Copyright © 2010 by The Institute for Public Relations
14
- 15. • Probability sampling
– “Sampling is the science of systematically
drawing a valid group of objects from a
population reliably.” (Stacks, p. 150)
• Non-probability sampling (informal
definition)
– Process of systematically drawing a group of
objects from a population sufficient to meet
information needs. (Adapted from Stacks)
Copyright © 2010 by The Institute for Public Relations
15
- 16. • Universe
– General concept of who or what will be sampled
• Population
– People or units to be sampled, formally defined
and described
• Sampling frame
– List of all people to be surveyed
– Example: List of all 90,000 registered veterinarians
under age 65
Copyright © 2010 by The Institute for Public Relations
16
- 17. • Sample
– Actual people chosen for inclusion in the research
– Example: Selection of 10,000 veterinarians from
the list
• Completed sample
– People who actually responded to the survey
– Example: 3,000 veterinarians completed the
survey
Copyright © 2010 by The Institute for Public Relations
17
- 18. • Sampling error
– Issue: Potential error or uncertainty as a result of not
sampling from all members of sampling frame
– How far off might we be if we interviewed a different
500 people?
• Coverage error
– Issue: The sampling frame does not contain all
members of a population or contains a biased list
– Example: People without landlines in a telephone poll
– Example: People with invalid e-mail addresses in
membership list
Copyright © 2010 by The Institute for Public Relations
18
- 19. • Measurement error
– Error when respondents misunderstand or
incorrectly respond to questions
• Nonresponse error
– Respondents unlike non-respondents
Copyright © 2010 by The Institute for Public Relations
19
- 20. • Understanding and reducing coverage
error
– Does the sampling frame (list) contain
everyone in the population?
– Does the list contain people who are not in
the sampling frame?
– How is the list maintained and updated?
– Does the list contain other information that
can be used to improve sampling?
Copyright © 2010 by The Institute for Public Relations
20
- 22. • Interview or measure all members of a
population
– Example: Wal-Mart annual employee survey
• No error due to sampling!!!
– Other types of error
• Rare in practice
• Is it worth the effort?
Copyright © 2010 by The Institute for Public Relations
22
- 23. • Every individual in a population has an
equal chance of being chosen
– In theory
– In practice
• Allows generalization or projection to the
population
• Known sampling error parameters
• What other sources of error?
• How much to invest, given objectives?
Copyright © 2010 by The Institute for Public Relations
23
- 24. Key types of probability sampling
• Simple random sampling
• Systematic sampling
• Stratified random sampling
• Cluster sampling
Copyright © 2010 by The Institute for Public Relations
24
- 25. • Interview or measure without access to every
individual in a population
– Examples
• Situations where it is difficult to fully specify
the population or sampling frame
– Examples
• Cannot generalize
– How far off might our result be if we interviewed
another group of individuals?
• Key: Understand limitations … justify choice
Copyright © 2010 by The Institute for Public Relations
25
- 26. • Convenience sampling
– Selecting based on availability
– Example: Hospital survey of nurses leaving a shift
• Quota sampling
– Selecting based on availability but weight based
on predetermined characteristics
– Example: Mall intercept sampling
Copyright © 2010 by The Institute for Public Relations
26
- 27. • Purposive sampling
– Selecting participants based on knowledge of the
population and focus or objectives of the research
– Example: Survey of most influential journalists
covering the air transport industry
• Volunteer sampling
– Select based on agreement to participate
• Snowball sampling
– Selecting participants based on recommendations
of other participants
Copyright © 2010 by The Institute for Public Relations
27
- 28. • Key questions:
– How much might our results differ had we
interviewed another 100 American voters?
– How much more would we learn, given our
objectives, had we interviewed another 100
customers?
– More technically, how much sampling and
measurement error can we tolerate?
• To reduce sampling error and measurement
error, you must increase sample size
Copyright © 2010 by The Institute for Public Relations
28
- 29. • “Normal” curve
– Mean, standard deviation,
we can calculate confidence
intervals
– See an interactive demo at
68%
95%
99%
http://geographyfieldwork.com/StandardDeviation1.htm
– Sample size calculators on Web
• National Statistical Service http://
www.nss.gov.au/nss/home.NSF/pages/
Sample+Size+Calculator+Description?
OpenDocument
Copyright © 2010 by The Institute for Public Relations
29
- 30. – Sample size of 385 is
necessary for a
confidence level of plus
or minus 5 percentage
points at the 95%
confidence level.
– Is this the biggest
source of error?
Copyright © 2010 by The Institute for Public Relations
30
- 31. • National random digit dialing completing surveys
with 1,000 adults
• Conducted Friday through Sunday
• Balanced post-survey to census figures for age,
gender, HHI, ethnicity (results only differ slightly)
• Evaluation
• Universe and population
• Sampling frame
• Sample and completed sample
• Sources of error or bias
• Final assessment – when is this appropriate?
Copyright © 2010 by The Institute for Public Relations
31
- 32. – National online panel survey with 1,000 adults
– Balanced post-survey to census figures for age,
gender, HHI, ethnicity (results only differ slightly)
– Evaluation
• Universe and population
• Sampling frame
• Sample and completed sample
• Sources of error or bias
• Final assessment
Copyright © 2010 by The Institute for Public Relations
32
- 33. – Four divisions
– Management vs. non-management
– Results by age, gender, tenure at company
– Which survey methods?
– Develop a sampling plan:
• Universe and population
• Sampling frame
• Sample and completed sample
• Sources of error or bias
• Final assessment
Copyright © 2010 by The Institute for Public Relations
33
- 34. – American Veterinary Medicine Association
• 90,000 veterinarians under age 65
• 50,000 valid email addresses
• Goal: Low-cost survey
– Evaluation
• Which methods?
• Universe and population
• Sampling frame
• Sample and completed sample
• Sources of error or bias
• Can we work around the limits?
• Final assessment and recommendation
Copyright © 2010 by The Institute for Public Relations
34
- 35. – Client: Financial services company
– Respondents: List of 1,000 journalists who cover
personal finance, the economy, and lifestyle.
– Which survey methods?
– Evaluation
• Universe and population
• Sampling frame
• Sample and completed sample
• Sources of error or bias
• Final assessment and recommendation
Copyright © 2010 by The Institute for Public Relations
35
- 36. Public relations research
• Don W. Stacks and David Michaelson. 2010. A
Practitioner's Guide to Public Relations Research,
Measurement and Evaluation. Businessexpert Press.
• Don W. Stacks. 2002. Primer of Public Relations
Research. New York: Guilford Press.
Market research (leading business school texts)
• Gilbert A. Churchill and Dawn Iacobucci. 2004.
Marketing Research: Methodological Foundations.
Mason, OH: South-Western Cengage Learning.
• Naresh K. Malhotra. 2007. Marketing Research: An
Applied Orientation. Upper Saddle River, NJ: Prentice
Hall.
Copyright © 2010 by The Institute for Public Relations
36