How to design an effective survey: The good the bad and the ugly

How to design an effective survey
Workshop

Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J

26.02.2013

Workshop overview
10.00-10.15

Introduction to survey design

10.15-11.15

Writing good questions

11.15-11.30

Morning tea

11.30-12.30

Tips and tricks
How to maximise response rate
Getting the most out of your data

12.30-1.00

Q&A and Close



Introduction
So you want to create a survey...
 What is the purpose of the survey?
• Needs assessment
• Impact assessment
• Knowledge assessment
• Attitudes assessment
• Behaviour assessment
• Client satisfaction / program feedback
• Program evaluation

 What survey mode will you use?
• Telephone, online, paper, interview


When to use a survey
 To collect information from a large number of
people or records
 To collect this information relatively inexpensively
 To say something about the population from which
the sample was drawn
 When you do not need very in-depth information
 When you want the data collection procedure to
be as standardised as possible


Some advantages of surveys
 Anonymity
 Asking sensitive questions

 Standardised data to monitor trends
 Quick to administer
 Simple data analysis (%, frequencies)

 Data can be presented as graphs/charts
 Multiple distribution channels (online,
paper, social networking sites)



Some limitations of surveys
 Limited explanations for the data
 Comments open to a range of
interpretations
 Don‟t know why questions were skipped
or why survey was partially completed

 No opportunity to explain questions that
people don‟t understand
 Can‟t guarantee the survey was even
received or completed by the right person

 Basic literacy levels needed


What makes a ‘good’ survey?
Have you ever created a survey and felt
disappointed with the data you collected?
Have you ever been asked to answer a survey and
felt irritated with the questions asked?
 A good survey considers both your needs for data
and the capabilities and experience of the
respondent.

 Collects complete, accurate, and reliable data.



What is a ‘bad’ survey?
 Too wordy
 Language is hard to understand

 Bad timing - e.g. respondents tired at the end of a session
 No „Other‟ response category
 Questions that seem irrelevant to the topic

 Repetitive
 Boring – e.g. same structure throughout, no images
 Does not consider the capabilities of the respondents



Is a long survey a bad survey?



HOW TO DESIGN AN
EFFECTIVE SURVEY
IN 3 EASY STEPS!

Step 1.
Establish your survey goals!
Ask yourself:
 Why am I creating this survey?
 What do I hope to learn or
accomplish?
 How do I plan on using the data
that I collect?
 How will the data influence my
decisions?



Step 2.
Create a ‘dummy table’
A tool you may not have used before. The dummy table
helps ensure all survey objectives are addressed and all
survey questions are relevant. Consider data analysis
before, during and after survey development.
Areas of inquiry

Questions

Demographics



A dummy table …
• Ensures you can interpret data
• Ensures each question will collect the
information you need
• Ensures you leave nothing of importance out
• Ensures you don‟t collect more information
than you need



Example: Health Care Experience of the
LGBTI* Community in WA
Activity – Find questions related to:
Area of inquiry

Questions

LGBTI health issues
Demographics and lifestyle
Current experience of healthcare
services
How and where to reorient current
health services
*LGBTI – lesbian, gay, bisexual, transgender, intersex


Answers

Area of inquiry

Questions

LGBTI health issues

10,11,24

Demographics and lifestyle

1,2,3,4,5,6,7,8,9,15,16

Current experience of healthcare
services

12,13,14,20,21,22,23

How and where to reorient current
health services

17,18,19,25



Step 3.
Write good questions
Try to use a range of question types:
 Open-ended questions
• To explore issues
• Open to a variety of responses (comment boxes)

 Closed-ended questions
• Dichotomous
• Likert scale
• Guttman scale

• Multiple choice (one response, multiple responses)
• Rank order


Example - Dichotomous



Example – Likert Scale



Example - Guttman Scale
 Agreement with any one item implies agreement with the lower-order
items.
 E.g. are you willing to permit _______ to...
-

live in your country?

-

live in your community?

-

live next door to you?

-

marry your child?



Example – Multiple Choice (one response)



Example – Multiple Choice
(multi-response)

Evaluation of Health Promotion to design an effective survey
How
382


Example – Rank Order
 Rank the following methods of receiving a positive STI
diagnosis in order of preference from highest to lowest
(1=your most preferred method; 5=your least preferred
method)
• SMS______
• Phone call ______
• In person ______
• Email ______
• Letter ______



Developing a Client Satisfaction
Survey
Example: The Wheatbelt Resilience Program
 Included a series of community workshops
 A survey was developed to find out how useful the
workshops were in terms of increasing awareness of the
Wheatbelt Resilience Program; understanding of suicide
and suicidal behaviours; and opportunities to help with this
issue in the community



Can you identify
any ways in which
this feedback sheet
could be improved?



The questionnaire
was adapted to
capture more
specific information
and to also gauge
satisfaction with the
workshop. The
pre-questionnaire
was on a separate
document to the
post-workshop
evaluation.



WRITING GOOD QUESTIONS


What do you want to know?
What do you need to know?
 Don‟t waste the opportunity to collect good quality
data by asking vague questions
 Be economical with the number of questions but
make sure you collect all the information required



Activity: Work in small groups
Re-write these questions and include closed
ended responses:
 (a) “How often do you participate in physical
activities on average?”
 (b) “When were you last physically active?”



What to avoid
 Double-barreled questions
 Biased/leading questions

 Questions with a double negative
 Loaded questions



Example: double-barreled question
Please indicate how much you agree or
disagree with each of the following statements
about our service:
I feel welcomed by staff and other youth at the
centre

Example: leading question
Negotiating condom use with a sexual
partner is difficult. Did this session help
you feel more confident to negotiate
condom use successfully?


Example: double negative
How do you feel about the following statement?
We should not reduce investment in sexual health
services.
Strongly Agree
Agree
Disagree
Strongly Disagree

Example: Loaded questions
Which of the following services most
urgently requires improvement?



What to avoid
 Questions which could create a response set
i.e. the tendency for a respondent to answer later questions
the same way they answered earlier questions

 Technical language/jargon
Use simple, common words

 Emotive language
Do you approve of the government‟s oppressive immigration
policy?



What to avoid
 Using “and”, “or”, “but” in the same question

 Long, complex, multi-part questions
 Questions which contain a negative and a positive
If you were in a foreign country and did not speak the language but knew
someone who did would you…
Huh?!
a. Let the person that spoke the language speak for you
b. Try to learn the language on your own
c. Not try to learn the language or let the person that you knew that spoke
the language speak for you.



What to avoid
Response categories that are not exhaustive
What is your current age?
10 to 20
21 to 30
31 to 40

I‟m 41!

Response categories that are not mutually exclusive
What is your current age?
10 or less
10 to 20
20 to 30
30 or greater

I‟m 30!


Avoid fuzzy words in the question

 Those words which are open to different
interpretation by the respondents.
• Have you ever had any major injuries during the past year?

 Avoid words like:
important, major, better, regular, often, occasional
(subjective interpretation).
 Best to give specific time & details


Irrelevant questions
 Questions which are irrelevant to the survey
objectives
• E.g. do you really need to know someone‟s income?

 Questions which are irrelevant to the respondent
• e.g. Don‟t ask nurses to compare the effectiveness of different
patient management techniques if they have no experience of the
alternatives.



Presuming knowledge
 Don‟t assume people either know or have
appropriate experience to answer the
question.
What is
• Have you been bullied this year?

bullying?

• How many standard drinks have you drunk in the
last week?
• How many cigarettes have you smoked in the
last week?

 Use filter questions or provide more
information to avoid these problems


What is a
standard
drink?
I don‟t
smoke!

Presuming knowledge

 Hypothetical or predictive questions
• Would you like to try marijuana?

• Avoid - many people might like to try anything once.

 Generally avoid:
• Would you like to...
• What would you do if..?



Personal questions
 Be sensitive!
 May need to preface personal questions to give
respondent „permission‟ to respond truthfully

 Consider the structure of response choices
• people don‟t like to consider themselves at extreme ends of a
scale



Activity: Design a brief survey to
evaluate this workshop
Workshop objectives
• Increase participants’ knowledge and skills in effective survey design
• Assess participants’ satisfaction with the SiREN training workshop
Areas of inquiry

Questions

Demographics



TIPS AND TRICKS


Critically assess your survey
questions
 Is the respondent‟s understanding of response
categories likely to be influenced by more than
words?

 The vaguer the question and answer categories,
the greater potential for error / poor data
 Is the survey information being collected in more
than one mode? E.g. phone, interview, email



questions
 Is changing a question acceptable to the survey
sponsor?
 Is the question necessary?
 Will it improve the validity of your study if you do
ask more than one question?



questions
 How easy is it to get an accurate answer?
 Can people accurately recall and report past
behaviours?
 Will the respondent be willing to reveal the
requested information?
 Will the respondent feel threatened or defensive
by the question?
 Will the respondent feel motivated to answer each
question?


Tips and tricks
 Insert dummy questions
• These may be irrelevant to the research to avoid respondents
guessing desirable answers

 Reverse the wording of some survey items
• This technique can avoid creating response sets
I always enjoy going to work
I dread going to work



Tips and tricks
 The order of questions may influence responses
• Put interesting questions first
• Sensitive questions towards end
• Demographic information – location of these questions depends
on importance
• Similar themes together

 Put instructions for questions before the question
not at the beginning of the survey



Tips and tricks
 Be specific
Example: How can we improve our service?
This is too vague and may elicit single responses about many
different aspects of the service without much guidance to help you
decide which ones are important.
Instead, consider asking respondents to rank from 1-5 the quality of
different aspects of the service

 What claims will you use the data to support?
Be honest and consider if this is realistic or credible. This technique
helps you to clarify the wording of questions



HOW TO MAXIMISE RESPONSE
RATE

Aim to get a 65% response rate
 That is: 65% of your sample complete and
return a questionnaire
 Remember non-respondents may bias your
results
 Choose a sample size that takes into account
response rates
 Sample size for pre- and post- surveys should be
large enough in case of attrition
 If possible, have participants complete survey at
the time of distribution


How to maximise response rate
 Involve your target group in design if possible
 Pre-test!
 Ask questions that people can and will answer

 Uplift, motivate and encourage the respondents to
give information
 Avoid bad survey questions
 Allow sufficient time for responding to the survey
and for participants to complete the survey


How to increase responses
 Plan multiple contacts
• Pre-notification (tell people you are planning to survey)
• Reminders for non-responders

 Consider incentives
• Individual incentives more effective than prize draws e.g. incentive
given for getting others to complete the survey
• Dependent on resources

 Obtain ethics approval for your survey



A note about survey ethics…
Considerations may include:

 Information and consent forms
 Confidentiality and anonymity
 Samples which include minors
 How to ask questions about potentially illegal activities
 Length of survey / burden on respondents
 Format of survey - e.g. acceptability, culturally appropriate
Please contact siren@curtin.edu.au for more advice about survey ethics


GETTING THE MOST OUT OF
YOUR SURVEY DATA

Planning and evaluation cycle
Needs
assessment

Re-planning

Survey
data

Program
planning

Survey
data

Survey
data

Evaluation
Implementation
Survey
data



Use different
types of survey
data as part of a
continuous
program
planning and
evaluation cycle

How will you use the data?
 Use survey data to inform the program planning cycle

 Adopt a mindset of continuous program improvement
 After implementing the survey, critically evaluate the
quality and completeness of the data collected
• Were any questions ambiguous?

• Were all the questions necessary?
• Did any questions appear to cause confusion?
• Did you get some unexpected responses?
• Were the data useful?

• What decisions could the data inform?
• Are there still gaps in the data?


Ask yourself what
will you change?

Close the feedback loop
 Finally, it is important to „close the
loop‟ when collecting data by
following up respondents to thank
them for their input and to share
the results of the survey
 If you intend to survey the same
group several times, this feedback
loop is particularly important and
may influence the response rate


Getting the most out of your data
 Don‟t just file the survey data away until the next activity
report for your funding provider is due!



Health promotion short course –
example survey data
How confident do you
feel to undertake a needs
assessment to plan a
health promotion
program?

How confident do you feel to
contribute to the design and
implementation of an
evaluation of a health
promotion initiative?


Pre course

Post course

11

11
5

3

2

0
Not
confident

Confident

Very
confident

Pre course

Not
Confident
Very
confident
Confident

Post course
9

10

3

7
3
0

Not
confident

Confident

Very
Confident


Not
Confident
Very
confident
confident

Health promotion short course –
example survey data
Pre course
How confident do you feel to
work collaboratively with
your community and with
other organisations?

Post course

13

12

4

3
0
Not
confident

0
Confident

Very
Confident

Not
confident

Confident

Very
confident

 What could you conclude from these data?
 What decisions about the course would you make as a
result of these data?
 Are these decisions valid? What other data might you want
to collect?


Any questions?



FOLLOW UP SESSION


Follow up session
SURVEY CLINIC
Tuesday 12th March
2013, 10am – 12 noon
Curtin University (Room TBA)

 In two weeks‟ time!
 Aim to put what you have learned into practice

 Either:
1.

Critically review a survey you have used previously OR

2.

Design a new survey using what you have learned.

 Bring your surveys and any questions to the survey clinic


Workshop evaluation
Please draw around
your hand (or both
hands) and provide
some feedback on
today‟s workshop.



THANKS FOR COMING!

Please don’t forget to complete the workshop evaluation!
Thank you 


How to design an effective survey: The good the bad and the ugly

Recommended

Recommended

More Related Content

Similar to How to design an effective survey: The good the bad and the ugly

Similar to How to design an effective survey: The good the bad and the ugly (20)

Recently uploaded

Recently uploaded (20)

How to design an effective survey: The good the bad and the ugly

Editor's Notes