Describes the nuts and bolts of designing a survey for research in the social sciences.

1. Survey Design: Introduction & Overview Dr. James Neill Centre for Applied Psychology University of Canberra 28 September, 2009

3. Questionnaire design

4. Types of questions

5. Response formats

6. Sampling

8. To consider implementation issues (such as sampling)

9. To demonstrate the importance of rigour in planning, developing, and implementing research questionnaires

11. Readings (Wikiversity)

12. Books Look for books about surveys design and survey research in libraries.

13. Research process

14. What is a survey? A “standardised” stimulus A measuring instrument

15. What is a survey? A way of converting “fuzzy stuff” into hard data for analysis

19. Linked to a hypothesis/research objective

21. Useful for marketing or assessing consumer behavior

22. Honours-MA-Ph.D survey research

23. Types of Questionnaires Self - administered Interview - administered Postal questionnaire Delivery and collection questionnaire Telephone survey Face to face structured interview Web-based

24. Designing a survey

25. Questionnaire planning/design 1. Formulate Generic Questionnaire 2. Expand the Questionnaire Based on study objectives Turn into separate sections Question styles & types 3. Finalise Questionnaire -Pre-test/pilot test -Several drafts needed Placement & Funnel Qs

27. Ensure all questions relate to research objectives

28. For explanatory objectives or hypotheses ensure both dependent and independent variables exist

30. What's involved?

31. Explain any risks/costs/rewards

32. Contact details

33. Human Ethics approval #

34. How is consent given/not give?

35. How to return?

36. Can choose not to continue anytime

38. Explain how to do the survey in a user-friendly manner

39. Example: Life Effectiveness Questionnaire

40. Expanding the survey

42. Ask screening questions first, rather than later

43. Use branching if there are conditional questions

45. Use funnel questions to move respondents through survey

46. Start off with easy to answer and engaging questions

47. More controversial questions in middle section

48. Personal questions – start or end?

49. Survey design principles Jenkins and Dillman (1995) 1. Use the visual elements of brightness, color, shape, and location in a consistent manner to define the desired navigational path for respondents to follow when answering the questionnaire.

50. Survey design principles Jenkins and Dillman (1995) 2. When established format conventions are changed in the midst of a questionnaire use prominent visual guides to redirect respondents.

51. Survey design principles Jenkins and Dillman (1995) 3. Place directions where they are to be used and where they can be seen. 4. Present information in a manner that does not require respondents to connect information from separate locations in order to comprehend it.

52. Types of questions

53. Types of questions Be able to justify and defend your choices ...

55. Useful for descriptive, exploratory work

56. Difficult and subjective to analyse

57. Time consuming

59. How many hours did you spend studying this week? _________

61. Useful for hypothesis testing

62. Easy and objective to analyse

63. Time-efficient

65. Multichotomous questions

66. The list (multiple response)

67. Ranking

68. Likert Scale

69. Graphical Scale

70. Semantic Differential

71. Non-verbal (Idiographic)

72. Dichotomous Simple Yes / No response e.g., Excluding this trip, have you visited Canberra in the previous five years? __ Yes __ No

73. Multichotomous Choice one of several answers e.g. How many hours did you spend studying this week? __ less than 5 hours __ > 5 to 10 hours __ > 10 to 20 hours __ more than 20 hours

74. Frequency scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment

75. The list (Multiple response) Provides a list of answers for respondents to choose from e.g., Tick any words or phrases that describe your perception of Canberra as a travel destination: __ Exciting __ Important __ Boring __ Enjoyable __ Interesting __ Historical

76. Ranking Helps to measure the relative importance of several items Rank the importance of these reasons for taking a holiday to Canberra (from 1 (most) to 4 (least)): __ to visit friends and relatives __ for business __ for educational purposes __ for holiday/ sightseeing

77. Likert Scale Assesses differences of perceptions and allows measurement and comparison of these differences Indicate your degree of agreement with this statement: “ I am an adventurous person.” (circle the best response for you)

78. Graphical rating scale How would you rate your enjoyment of the movie you just saw? Mark with a cross (X) not enjoyable very enjoyable

79. Semantic differential What is your view of smoking? Tick to show your opinion. Bad ___:___:___:___:___:___:___ Good Strong ___:___:___:___:___:___:___ Weak Masculine ___:___:___:___:___:___:___ Feminine Unattractive ___:___:___:___:___:___:___ Attractive Passive ___:___:___:___:___:___:___ Active

80. Non-verbal (Idiographic) Scale Point to the face that shows how you feel about what happened to the toy.

82. Watch out for too few or too many options

84. Average = 3 to 7

85. Maximum = 10? Scale of measurement guidelines

86. FEELING ABOUT SOMETHING EXTREMELY POSITIVE EXTREMELY NEGATIVE 2-Categories GOOD NOT GOOD 3-Categories GOOD FAIR POOR 4-Categories VERY GOOD GOOD FAIR POOR 5-Categories EXCELLENT VERY GOOD GOOD FAIR POOR

87. Watch out for too many or too few responses “ Capital punishment should be reintroduced for serious crimes” 1 = Agree 2 = Disagree 1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral

89. Is the question stated as briefly as it can be? (If too long, restate it more briefly.)

91. Use only core vocabulary - words and phrases people use in casual speech

93. Use simple sentences where possible and complex sentences only when actually required

94. Use two or more short, simple sentences rather than one compound or complex sentences

96. Only ask questions that relate to objectives

97. Tricks? Font size/double sided photocopying/numbering sections

99. Pilot test – try out on a small sample from the target population & revise

100. Be assertive and interactive about seeking feedback – ask questions & observe

101. “The customer is always right.”

103. Respondent’s level of interest

104. Colour of paper

105. Accompanying letter / introduction

106. Mail surveys - self-addressed stamped return envelope

107. Rewards

108. Reminders or follow up calls

110. What is wrong with the questions, if anything?

111. Example 1 How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over

112. Example 2 Are you satisfied with your marriage and your job? __________________________

113. Example 3 You didn’t think the food was very good, did you? _____ Yes _____ No

114. Example 4 Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No

115. Example 5 What information sources did you use to locate your restaurant for today’s meal? (please tick appropriate spaces) ____ Yellow pages ____ Internet ____ Word of mouth

118. redundancy

119. skewed response items

120. misinterpreted items

123. Consider order effects, habituation, fatigue, switching between response formats

125. Demographics - usually beginning or end; only use relevant questions

126. Space for comments?

127. Ending – say thanks!

128. Pre-test & revise/refine

129. Implementing surveys

130. Comparison of Data Collection Methods Alreck and Settle (1995:32)

131. Sampling

133. Sampling Frame

134. Sample

135. Representativeness

138. If the sample is representative, allows inferences to be drawn concerning the total population

139. What is sampling? “ Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” - Trochim, 2002

142. Membership Lists (organisations, graduates association)

143. Telephone Book

144. Members of Specific Groups or Clubs (Fishing, Ramblers)

145. Households or post codes

147. Selection strategy

148. Adequacy of sample size

149. Response rate – both the % & representativeness of people in sample who actually complete survey

150. Note: It is better to have a small, good sample than a large, poor sample.

151. Sampling example: Shere Hite ‘American Sexology’

153. Media furors & worldwide attention

154. 127-item questionnaire about marriage & relations between sexes

155. 4500 USA women, 14 to 85 years

156. Society and men need to change to improve lives of women

158. 87% had a closer female friend than husband

159. 98% wanted “basic changes” to love relationships

160. only 13% married for 2+years were still in love

161. 84% were emotionally unsatisfied

162. 95% reported emotional & psychological harassment from their men

164. “The survey often seems merely to provide an occasion for the author’s own male-bashing diatribes.”

165. “Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.”

166. Response rate & Selection bias - 1 100,000 questionnaires Sent to a variety of women’s groups - feminist organisations, church groups, garden clubs, etc. 4,500 replied (4.5% return rate)

167. “ We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.” - Regina Herzog, University of Michigan Institute for Social Research Response rate & Selection bias - 2

169. Systematic

171. Convenience

172. Snowball

174. Selection occurs entirely by random chance

175. Often called representative sampling

177. Useful if clear study area or population is identified

178. Similar to a lottery

179. List of names are assigned #s and randomly select #s of respondents

180. Randomly select # through table of random #s or by computer

182. Respondents (units) selected from a list/file.

183. Useful when survey population is similar e.g. List of Students, List of Package Tourists

184. Select sample at regular intervals from the population e.g., every 5 th person on a list

186. As then 4 people out of 5 stand no chance of being selected

187. Select a random starting point between 1 and 5

189. Then random selection from within each stratum

190. Improves representativeness

191. e.g., Telephone interviews using post-code strata

193. Useful for exploratory research and case study research

194. Able to get large sample size quickly and useful when can’t find a sample frame

196. Limitations include potential bias and applicability

198. Due to time/financial constraints

199. e.g. surveying all those at a tourist attraction over one weekend

201. e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra

202. e.g., Frequent flyers to contact regarding service quality in an airline setting

204. Respondents recommend other respondents

205. e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents

207. Selection sampling method

208. Calculate required sample size

209. Maximise return rate

211. How might you get hold of a sample frame?

212. What sampling technique would you use?