This document provides an introduction to self-administered mobile surveys. It defines mobile surveys and notes the focus is on self-administered surveys using a mobile phone's browser. Standard question formats that can be used are described including single choice, multiple choice, dropdown menus, text fields, matrices, and voice/image capturing. Usability of these formats was found to generally be good based on participant feedback. The document outlines a series of mobile survey studies conducted from 2008-2011 focusing on measurement and nonresponse issues. Key findings include that enjoyment and image congruence were stronger motivators for participation than costs or opinions of others. Mobile surveys saw faster responses than web with most participation occurring at home
'How mobile research can use consumers' 'lost’ moments' - Valuewait B.V. (Mob...QuestBack AG
Weitere ähnliche Inhalte
Ähnlich wie Workshop: 'Self-administered Mobile Survey Workshop' - Dr Michael Bosnjak, Free University of Bozen-Bolzano (Mobile Research Conference 2011)
Ähnlich wie Workshop: 'Self-administered Mobile Survey Workshop' - Dr Michael Bosnjak, Free University of Bozen-Bolzano (Mobile Research Conference 2011) (20)
Workshop: 'Self-administered Mobile Survey Workshop' - Dr Michael Bosnjak, Free University of Bozen-Bolzano (Mobile Research Conference 2011)
1. Self-Administered
Mobile Surveys
MRC 2011 Workshop (Part 1)
London (UK)
April 18th, 2011
Michael Bosnjak, PhD, Assoc. Prof.
Free University of Bozen-Bolzano, School of Economics and Management
1
2. 2
Self-Administered Mobile Surveys?
Mobile Surveys?
• Definitions of mobile surveys
– Interviewer-administered surveys
• Interviews among mobile phone users
• Interactive voice response surveys among mobile phone
users
– Self-administered surveys
• SMS (text messaging) surveys
• Browser-based surveys on mobile devices (e.g., mobile
phones having mobile Internet-access, Smartphones,
etc.)
• Our focus:
– Self-administered surveys AND
– using a mobile phone AND
– browser-based.
3. 3
Selected Applications
Directly at point of sale:
in shopping malls and At trade fairs
at points of service
At public
Insights In training
from difficult to seminars
venues, such
as concerts reach target
groups, event/
At schoolyards, in incident-based
surveys, In workplaces
universities &
without internet
recreational immediacy access
facilities
En-route En-route
with bus or
train & at B2C B2B with bus or
train & at
the airport the airport
4. Overall Goal
• Providing a very brief introduction into the
methodological foundations of self-administered
mobile surveys (esp. sources of biases known from
survey methodology)
• Summarizing key findings of an own methodological
study series conducted between 2008-2011
• Discussing practical, evidence-based
recommendations (esp. on measurement and
nonresponse issues)
5. Agenda
• Background
– Survey research: Overall aims and scope
– The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)
• Measurement issues
– What can be presented/assessed?
– How usable are mobile question formats?
– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?
• Nonresponse issues
– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?
– Speed of participation?
– Optimal length of mobile surveys?
• Take-home messages and discussion
6. 6
Background: Overall Aim of Surveys
• Measuring ´true scores´, i.e. yielding unbiased
estimates for facts and/or latent variables.
– Examples of factual questions to measure facts:
• Household-level income/expense estimates > Disposable
income
• Behavioral frequency estimates > Behavior
– Examples of indicators supposed to measure latent
variables:
• Evaluative judgments > Attitudes
• Behavioral likelihood scales > Intentions
• Brand/product related attributes > Image
• Sources of errors in surveys:
• Representation-related biases: Coverage, Sampling,
Nonresponse
• Measurement-related biases/errors
7. 7
Background: Total Survey Error
Measurement Representation
Construct Population
Measurement Coverage
Measurement
Inappropriate
operationalization
(range restriction,
Sampling Frame
reliability, validity)
Sampling
Measurement Sample
Inappropriate
implementation into Nonresponse
a specific mode:
Undesired design-
related effects Response Respondents
Representative (valid)
Representative for the
for the construct in Survey estimate population in question?
question?
8. 8
Background: Survey Errors/Biases
• Coverage Error
Members of the target population have no chance of being
selected in the sample (e.g., no access to the Internet,
incomplete lists etc.). Error due to the fact that not every unit
in the population is represented on the frame.
• Sampling Error
... arises from the fact that not all members of the frame
population are measured.
• Nonresponse Error
The responses of people who have not been surveyed are
different from those who actually have participated in a survey.
• Measurement Error
Deviation of the answers of respondents from their true values
on the measure, e.g. due to inappropriate operationalizations
of (latent) constructs, design features and context effects.
9. 9
Mobile Survey Methodology: Study Series
1. Web: Item development: Determinants of the willingess to
Mobile Study I
(1.7.-2.9.08) participate in mobile surveys (Sozioland Web-Panel)
2. Pre-Testing: Expert usability assessment at YOC
3. Web: Determinants of the willingess to participate S4
(YOC Mobile-Panel; 979 panelists, 272 participants)
4. Olympic Games 2008 Mobile Survey
(YOC Mobile-Panel; 979 panelists, 413 participants)
5. Web: Usability of S4 from participants´ perspective
(YOC Mobile-Panel; 413 panelists from S4, 187 completes)
Mobile Study II 6. Mobile survey: Evaluation of last vacation
(29.9.-18.10.09) (Respondi Web-Panel; 3270 panalists, 540 completes)
7. Web: Usability of S6 from participants´ perspective
(Respondi Web-Panel; 540 panelists from S6, 318
completes)
Mobile Study III 8. Usability of voice capturing/recognition technology
(March/April 2011) (presentation of results at tomorrow at MRC 2011, April 19,
2011)
11. Agenda
• Background
– Survey research: Overall aims and scope
– The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)
• Measurement issues
– What can be presented/assessed?
– How usable are mobile question formats?
– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?
• Nonresponse issues
– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?
– Speed of participation?
– Optimal length of mobile surveys?
• Take-home messages and discussion
12. 12
What can be presented/assessed? (I)
Single Multiple Drop-Down
choice choice menu
13. 13
What can be presented/assessed? (II)
Matrix /
Textfield Polarity Voice / image /video
profile capturing
14. 14
How ´usable´ are standard formats?
Subjective Usability Assessment
Observed
Post-hoc survey (Web) one week after mobile survey completion
Indicators for usability score: fluency, simplicity, ease of use
Item- Drop-
NR Out
Single choice
Einfachauswahl untereinander 89,2
Multiple choice 9%
Mehrfachauswahl untereinander 87,3
Fragetyp
Drop-Down menu
Geschlossene Auswahlliste 82,7
Textfield
Textfeld einzeilig 74,7
Voice recognition / capturing ?
45% 9%
Image mit Bild
Fragetyp map 87,9
23%
65,00 73,75 82,50 91,25 100,00
Usability score (Range: 0-100 Punkte)
Sources: MS I and MS II combined
17. 17
GPS positioning: Privacy concerns?
Acceptance of 9%
GPS-Location
among
participants with
an iPhone
(MS II; n=45)
91%
Yes (willing to disclose)
No (not willing to disclose)
18. Agenda
• Background
– Survey research: Overall aims and scope
– The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)
• Measurement issues
– What can be presented/assessed?
– How usable are mobile question formats?
– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?
• Nonresponse issues
– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?
– Speed of participation?
– Optimal length of mobile surveys?
• Take-home messages and discussion
19. 19
Nonparticipation: Industry perceptions?
Mobile Research Barometer 2/2011
• Survey among 327 market researchers about
acceptance and use of mobile surveys in D/A/CH
• Top 3 advantages of mobile surveys:
– 51%: Independence of time/location
– 49%: Context-sensitive, fast surveys
Mobile Research
– 43%: Reachability of hard-to-reach, mobile target groups
Barometer
• Top 3 barriers for mobile surveys:
– 35%: Costs incurred to survey participants (data traffic)
– 35%: Difficulties entering information (esp. open-ended
questions)
– 33%: Software/platform heterogeneity
Februar 2011
21. 21
´True´ Reasons for (Non)Participation I
• What is the influence of the following potential
determinants of the willingness to participate?
1. Attitude towards participating
2. Hedonic aspects (perceived enjoyment)
3. Social aspects (subjective norm)
4. Image and perceived self-congruity
5. Perceived benefits and costs
• Hypothetical model
Extended technology acceptance model
(Venkatesh et al., 2003)
• Prospective study design (MS I)
– S1: Developing and optimizing measurement models
– S3: Assessing all above mentioned determinants
– S4: Olympic games mobile survey (non)participation
22. 22
Bosnjak et al. 357
´True´ Reasons for (Non)Participation II
Highest influences:
> Hedonic aspects
> Self-congruity
Not relevant:
Fit Indices (robust)
SB-Χ²=407; df=296 > Expected costs (!)
p<.05, Χ²/df= 1.37 > Opinions of others
NNFI=.98
RMSEA=.04
(.03-.05)
*std.β, sig. at α=.
05, N= 272
23. 23
´True´ Reasons for (Non)Participation III
• If hedonic factors outperform cost/benefit-related,
then
– ´exciting´ incentives (lottery drawing) should
increase participation rates
– compensation for incurred costs should undermine
hedonic motivation (salience of costs is increased)
• MS I experiment, manipulating basic compensation
(1 EUR, yes/no) and announced prize draw (100 EUR
voucher, yes/no)
• Results confirmed our expectations (see Appendix):
– highest access and participation ratesfor „lottery &
no incentive condition“
24. 24
Speed of participation? (MS I)
Geschwindigkeit des Zugriffs auf Welle 1 und 2
For about 4.5 hours, Mobile
response rates are higher
compared to Web
Kumuliertert prozentualer Anteil
Faster responses for Mobile
compared to Web:
approx. 35% Mobile versus
aaprox. 10 % Web
Stunden seit Einladung
25. 25
Speed of participation? (MS II)
6
5 MS I
3
2
0
8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00
Mean response speed in hours for different contact/invitation time points
(sent out via SMS)
26. 26
Current context of participation? (MS I)
At home 63,79%
In the office / at work 17,24%
„Were have you been
taking part in the In a car 6,90%
survey?“ At the bus or train station 4,31%
(Wave 3; N=116) Using public transport 2,59%
On the move (other reasons) 5,17%
At home busy with my PC 17,35%
Watched TV 14,29%
Worked at home 10,20% "Which activity did you have to
Read 8,16% disrupt to take part in the mobile
In the office / at work 11,22%
survey?/ What have you done in
Preparing / eating a meal 7,14%
that very situation?“
(Wave 3; N=98; open responses)
On the move 8,16%
Nothing disrupted 23,47%
27. 27
Optimal length of mobile surveys? (MS II)
„Do you want to continue answering the
survey mobile or online (in this case you
will get a link via email)?“
Participants: n= 540
100 % 30,0
10,3 %
75 % 22,5 19,8 20,1
19,3
68,9 %
Minnutes
50 % 15,0
89,7 %
25 % 7,5 5,2 5,1 5,2
31,1 %
0% 0
non iPhone iPhone Total iPhone Non iPhone
mobile Part 1: Mobile- Initial Survey
online Part 1 & 2: Mobile Survey
28. Agenda
• Background
– Survey research: Overall aims and scope
– The ´Total Survey Error´ concept
– Factsheet: Mobile Survey Study series (2008-2011)
• Measurement issues
– What can be presented/assessed?
– How usable are mobile question formats?
– Voice capturing/recognition: Why and how?
– Acceptance of GPS positioning?
• Nonresponse issues
– Industry perceptions on mobile survey (non)participation?
– Reasons for (non)participation: What do mobile survey participants tell us?
– ´True´ reasons for (non)participation?
– Speed of participation?
– Optimal length of mobile surveys?
• Take-home messages and discussion
29. 29
Take-Home Messages & Discussion
• ... on mobile survey measurement:
– Most ´classical´ closed-ended item formats can be included and
are in many cases sufficiently usable
– Various measurement options ´beyond´ the usual self-
administered formats do exist (e.g. GPS positioning, multimedia
upload)
– Open-ended text may need to be replaced by voice capturing/
recognition (to be discussed tomorrow)
• ... on mobile survey (non)response:
– Industry perceptions and self-perception of potential mobile survey
participants on the reasons for nonresponse may be misleading
– Most probable motivators: anticipated enjoyment, image
– Boomerang effects for (over-)compensation
– Fast responses, given in various contexts
– ´Optimal length´ may not exist, various factors appear to
influence the willingness to spend time on (mobile) surveys
32. Acceptance and users‘ behavior
Influencing participants‘ behavior: design
Basic compensation (1 €): participation in mobile survey
yes no
Prize draw Prize draw
(100 € Amazon voucher) (100 € Amazon voucher)
yes no yes no
Incentive information (timing)
in the
Group 1 Group 2 Group 3 Group 4
SMS
on the
survey
Group 5 Group 6 Group 7 Group 8
landing
page
33. Acceptance and users‘ behavior
Influencing participants‘ behavior
Basic compensation (1 €): participation in mobile survey
Landing
yes no
page
Prize draw Prize draw
access (100 € Amazon voucher) (100 € Amazon voucher)
yes no yes no
Incentive Information (Timing)
Group 1 Group 2 Group 3 Group 4
in the
SMS 8,9% 17,3% 21,2% 12,3%
on the
survey Groups not relevant, first contact on7landing page 8
Group 5 Group 6 Group Group
landing
page
34. Acceptance and users‘ behavior
Influencing participants‘ behavior
Response rates in wave 2 against time
Response rates maximized
with price draw (group 3),
additional compensation
Cumulated response rate
undermines motivation
(see group 1: 1€ and price
draw).
SMS information
Group 1 (1 EUR + price draw)
Group 2 (1 EUR)
Group 3 (prize draw)
Reminder
Group 4 (no incentive information)
Hours since SMS invitation
35. Acceptance and users‘ behavior
Influencing participants‘ behavior
Basic compensation (1 €): participation in mobile survey
All yes no
questions Prize draw Prize draw
answered (100 € Amazon voucher) (100 € Amazon voucher)
yes no yes no
Incentive Information (Timing)
Group 1 Group 2 Group 3 Group 4
in the
SMS 5,9% 12,8% 14,4% 9,2%
on the Group 5 Group 6 Group 7 Group 8
survey
landing
page
9,8% 9,6% 9,1% 10,5%
36. Nonresponse issues: Background
Why increasing response rates to surveys?
nonresponse
rate
nonresponse true difference
error
Black Box
yr ! = statistic of interest for respondents
yt ! = statistic of the total sample
ynr ! = statistic of interest for nonrespondents 36
38. Nonresponse: Background:
Generic reasons for nonresponse
• Failure to deliver the survey request
• Spam guards
• Unused or infrequently checked e-mail addresses
• Non-availability during fielding period
• Inability to provide the requested data
• Lack of knowledge
• Insufficient information readily available
• Noncompliance: Refusals to survey requests
38
39. Nonresponse: Theory:
Why do people (not) respond to surveys?
• Economic exchange view
• Human needs and values
• Compliance heuristics
• Transactional view
• Planned behavior approach
• Leverage-salience theory
• Social exchange theory
39
40. Nonresponse: Theory:
Why do people (not) respond to surveys?
Rationale:
• Economic exchange view Respondents are motivated by the
monetary benefits promised/
• Human needs and values expected.
Actionable recommendations:
• Compliance heuristics „Pay respondents“ according to the
time/effort invested
• Transactional view
Caveats:
• Planned behavior approach • Peoples´ price points vary greatly
and are unknown a-priori
• May largely increase non-
• Leverage-salience theory response bias
•Undermines intrinsic motivation
• Social exchange theory and may increase measurement
error (low survey involvement)
• Promised monetary incentives
NOT consistently effective (!)
40
41. Nonresponse: Theory:
Why do people (not) respond to surveys?
Rationale:
• Economic exchange view
Some values are
• Human needs and values systematically related to
• Compliance heuristics the propensity to
respond (higher order
• Transactional view needs, civit duty
• Planned behavior approach orientation, etc.)
• Leverage-salience theory Caveats:
• Social exchange theory • Effects small (if any)
• Actionable
recommendations?
41
42. Nonresponse: Theory:
Why do people (not) respond to surveys?
Rationale:
• Economic exchange view Certain aspects of the survey
announcement and survey
• Human needs and values implementation do induce
compliant behavior:
1. Reciprocity
• Compliance heuristics 2. Scarcity
3. Authority
• Transactional view 4. Consistency
5. Consensus
• Planned behavior approach 6. Liking
Actionable recommendations:
• Leverage-salience theory Can be derived from persuasion
literatures, but specific prescriptive
• Social exchange theory models on how to tailor them
toward survey situations are rare.
Groves, Cialdini & Couper (1992); Cialdini (2008);
http://www.influenceatwork.com/ 42
43. Nonresponse: Theory:
Why do people (not) respond to surveys?
Rationale:
• Economic exchange view Larger response propensity
if communication style
• Human needs and values reflects positive regard and
• Compliance heuristics avoids adult-to-child
communication styles.
• Transactional view
Caveats:
• Planned behavior approach • Limited scope
• Leverage-salience theory • Empirical evidence scarce
• Covered by other theories
• Social exchange theory (compliance heuristics,
social exchange)
Comley (2006)
43
44. Nonresponse: Theory:
Why do people (not) respond to surveys?
Rationale:
• Economic exchange view The propensity to respond to
surveys is primarily a function of
• Human needs and values three factors:
• Attitude to participate
• Subjective norms
• Compliance heuristics • Perceived behavioral control
• Moral obligation
• Transactional view
Actionable recommendations:
• Planned behavior approach If weights are known for a specific
population/sample: Enables the
researcher to design survey
• Leverage-salience theory participation requests
• Social exchange theory Caveats:
Restricted to optimize survey
announcements
Bosnjak (2002); Bosnjak, Tuten & Wittmann (2005)
44
45. Nonresponse: Theory:
Why do people (not) respond to surveys?
Rationale:
• Economic exchange view Respondents are differentially
motivated by
• Human needs and values • different aspects of the survey
(leverage, e.g. type of incentives)
and by
• Compliance heuristics • how much emphasis is put on
each aspect by the surveyor
• Transactional view (salience, e.g. preference for certain
incentives )
• Planned behavior approach Actionable recommendations:
Because of the interaction between
• Leverage-salience theory leverage*salience, improving
response rates is not always
• Social exchange theory desirable!
Nonresponse bias may be
influenced by leverage*salience
interaction.
Groves, Singer & Corning (2000)
45
46. Nonresponse: Theory:
Why do people (not) respond to surveys?
Rationale:
• Economic exchange view Survey participation as social
exchange: The likelihood of
• Human needs and values responding is greater when the
respondent trusts that the expected
rewards will outweigh the
• Compliance heuristics anticipated costs of responding.
• Transactional view Actionable recommendations:
Tailored Design Method, a well-
• Planned behavior approach developed set of practical
recommendations on all aspects of
survey design/implementation,
• Leverage-salience theory aimed at:
•establishing trust
• Social exchange theory •increasing participation benefits
•decreasing participation costs
Dillman, Smyth & Christian (2009)
46
47. Nonresponse: Theory:
TDM-based recommendations (selection)
To increase benefits To decrease costs of
To establish trust
of participation participation
•Obtain sponsorship by •Provide information about •Make it convenient to
legitimate authority the survey respond
•Provide a token of •Ask for help or advice •Avoid subordinate language
appreciation in advance •Show positive regard •Make the questionnaire short
•Make the task appear •Say thank you and easy to complete
important •Support group values •Minimize requests to obtain
•Ensure confidentiality and •Give tangible rewards personal or sensitive
security of information •Make the questionnaire information
interesting •Emphasize similarity to other
•Provide social validation requests or tasks to which a
•Inform people that person has responded
opportunities to respond are
limited
Dillman, Smyth & Christian (2009, p. 38) 47
48. Nonresponse: Evidence: Mail surveys:
Effective methods & procedures I
• Most effective factors in mail surveys
(only factors under the researchers full control listed):
• Personalization of requests to participate
(Dillman, 1978, 2000; Edwards et al., 2007; Fox, Crask, & Kim,
1988; Heberlein & Baumgartner, 1978; Yammarino et al.,1991;
Yu & Cooper, 1983)
• Prepaid monetary incentives
(Church, 1993)
• Number of contacts made (esp. if prenotifier is included)
(Armstrong & Lusk, 1987; Edwards et al., 2007; Fox et al., 1988;
Heberlein & Baumgartner, 1978; Yammarino et al.,1991;
Yu & Cooper, 1983)
➡ Integrated and refined within the Total-Design-Method
(Dillman, Smyth & Christian, 2009)
48
49. Nonresponse: Evidence: Mail surveys:
Effective methods & procedures II
• Effective, but not covered because of limited control:
• Survey topic / topic involvement
• Length
• Sponsorship (University / commercial)
• Factors reducing response rates
(1, 2: Edwards et al., 2007; 3: Singer, Hippler & Schwarz, 1992):
1. Starting with the most general question (e.g. demographics)
2. Opportunity to opt-out of the study
3. Over-emphasizing data protection/confidentiality
• Partly covered later for Web surveys:
• Questionnaire design effects on nonresponse
49
50. Nonresponse: Evidence: Web surveys:
Effective methods & procedures III
• Personalization:
• Personal salutation (name) is effective (esp. for powerful sender)
(e.g., Heerwegh, et al., 2005; Joinson & Reips, 2007)
• (Monetary) Incentives:
• In general effective but small overall effect (Göritz, 2006)
• Pre-paid monetary incentives need to be tangible to be effective
(Birnholtz et al., 2004; Bosnjak & Tuten, 2003)
• Lotteries esp. effective, timing important (immediate notification)
(Bosnjak & Tuten, 2003; Tuten, Galesic & Bosnjak, 2004)
• Contact features:
• No of contacts very effective (Cook, Heath & Thompson, 2000)
• SMS prenotifier very effective (Bosnjak et al., 2008)
50