Crowdsourcing platforms are revolutionizing research by providing a way to collect clinical and behavioral data with unprecedented speed and efficiency. This seminar explores another digital platform called TurkPrime that is designed to suuport research participant recruitment. TurkPrime is a relatively new panel service that allows researchers to target specific demographic groups. If you watched our previous webinar on Amazon’s Mechanical Turk, also known as MTurk, you may find it interesting that TurkPrime offers a proportional matching sampling approach rather than MTurk’s opt-in, convenience sampling approach. Tasks that can be implemented with TurkPrime include: excluding participants on the basis of previous participation, longitudinal studies, making changes to a study while it is running, automating the approval process, increasing the speed of data collection, sending bulk e-mails and bonuses, enhancing communication with participants, monitoring dropout and engagement rates, providing enhanced sampling options, and many others.

1. Digital Scholar
Webinar
November 1st, 2017
Hosted by the Southern California Clinical and Translational Science Institute (SC CTSI)
University of Southern California (USC) and Children’s Hospital Los Angeles (CHLA)

2. Katja Reuter, PhD,
Director of the Digital
Scholar Program
About Today’s Session

3. TurkPrime
Web link: https://www.turkprime.com

4. Today’s Learning Objectives
 Understand the methodological and other advantages of using Mechanical
Turk
 Understand how TurkPrime’s Toolkit significantly expands Mechanical
Turk’s functionality
 Understand how PrimePanels helps to overcome the limitations of
Mechanical Turk’s population and sampling
 Describe the tradeoffs of using MTurk, TurkPrime, and PrimePanels

5. Leib Litman, PhD
Today’s Speaker
Topic: Using the research platform TurkPrime to
crowdsource data for the health sciences
Speaker: Leib Litman, PhD,
Associate Professor of Psychology, Lander College
Director of Research, TurkPrime

6. Questions: Please use the Q&A
Feature
1. Click on the tab here to
access Q&A
2. Ask and post question here
1
2

7. Online Data Collection
o Research in numerous scientific fields relies on online sources of
human participants (Bohannon, 2016).
o Mechanical Turk (MTurk) is currently the most commonly used
participant recruitment platform.
o While MTurk is the original and most popular online participant
recruitment platform it has several limitations.
o In this talk we will describe:
– MTurk’s API-based add-on tools available to supplement MTurk
functionality
– Alternative platforms which provide access to larger and more
diverse populations
– Strengths and weaknesses of each

8. Outline of today’s talk
o Describe the advantages of Mechanical Turk
– Sample study: Examining the relationship between the
internalization of media ideals of beauty and disordered eating
– A brief look at the data: Replication and data quality
o Describe the TurkPrime Toolkit
– How to create a TurkPrime account and link it to Mechanical
Turk
– Describe several Toolkit features
o Describe PrimePanels
– Strength of PrimePanels
– Strengths and weaknesses of PrimePanels and MTurk samples
o Not a detail tutorial

9. Mechanical Turk
Technical challenges Sampling challenges

10. Mechanical Turk
Exclusion from previous/simultaneous
studies, longitudinal, communication,
sampling, data quality, anonymity.
Technical challenges

11. TurkPrime Toolkit
Solution
Mechanical Turk
Exclusion from previous/simultaneous
studies, longitudinal, communication,
sampling, data quality, anonymity.
Technical challenges

12. Mechanical Turk Sample study
Suppose a researcher wants to conduct a study examining the relationship
between Thin-ideal internalization and disordered eating.
Research question: Is there a relationship between media ideal
internalization and disordered eating, controlling for covariates such as
depression?
This study addresses issues relevant to various disciplines including clinical,
health and social psychology, sociology, psychiatry, and public health.
Methods are representative of many HITs that are posted on Mechanical Turk
by social and behavioral scientists.

13. Why conduct this study on Mechanical
Turk?
o Mechanical Turk is a convenient and easily accessible source of participants
and can be used to selectively recruit specific populations such as women.
o Once launched, data collection is expected to complete within a few hours.
o The cost is expected to be less than $200.
o The stimuli used in the study can be easily developed online using survey
platforms, such as Qualtrics, and then seamlessly linked to Mechanical Turk.
o Once data are collected a fully labeled csv or SPSS file will be available for
download immediately, saving time over traditional pencil and paper
approaches which involve time consuming manual data entry.

14. Methodological advantages.
o MTurk samples are more representative compared to typical
samples collected from the undergraduate subject pool.
o The sample is expected to be more diverse in terms of age,
race, education, occupation and geographical location.
o Ability to follow up with the sample longitudinally with relatively
low dropout over time.
o Easily conduct follow-up studies to examine how the relationship
between thin- ideal internalization and disordered eating may
interact with factors such as age, education, SES, religious
background and many others.

15. Mechanical Turk is more than a platform for conducting a single
study, but rather an environment on which a phenomenon of
scientific interest can be systematically explored across a series of
studies using a wide range of research methodologies.

16. Basic design and methodology
The study uses survey
instruments to
measure multiple
constructs
Thin ideal
internalization will be
measured with the
SATAQ-3
Disordered eating will
be measured with the
EAT-26.
Depression will be
used as a covariate,
and measured using
the BDI.
We will also ask
various demographic
and personal
information questions.

17. SATAQ – Social attitudes towards
appearance questionnaire
on
Thompson, J. K., van den Berg, P., Roehrig, M., Guarda, A. S., & Heinberg, L.
J. (2004). The sociocultural attitudes towards appearance scale‐3 (SATAQ‐3):
Development and validation. International Journal of Eating Disorders, 35(3),
293-304.

18. EAT– Eating Attitudes Test
Garner, D. M., Olmsted, M. P., Bohr, Y., & Garfinkel, P. E.
(1982). The eating attitudes test: psychometric features and
clinical correlates. Psychological medicine, 12(4), 871-878.

19. BDI– Beck depression Inventory

20. MTurk does not provide sophisticated stimulus
development software. Most studies are designed
on Qualtrics, Survey Monkey Survey Gizmo etc.
o Final block structure in Qualtrics. Creating studies with Qualtrics is not
covered in this webinar, but a Power Point Qualtrics tutorial is available
upon request.

21. Once a link has been generated, it can be sent to study participants.
Clicking on this link or pasting it into the address bar will take a
participant to the survey. This link should be copied and pasted on
MTurk to distribute to MTurk participants.

22. Creating an MTurk account
Click on sign in as a Requester

23. Create Page
o On the left side there are a series of options for different study types. Most of these are
not relevant for psychological research. The Survey Link option is for a study that
allows a Requester to paste a link to a study hosted on a third party platform, such as
Qualtrics.
1.Click Survey Link  2. Click Create Project

24. Enter Properties tab
o Creating a study requires sequentially filling fields for three tabs.
In the Enter Properties tab under Create, fill out basic information about the
study’s specifications.

25. Enter Properties tab
o The drop-down menu lists system (MTurk) qualifications, including
gender.

26. Launching the study
o After a few confirmation screens you will see the option to publish the
study.

27. Monitoring the study
o You are able to see the study’s progress, and average completion time.

28. Brief overview of results
o The full dataset was collected within 3 hours.
o The Cronbach’s alpha reliabilities for the EAT, SATAQ, and BDI were
.75, .89 and .95 respectively and are all well within acceptable norms.
o Disordered eating, thin-ideal internalization, and depressive symptoms
were strongly correlated with each other (see Table 1).
o This short project demonstrates that it is possible to conduct a study
within a matter of hours.

29. Methodologically complex studies are difficult to conduct using
MTurk’s GUI. Here are a few examples of follow-up studies we
may want to conduct.
o Conduct follow-up study with 200 more female workers.
– Methodological requirements – Exclude those who already participated. Sample across
the whole day/week. Pay workers automatically when they provide the correct secret
code, prevent duplicate IP addresses, verify that workers are from the US.
o Longitudinal follow-up with the same participants
– Methodological requirements – Send email invitations. Communication. Provide bonuses
for longitudinal participation. Automatically add worker IDs to the data file in order to
match participants across data sets.
o Diary study to monitor media exposure.
– Methodological requirements – Longitudinal follow up. Communication. Set daily studies
to start automatically at the same time each day.
o Video interviews to hear participant’s experiences with disordered eating
and eating disorders.
– Methodological requirements – Set up Audio/Video HIT to interact with workers. Record
each interview.

30. Mechanical Turk’s Application Programming
Language (API) offers significantly greater
functionality, compared to the GUI.
o Mechanical Turk was designed to provide basic functionality through its
graphical user interface (GUI) but to provide significantly more
functionality through its API.
o Third party platforms that utilize MTurk’s API provide researchers with
powerful tools that can be used together with Mechanical Turk.
o There are now multiple platforms that provide various add-on tools that
facilitate the research process: RTurk, MmmTurkey, Longii, TurkGate
(Goldin & Darlow, 2013), PsiTurk (Gureckis, et al., 2016), TurkServer
(Mao et al., 2014).
o In this webinar we describe TurkPrime (Litman, Robinson and
Abberbrock, 2017).

32. Getting started with TurkPrime’s Toolkit.
Step 1
Create a
TurkPrime
account
Step 2
Link to your
MTurk
Requester
account
Step 3
Design
Survey on
Qualtrics,
SurveyMonke
y, etc
Step 4
Launch HITs
through
TurkPrime
Step 5
HIT Launches
on
Mechanical
Turk
Step 6
Workers
Complete
HIT
Step 7
Monitor
Progress on
TurkPrime’s
Dashboard

33. Creating a Turkprime account
• To set up a TurkPrime account go to: https://www.turkprime.com
• If you don’t have an account you will be asked to sign up and to create a
username and a password
• Once an account is created a video tutorial will be sent to show you how to link
your MTurk account to your Turkprime account. The process takes five minutes.

34. Creating a study
To start creating a study click on Design Study  Basic and Panel Studies

35. Study Design Interface
o The Design page has 9 sequential tabs.
o The first tab has several options for targeted recruitment.
o Can select between video and survey HITs.
o What follows is not a systematic tutorial, but rather a few slides
highlighting some of TurkPrime’s features

36. Initial setup and examples of targeting
criteria to chose from

37. Examples of targeting criteria:
Personality profile

38. Video, focus group, and screen
capture HITs

39. Requester
Worker 1
Worker 1
Worker 2
Add more
workers
Switch
between
audio, video
and text
Video, focus group, and screen
capture HITs

40. Many steps during setup are similar to
MTurk
o The Title, Description, Custom Instructions and Keywords fields are the
same as on MTurk

41. Many steps during setup are similar to
MTurk
o These fields are also the same as on MTurk

42. Examples of features not available on
MTurk
o Delayed launch – schedule a HIT’s time of launch. Useful for diary
studies, researchers from outside of the US, and night owls.

43. Select past studies from a dropdown menu to exclude
workers, or include workers for longitudinal studies.
o No need to set qualifications

44. When studies are added to a group,
workers are excluded from HITs that are
running at the same time.
o Useful for labs where multiple projects are being run at the same
time.

45. • For longitudinal studies, once the study is launched, Workers
receive invitation emails within minutes
• Emails sent from MTurk
• Worker Identities remain hidden

46. To match workers’ IDs across multiple waves, TurkPrime
automatically sends their IDs to the data file – SPSS for
example.

47. Microbatch spreads HITs across time for
enhanced sample representativeness.

48. Worker Groups
o Requesters can create groups such as Republicans and White
Males. These groups are created from the workers who have
previously completed studies with a Requester.
o Worker groups can also be used to create universal exclude lists.

49. Groups can be selected from a
dropdown menu during study setup

50. Additional features

51. Ongoing studies are monitored on the
Dashboard

52. Extended Dashboard view offers more control
over HITS and advanced indicators

53. Mechanical Turk
Small population, skewed population,
Difficult to reach rare groups, difficult to
selectively recruit
Sampling challenges

54. Mechanical Turk
PrimePanels
Solution
Small population, skewed population,
Difficult to reach rare groups, difficult to
selectively recruit
Sampling challenges

55. Limitations of the Mechanical Turk
worker pool
o Small population
o According to MTurk there are 500,000 participants in
it’s Worker pool. The number of active workers is
closer to 25,000 on any given month.
o Market Research platforms from which PrimePanels
draws samples have access to tens of millions of
participants.

56. Naivete: MTurk workers are significantly
more likely to have been exposed to
common manipulations.
Litman et al., (in prep)

57. Representativeness and access: Close to 70% of
MTurk workers are below 40. Very limited access to
60+ age group.
Litman et al., (in prep)

58. Representativeness and access: MTurk workers are
much less religious compared to the general
population, and compared to PrimePanels
Litman et al., (in prep)

59. Representativeness and access: MTurk workers are
much more liberal compared to the general population,
and to PrimePanels.
Litman et al., (in prep)

60. Representativeness and access: MTurk workers are
much more likely to be college educated compared to
the general population, and to PrimePanels.
Litman et al., (in prep)

61. Advantages of market research panels
o Large pool of participants > 20,000,00
o More representative of the US population
o Worldwide
o Ability to selectively target very specific populations
o Little exposure to psychological manipulations

62. Examples of selective targeting studies
run on PrimePanels that could not be run
on MTurk

63. Examples of studies run on
PrimePanels that could not be run on
MTurk

64. Creating a PrimePanels study
To start creating a study click on Design Study  PrimePanels

65. Initial setup – similar to MTurk

66. Multiple targeting variables to
choose from

67. Census template automatically matches the
sample to the US Census along multiple
variables

68. Numerous variables can be added and
percentages can be adjusted within each
bin

69. Tradeoffs between MTurk and
PrimePanels
o PrimePanels studies should be under 20 minutes
o MTurk data quality for open ended responses is much better
o MTurk is better for tasks requiring sustained and effortful attention
over long time periods (e.g. website testing, aptitude tests)
o PrimePanels data is more representative
o PrimePanels has a larger, global pool of participants who have not
been exposed to common protocols
o PrimePanels has more sophisticated targeted recruitment
capabilities

70. Thank you!
For inquiries about Mechanical Turk, TurkPrime or Prime Panels please contact
Leib Litman, PhD
TurkPrime Director of Research
leib.litman@turkprime.com
Or
Jon Robinson, PhD
Chief Technology Officer
Jonathan.robinson@turkprime.com

71. Q u e s t i o n s
Program director: Katja Reuter, PhD
Email: katja.reuter@usc.edu
Twitter: @dmsci
Next Digital Scholar Webinar
I n f o r m a t i o n a b o u t
t h e p r o g r a m
http://sc-ctsi.org/digital-scholar/
Dec 6, 2017 | 12-1PM PST
Topic: Accelerating systematic review studies using the online
tool Covidence
Speaker: Anneliese Arno, community manager at Covidence
product@covidence.org
Register at: sc-ctsi.org/digital-scholar/register