1. Experimental Sabotage: Faking in a Subject Pool Study
Divina Wiley and Darryl B. Hill, Ph.D.
Macaulay Honors College, City University of New York
Introduction
As researchers increasingly rely on online
subject pool research as a cost effective and
convenient source of data collection, they
need to be careful to ensure the integrity of
their studies. Two classic threats to internal
validity are the failure to achieve
experimental manipulation and participant
faking (Campbell & Stanley, 1963). Thus,
researchers sometimes deploy manipulation
checks to determine whether the
independent variable had its intended effect
on the participants (Cozby, 2009). Careful
researchers also check for faking by
examining the data for common faking
patterns (completing survey too quickly, all
neutral responses, etc.).
Results
After completing an analysis on responses that were within +/- 25%
on the manipulation check, we found that an alarming 51% (n = 153)
of participants were eliminated from the study: 123 participants
failed the manipulation check, and 30 participants displayed signs of
faking (i.e. completed the survey in less than 5 minutes, entered all
neutral responses, scores on the scales were extreme outliers, etc.).
Method
This study tested whether students in an on-
line experiment and survey would pass a
manipulation check and a faking check. Two
hundred forty-four undergraduate students
were asked to read one of four passages
describing people, complete a task designed
to assess whether the experimental
manipulation was successful, and then
complete a number of questionnaires.
Conclusion
Based on these results, researchers should be
very wary of data from subject pool on-line
experiments and surveys, and are encouraged to
consider implementation of manipulation, faking
and lie checks, and enhancing experimental
realism. Since the data of 153 participants was
disregarded, due to their failure to pass the
manipulation and faking checks, perhaps we
could have enhanced experimental realism by
being more explicit within our survey items.
Participants may have had a better understanding
of the survey items had we been more descriptive.
Failed
manipulation
check
51%
Failed
faking
check
12%
Final sample
37%
Survey Respondents
Participants
One hundred seventy-five female, sixty-eight male, and one
undergraduate student of unknown sex (n = 244) enrolled as PSY
100 students at the College of Staten Island served as participants
in this study. The age of the participants ranged from 18 to 45 years
of age (M = 20.1; n = 244).
Figure 1. Survey respondents in relation to their ability to pass the
manipulation and faking checks.
References
Campbell, D. T., & Stanley, J. C. (1963).
Experimental and quasi-experimental designs
for research on teaching. In N. L. Gage (Ed.),
Handbook of research on teaching (pp. 171–246).
Chicago, IL: Rand McNally.
Cozby, P. C. (2009). Methods of Behavioral
Research: Tenth Edition. New York, NY: McGraw-
Hill.