Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Does Social Psychology Really Have More Retractions?
Similar to Does Social Psychology Really Have More Retractions? (20)
Recently uploaded
Recently uploaded (20)
Does Social Psychology Really Have More Retractions?
- 1. Hot Topic: Assuring the Quality of Psychological Research Research Misconduct and the Development of Article Retractions in Psychology and its Fields Armin Günther 50. Kongress der Deutschen Gesellschaft für Psychologie Universität Leipzig, Leipzig, Germany 18.–22. Sept. 2016 Leibniz Institute for Psychology Information (ZPID) Trier, Germany Research misconduct and the development of article retractions in Psychology and its fields by Armin Günther is licensed under a Creative Commons Attribution 4.0 International License.
- 2. Retracted publications per 10,000 published publications by year in PubMed and PsycINFO. PubMed data by Saunders, N. (13.9.2016). PubMed retractions report. Retrieved from https://rpubs.com/neilfws/65778 Increasing retraction rates
- 3. Retracted publications per 10,000 published publications by year in PubMed and PsycINFO. PubMed data by Saunders, N. (13.9.2016). PubMed retractions report. Retrieved from https://rpubs.com/neilfws/65778 Increasing retraction rates
- 4. Why does this happen? Two explanations: 1. Declining quality of published articles (cf. Fanelli 2013: “growing misconduct hypothesis”) 2. Increasing sensibility of scholarly communication system (cf. Fanelli 2013: “stronger system hypothesis”)
- 5. Stefanie Kara (7.5.2016). Zu schön, um wahr zu sein. ZEIT-Online, retrieved from http://www.zeit.de/2015/17/sozialpsychologie-professor-daten-manipulation Are some fields of psychological research more affected than other? • Is Social Psychology affected more by research misconduct?
- 6. Enserink, M. (28.11.2012). Final report: Stapel affair points to bigger problems in social psychology. Science, retrieved from http://www.sciencemag.org/news/2012/11/final-report-stapel-affair- points-bigger-problems-social-psychology Are some fields of psychological research more affected than other? • Is Social Psychology affected more by research misconduct?
- 8. Method: Variables Source: JPSP 103, 605. Reason for retraction: …… Accused author: …… Subject field(s): …… Year of publication: 2012 Year of retraction: 2012
- 9. Reason for retraction: …… Accused author: …… Subject field(s): …… Year of publication: 2012 Year of retraction: 2012 Method: Variables Source: JPSP 103, 605. Reasons for article retractions 1 Fraud Data fraud; data falsification; biasing design 2 Plagiarism Plagiarism; self-plagiarism, duplicate publication 3 Other misconduct e.g., authorship issues; legal issues etc. 4 „Error“ Honest error; dubious error (maybe unproven misconduct) 5 Publisher error e.g., article published in wrong issue or wrong journal 6 Other reasons Not matching any other category
- 10. Method: Variables Reason for retraction: Fraud Accused author: …… Subject field(s): …… Year of publication: 2012 Year of retraction: 2012 Source: JPSP 103, 605.
- 11. Method: Variables Source: JPSP 103, 605. Reason for retraction: Fraud Accused author: Smeesters, Dirk Subject field(s): …… Year of publication: 2012 Year of retraction: 2012
- 12. Reason for retraction: Fraud Accused author: Smeesters, Dirk Subject field(s): …… Year of publication: 2012 Year of retraction: 2012 Method: Variables Source: JPSP 103, 605. PsycINFO content classification 21** General Psychology 22** Psychometrics & Statistics & Methodology 23** Human Experimental Psychology 24** Animal Experimental & Comparative Psychology 25** Physiological Psychology & Neuroscience 26** Psychology & The Humanities 27** Communication Systems 28** Developmental Psychology 29** Social Processes & Social Issues 30** Social Psychology 31** Personality Psychology 32** Psychological & Physical Disorders 33** Health & Mental Health Treatment & Prevention 34** Professional Psychological & Health Personnel Issues 35** Educational Psychology 36** Industrial & Organizational Psychology 37** Sport Psychology & Leisure 38** Military Psychology 39** Consumer Psychology 40** Engineering & Environmental Psychology 41** Intelligent Systems 42** Forensic Psychology & Legal Issues
- 13. Method: Variables Source: JPSP 103, 605. Reason for retraction: Fraud Accused author: Smeesters, Dirk Subject field(s): Personality Psychology [31** ] Year of publication: 2012 Year of retraction: 2012
- 14. Results: Development of retractions and reasons for retractions
- 15. How shape authors with very high numbers of retractions (outliers) the overall picture? D. Stapel Most authors have one, nearly all less than five articles retracted, one author (D. Stapel) more than 50, accounting for more than 20% of all retractions because of misconduct in the data. (Base: PsycINFO) 0 20 40 60 80 100 120 1 2 3 4 5 6 7 8 9 10 15 20 25 30 35 40 45 50 Number of authors with retractions Number of retracted articles per author
- 16. Results: Types of research misconduct
- 17. 1 5 15 3 41 0 3 7 9 34 18 41 36 4 7 11 3 0 9 7 3 1 General Psychology Psychometrics & Statistics & Methodology Human Experimental Psychology Animal Experimental & Comparative Psychology Physiological Psychology & Neuroscience Psychology & The Humanities Communication Systems Developmental Psychology Social Processes & Social Issues Social Psychology Personality Psychology Psychological & Physical Disorders Health & Mental Health Treatment & Prevention Professional Psychological & Health Personnel Issues Educational Psychology Industrial & Organizational Psychology Sport Psychology & Leisure Military Psychology Consumer Psychology Engineering & Environmental Psychology Intelligent Systems Forensic Psychology & Legal Issues 0 5 10 15 20 25 30 35 40 45 Results: Research misconduct in different fields of psychology Number of retracted articles
- 18. 1 5 15 3 41 0 3 7 9 34 18 41 36 4 7 11 3 0 9 7 3 1 General Psychology Psychometrics & Statistics & Methodology Human Experimental Psychology Animal Experimental & Comparative Psychology Physiological Psychology & Neuroscience Psychology & The Humanities Communication Systems Developmental Psychology Social Processes & Social Issues Social Psychology Personality Psychology Psychological & Physical Disorders Health & Mental Health Treatment & Prevention Professional Psychological & Health Personnel Issues Educational Psychology Industrial & Organizational Psychology Sport Psychology & Leisure Military Psychology Consumer Psychology Engineering & Environmental Psychology Intelligent Systems Forensic Psychology & Legal Issues 0 5 10 15 20 25 30 35 40 45 1 4 8 3 36 0 3 5 8 5 4 37 29 4 7 8 3 0 6 5 3 1 General Psychology Psychometrics & Statistics & Methodology Human Experimental Psychology Animal Experimental & Comparative Psychology Physiological Psychology & Neuroscience Psychology & The Humanities Communication Systems Developmental Psychology Social Processes & Social Issues Social Psychology Personality Psychology Psychological & Physical Disorders Health & Mental Health Treatment & Prevention Professional Psychological & Health Personnel Issues Educational Psychology Industrial & Organizational Psychology Sport Psychology & Leisure Military Psychology Consumer Psychology Engineering & Environmental Psychology Intelligent Systems Forensic Psychology & Legal Issues 0 5 10 15 20 25 30 35 40 45 Results: Research misconduct in different fields of psychology Number of retracted authors
- 19. Results: Research misconduct in different fields of psychology
- 20. Results: Research misconduct in different fields of psychology
- 21. Results: Concluding remarks • Are article retractions a useful tool for assuring the quality of psychological research? No. Retractions mostly indicate, that processes of quality control have failed. • Do retractions destroy knowledge and the advancement of knowledge? Generally not. We constantly re-build our knowledge in the light of new (positive or negative) evidence, For this, we need procedures and intelligent tools to update our knowledgebase. • Why do retractions matter at all? The real problem with retractions is not, that single research results may be invalidated. The real problem is that – if retractions are based on research misconduct – they may undermine trust in the general reliability and integrity of research, which is fundamental for building scientific knowledge.
- 22. References • Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), e5738. doi:10.1371/journal.pone.0005738 • Fanelli, D. (2013). Why growing retractions are (mostly) a good sign. PLoS Med, 10(12), e1001563. doi:10.1371/journal.pmed.1001563
- 23. Contact: Armin Günther Leibniz Institute for Psychology Information (ZPID) armin.guenther@zpid.de
Editor's Notes
- During the last two decades there has been a marked increase in the number of retracted journal articles. This it true not only for psychology but across disciplines. My first slide shows the timeline for the number of retracted articles as recorded in PubMed. Each bar represents the number of retractions per 10,000 published articles. The increase in retraction rates is quite obvious.
- PubMed represents the biomedical literature. Comparable data focused on psychology can be obtained from the PsycINFO database. There we get roughly the same picture: increasing retraction rates.
- This observation rise several questions. First of all: Why does this happen? Basically, there are two competing explanations for this development: The first explanation assumes that the quality of published articles declines, leading to the detection of more and more serious problems even after an article is published. A more specific version of this hypothesis is that the decline of quality is mainly caused by an increase in research misconduct. The second explanation states that the mechanisms of retracting articles have changed: Articles are checked more rigorously. They are seen by a larger audience. Publisher standards have become more strict. New tools, for example plagiarism detection software, are available. And so on. I will not try to defend one of these explanations here. Instead, I will turn to another question, which might be of specific interest for the psychology community.
- Are there differences between fields of psychology with regard to research misconduct, as reflected in article retractions? Especially: Are there any indications, that Social Psychology is affected more by scientific misconduct than other fields of psychology? As you know, there have been some highly visible and broadly discussed cases of social psychologist, accused of data manipulating and fabricating their research results. In consequence, the whole field of Social Psychology has been suspected to be a playground for bad research – as this story, published last year suggests.
- Most spectacular however was the case of social psychologist Diederik Stapel, who admitted to have faked numerous studies. More than fifty of his articles were retracted after this got uncovered. Clearly, this case highlighted severe problems to assure the quality of research. And again this was linked to the field of Social Psychology: “Stapel Affair Points to Bigger Problems in Social Psychology” Science magazine titled. So, can we find indications in our data, that Social Psychology is indeed more affected by scientific misconduct?
- METHOD Now here is how I collect data about article retractions in psychology. As my basic sample of psychological publications I selected the publications included in the PsycINFO database. To identify retractions within this sample I basically searched for the string „retract“ within the title and abstract field of PsycINFO. Thus, either the article itself or the published retraction notice or both had to be included in the PsycINFO database to be considered in the analysis. This procedure ultimately resulted in 350 cases of retractions.
- In addition to the PsycINFO metadata, I gathered some more variables for each retraction: As an example I take the retraction of an article by Johnson, Smeesters and Wheeler, published and retracted 2012.
- First I wanted to know, why this articles was retracted. For an answer I retrieved the original retraction notice. Based on these retraction notices (and sometimes on additional external information) all retractions were classified into 6 main categories: Articles are retracted because of Fraud - including manipulating and fabricating data, as well as biasing research design Plagiarism, including self-plagiarism and redundant publication other misconduct, e.g. authorship issues These three categories together build the category “research misconduct”. Other retraction reasons are „errors“ by authors, comprising honest errors as well as more dubious forms of errors Sometimes, articles are retracted because of publisher errors, for example if an article is publishing in a wrong issue or even a wrong journal The last category is for all reasons, that do not fit in any other.
- For the example, the retraction is easily classified as “Fraud”. The retraction note states that Smeesters “removed data in order to achieve a significant outcome”.
- Next, I tried to identify the responsible or accused author. This is easy in our example, as Smeesters is clearly identified as the accused author in the retraction notice. Additionally he is described as the sole responsible authors: “His co-workers”, the notice states, “were unaware of his actions”.
- Finally, each retracted article was assign to one or more research field. Here I simply used the content classification at its PsycINFO record. Thus, every article was assigned to one or more of these 22 main PsychINFO content categories.
- For our example this content category was “Personality Psychology”.
- RESULTS Now, here are some results. This chart shows once more the development of retractions rates, now differentiated by reasons for retraction. Retraction rate increased from less than 1 to about 3 per 10,000 published articles in the last two decades. Overall, about two third of all article were retractions because of some form of research misconduct. And about one quarter by some form of (honest or not so honest) “Error”.
- However, the variable “numbers of retracted articles” needs to be interpreted with care: Authors with very high numbers (like D. Stapel) of retractions may affect this variable considerably. If we just count retracted articles we might get biased or misleading results.
- If we look more closely at research misconduct we see, that the dominant form is “Fraud”. However, this result is biased by the inclusion of over 50 papers by Diederik Stapel in this Fraud category. Therefore, I also counted how many different authors were accused for the different forms of research misconduct. Now, on the basis of “unique authors” (per category), Plagiarism and not Fraud clearly was the dominant type of misconduct.
- The next chart shows the distribution of retracted articles over fields of research. The areas with highest absolute numbers of retracted articles are: Physiological Psychology & Neuroscience Social Psychology Health & Mental Health Treatment & Prevention Psychological & Physical Disorders
- However, if we count unique responsible authors (per field) the picture changes especially for Social Psychology: In fact, only papers of 5 different authors were retracted in the field of social psychology, but for example 37 in the field of Psychological & Physical Disorders 36 in the field of Physiological Psychology & Neuroscience, and 29 in the field of Health & Mental Health Treatment & Prevention However, these are absolute numbers and the high number of retracted authors in some fields may just reflect the amount of publications (and thus authors) in these fields. This is confirmed by the next chart.
- Here the number of retracted articles within fields of research is plotted against the size of these fields, that is the total number of articles assigned to a field. Obviously, the two variables are correlated. However, Social Psychology is clearly overrepresented relative to the size of the field.
- However, this picture changes, if we do not count retracted articles but retracted authors – thus mainly considering the bias resulting from D. Stapel, who mainly published in the fields of Social Psychology and Personality Psychology. In this chart the number of unique authors retracted within fields of research is plotted against the size of these fields. Now we see is a nearly perfect correlation. Especially Social Psychology nearly perfectly matches our expectation based on the relative size of the field. This is equally true for most of the other fields, with one exception: “Physiological Psychology & Neuroscience”. In this field significantly more authors are retracted than would be expected by the size of the field alone. So, if any field at all looks suspicious by this analysis, it‘s not Social Psychology but Physiological Psychology & Neuroscience. This might be not such surprising. There is a high overlap between this field of psychology and biosciences. And there are some indications that retraction rates in biosciences might be higher than in other disciplines. Additionally, in a meta-analysis pubished 2009 Fanelli found that „misconduct was reported more frequently by medical/pharmacological researchers than others”.
- Let me end with some general remarks. Are article retractions a useful tool for assuring the quality of psychological research? I do not think so. Retractions currently are not the result of a systematic process of research evaluation and quality control but mostly the result of chance, good or bad luck and whistleblowing. Retractions mostly come too late, they indicate, that processes of quality control have failed. Is the growing number of retractions a thread for the advancement of scientific knowledge? With regard to knowledge building, retractions are like failures to replicate a published result or even like unexpected new evidence. Knowledge is always dynamic and constantly re-build, this should not be a fundamental threat for science on the whole (but a problem for individual papers by other researchers, relying on retracted evidence). We always need procedures to dynamically update our knowledgebase – whether we look at replications, retractions or simply new evidence. The real problem with retractions is not, that particular findings may be invalidated. The real problem is that – if these invalidation is based on research misconduct – they may undermine trust in the general reliability and integrity of research, which is fundamental for building scientific knowledge.