Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Multi-Stage Open Peer Review
Integrating the Strengths of Traditional Peer Review
with the Virtues of Transparency & Self-...
Outline
Introduction
 motivation & challenges
Multi-Stage Open Peer Review and
Interactive Open Access Publishing
 conce...
Motivation for Open Access & Public Discussion
Scientific, educational & economic advantages of free online
availability &...
Open Access & Quality Assurance
Open Access is not a threat to scientific quality assurance
but an urgently needed opportu...
Quality Assurance Problems I
Tip of the Iceberg: fraud
 plagiarism & selective omission, tuning or fabrication of results...
Quality Assurance Problems II
Editors & Referees: limited capacities
 few editors for large subject areas  limited knowl...
Dilemma: Speed vs. Quality
Rapid Publication: widely pursued
 required for efficient exchange (new findings, critical que...
Solution: Speed & Quality
Stage 1: Rapid publication of Discussion Paper
pre-selected by editors (optionally supported by ...
Outline
Introduction
 motivation & challenges
Multi-Stage Open Peer Review and
Interactive Open Access Publishing
 conce...
Interactive OA Publishing with Multi-Stage Peer Review
1. Pre-publication
review & selection
short term
Discussion Forum (...
Advantages
All-win situation: authors, referees, editors, readers, community
Discussion Paper
 free speech, rapid publica...
ACP Online Library “Most Commented Papers”:
www.atmos-chem-phys.net/most_commented.html
Hansen et al. 2016: climate, 110 c...
ACP Publication & Discussion Statistics
Discussion Papers (ACPD)
 submissions (increasing): ~ 100 month-1 (US, D, UK, F, ...
Atmospheric Chemistry & Physics (ACP)
launched 2001 with Nobel laureate P. Crutzen &
European Geosciences Union (EGU)
15 E...
Conclusions from ACP/EGU
ACP/EGU interactive open access journals demonstrate that:
1) Strengths of traditional publishing...
Alternative Concepts
Open Peer Review w/o Anonymity
 e.g. J. of Interactive Media in Education, BMC Biology Direct, Brit....
Future Perspectives of Multi-Stage Open Peer Review
Combine & integrate with
 repositories (arXiv.org …): SciPost/arXiv
...
Multi-Stage Open Peer Review
QAIntensity/Level
Discussion Paper
ACPD, Economics
Preprint/e-print
(arXiv)
Manuscript
(autho...
Provide access to high quality scientific publications
review & revision involving the community
 more & better informati...
Propositions
1) Continue & promote experiments with improved forms of peer review
 build on existing models & experience ...
Outside Perspectives on ACP/EGU
Ho et al., Views on the peer review system of biomedical journals,
BMC Med. Res. Method., ...
Further References I
The following references and links provide orientation about the development and perspectives of open...
Further References II
2.3. A Short History of Interactive Open Access Publishing
http://publications.copernicus.org/A_shor...
Ulrich Pöschl (MPIC/ ACP) -- Multi-stage open peer review integrating the strengths of traditional peer review with the vi...
Nächste SlideShare
Wird geladen in …5
×

Ulrich Pöschl (MPIC/ ACP) -- Multi-stage open peer review integrating the strengths of traditional peer review with the virtues of transparency and self-regulation

507 Aufrufe

Veröffentlicht am

Presentation of Ulrich Pöschl from the Max Planck Institute for Chemistry, Mainz/Germany and ACP (Atmospheric Chemistry and Physics) on the OpenAIRE workshop "Open Peer Review: Models, Benefits and Limitations"

Veröffentlicht in: Wissenschaft
  • Als Erste(r) kommentieren

Ulrich Pöschl (MPIC/ ACP) -- Multi-stage open peer review integrating the strengths of traditional peer review with the virtues of transparency and self-regulation

  1. 1. Multi-Stage Open Peer Review Integrating the Strengths of Traditional Peer Review with the Virtues of Transparency & Self-Regulation Ulrich Pöschl Max Planck Institute for Chemistry Mainz, Germany u.poschl@mpic.de
  2. 2. Outline Introduction  motivation & challenges Multi-Stage Open Peer Review and Interactive Open Access Publishing  concepts & effects  achievements & statistics (ACP, EGU) Conclusions  vision & propositions
  3. 3. Motivation for Open Access & Public Discussion Scientific, educational & economic advantages of free online availability & usability of scholarly research publications & comments Educational:  inform & stimulate interested public (school teachers, students, et al.)  equal opportunities in the information society (global & social)  re-integrate scholarly & common knowledge (wikipedia, etc.) Economic:  liberate distorted scientific information market (cost/benefit, copyright, etc.)  facilitate technological applications & innovations (text mining by SME, etc.) Scholarly:  enhance interdisciplinary exchange & collaboration  foster discussion & peer review: public comments, ratings, recommendations, etc.  advance evaluation & quality assurance: machine-reading & statistics, transparency & new metrics beyond current citation counting oligopoly Open Access Variants:  OA archiving (“green”): good but not enough (delays & limits in usability & sustainability)  OA publishing (“gold”): immediate & full benefits & sustainability
  4. 4. Open Access & Quality Assurance Open Access is not a threat to scientific quality assurance but an urgently needed opportunity for improvement Traditional Peer Review: compatible with OA  successful OA journals with traditional & refined review procedures: New J. Phys., Living Reviews, BMJ, BMC Medicine, PLOS Biology, Frontiers, eLife, et al. Information for Reviewers: augmented by OA  easy & interdisciplinary access to relevant publications Public Review & Interactive Discussion: enabled by OA  multi-stage open peer review involving the scientific community: ACP/EGU, Economics e-journal, Biology Direct, F1000 Research, et al. Post-Publication Review & Evaluation: advanced by OA  transparent & comprehensive analyses of article contents & impact (diversity vs. oligopoly): Article Level Metrics (ALM): downloads, views, citations, scientific & social media, … Predatory OA Publishers: side-issue, transition problem & red herring  low quality outlets not new but attracted by “OA vacuum”; OASPA vs. Beall‘s list ...
  5. 5. Quality Assurance Problems I Tip of the Iceberg: fraud  plagiarism & selective omission, tuning or fabrication of results (Schön et al., 2002/2003; Hwang et al. 2004/2005, …) Common Practice: carelessness  superficial & irreproducible description of experiments & models  non-traceable arguments & conclusions, duplicate & split papers, etc.  dilution rather than generation of knowledge Consequences: waste & misallocation of resources  costly reconstruction of poorly described methods & results  propagation of errors & misinterpretations  misevaluation of projects & scientists Pöschl Learned Publishing 2004; Frontiers Comp. Neuroscience 2012 Manuscripts & publications are often carelessly prepared & faulty
  6. 6. Quality Assurance Problems II Editors & Referees: limited capacities  few editors for large subject areas  limited knowledge of details & specialist referees  work overload, conflicts of interest, little reward & incentive for constructive reviews  superficial or prejudiced review & evaluation Traditional Pre-Publication Peer Review: retardation & loss of information  delay of publication, dilution of messages, hidden obstruction/plagiarism  critical & complementary comments unpublished/lost (often as interesting as paper)  waste of reviewer capacities as most limited resource Traditional Discussion: sparse & late commentaries  laborious, delayed & diluted by review (comment/article 1978  1998: 1/20  1/100) Replacement of traditional pre-publication review & by post-publication commenting not really successful (comments/article < 5/100) Evolution into Multi-Stage Open Peer Review: combine & integrate strengths of traditional peer review with virtues of transparency, discussion & self regulation Pöschl Learned Publishing 2004; Frontiers Comp. Neuroscience 2012 Traditional peer review is insufficient for efficient quality assurance in today’s highly diverse & rapidly evolving world of science
  7. 7. Dilemma: Speed vs. Quality Rapid Publication: widely pursued  required for efficient exchange (new findings, critical questions)  traditionally pursued through hasty reviews & short papers (lack of time & detailed information) Thorough Review & Discussion: often neglected  required to identify scientific flaws & duplications  traditionally limited by availability of refereeing capacities (referee qualifications & interests, time, access to relevant information) Conflicting needs of scientific publishing: rapid publication vs. thorough review & discussion Pöschl Learned Publishing 2004; Frontiers Comp. Neuroscience 2012
  8. 8. Solution: Speed & Quality Stage 1: Rapid publication of Discussion Paper pre-selected by editors (optionally supported by referees), fully citable & permanently archived (more than traditional preprint, comparable to arXiv.org, Nature Precedings, PLOS ONE …) Public Peer Review & Interactive Discussion referee & community comments published alongside discussion paper (anonymous or by name), non-reviewed but individually citable & permanently archived Stage 2/X: Review completion & publication of Final Paper analogous to traditional peer review & journal publication Pöschl Learned Publishing 2004; Frontiers Comp. Neuroscience 2012 Two-stage/multi-stage publication with public peer review & discussion
  9. 9. Outline Introduction  motivation & challenges Multi-Stage Open Peer Review and Interactive Open Access Publishing  concepts & effects  achievements & statistics (ACP, EGU) Conclusions  vision & propositions
  10. 10. Interactive OA Publishing with Multi-Stage Peer Review 1. Pre-publication review & selection short term Discussion Forum (Stage 1) Journal (Stage 2) 3. Peer review completion mid term 4. Post-publication review & evaluation long-term, ALM open access required 2. Public peer review & interactive discussion mid-term, integrative ! days ↔ weeks weeks ↔ months/years
  11. 11. Advantages All-win situation: authors, referees, editors, readers, community Discussion Paper  free speech, rapid publication, citable record (authors, readers) Public Peer Review & Interactive Discussion  direct feedback & public recognition for high quality papers (authors)  prevent hidden obstruction & plagiarism (authors, editors)  foster & document scientific discourse: critical comments, constructive suggestions, complementary information (authors, referees, readers, editors)  document controversial arguments & innovations or flaws & misconduct (referees, editors, readers)  deter submission of weak & false papers  save reviewer capacities (referees, editors) Final Paper  maximize quality assurance & information density through integration of peer review, public discussion & final revision (readers) Pöschl, Learned Publishing 2004; Frontiers Neuroscience 2012
  12. 12. ACP Online Library “Most Commented Papers”: www.atmos-chem-phys.net/most_commented.html Hansen et al. 2016: climate, 110 comments www.atmos-chem-phys.net/16/3761/2016/acp-16- 3761-2016-discussion.html 125,000 views & downloads (77,000 PDF) Makarieva et al. 2008, 2013: meteorol. discussion, 33+20 comments www.atmos-chem-phys-discuss.net/acpd-2008-0250/ www.atmos-chem-phys.net/13/1039/2013/acp-13-1039- 2013-discussion.html
  13. 13. ACP Publication & Discussion Statistics Discussion Papers (ACPD)  submissions (increasing): ~ 100 month-1 (US, D, UK, F, … )  rejections (access review): ~ 10 %  submission-to-publication time: ~ 1 month (min: 1 week)  publication charge (author): ~ 1000 EUR/paper (incl. final paper) Final Papers (ACP)  rejections (review completion): ~ 5 % (~15 % total, save referees !)  submission-to-publication time: ~ 1 month (3-6 months in total) Interactive Discussion  interactive comments / discussion paper: ~ 5 (up to 110)  comment pages / paper pages: ~ 50 %  referee anonymity (exp. vs. mod.): ~ 70 % (80% vs. 60%)  community comments / discussion paper: ~ 1/4 (up to ~20)  constructive suggestions, harsh criticism, compliments Extended Discussion  peer-reviewed commentaries / paper: ~ 1/100 ( trad. journals) Pöschl Frontiers Comp. Neuroscience 2012
  14. 14. Atmospheric Chemistry & Physics (ACP) launched 2001 with Nobel laureate P. Crutzen & European Geosciences Union (EGU) 15 EGU sister journals since then: Biogeosciences, Climate, Hydrology ... Large-scale move to interactive OA publishing in geosciences Concept spreading to other communities: Economics e-journal, F1000 Research, SciPost/arXiv.org, Winnower ... Unique combination:  top speed: 1+x weeks from submission to citable publication (discussion paper)  top impact & visibility (across atmos, environ. & geosciences)  large volume (~10% market share)  low rejection rates (~15% vs. ~50%)  low cost (~1000 EUR/paper vs. ~2000-5000 EUR/paper)  fully self-financed & sustainable (incl. review, production, archiving & 10-20% surplus for publisher & society), 2013: ~3000 papers, ~3 MEUR turnover, ~500 kEUR surplus for EGU Achievements Pöschl Frontiers Comp. Neuroscience 2012
  15. 15. Conclusions from ACP/EGU ACP/EGU interactive open access journals demonstrate that: 1) Strengths of traditional publishing & peer review can be efficiently combined with the opportunities of open access, interactive discussion & public peer review 2) Interactive open access peer review (public review & discussion) enables efficient quality assurance  high quality & top visibility at low rejection rates; flexibly adjustable to different communities 3) Transparency enhances self-regulation and saves the most limited resource in scientific publishing & QA: refereeing capacities 4) Scientific societies & commercial publishers can establish new OA journals & improve quality assurance mechanisms 5) Traditional journals can be efficiently & successfully converted into (interactive) open access journals 6) Interactive open access publishing can be realized at moderate costs (~ 1 kEUR/paper), and technology may reduce costs further Pöschl Frontiers Comp. Neuroscience 2012
  16. 16. Alternative Concepts Open Peer Review w/o Anonymity  e.g. J. of Interactive Media in Education, BMC Biology Direct, Brit. Medical Journal …  no opportunity for referees to remain anonymous  difficulties with critical comments & refereeing capacities Pre-Publication History & Post-Commenting (Peer Commentary)  e.g. BioMed Central Medical Journals, Behavioral & Brain Sciences, PLOS One, BMJ, PeerJ …  no integration of peer review & public discussion  less opportunity & incentive for community participation Multi-Stage Open Peer Review  e.g. ACP & EGU/Copernicus, Economics e-journal, F1000 Research, SciPost/arXiv …  do not abandon traditional peer review but maintain its strengths & reduce its weaknesses by transparency & interactive discussion  optional anonymity, integrate peer review & public discussion, iterate review & revision  evolutionary & modular approach, flexibly adjustable to different communities Pöschl Frontiers Comp. Neuroscience 2012 Details & subtleties can make a difference.
  17. 17. Future Perspectives of Multi-Stage Open Peer Review Combine & integrate with  repositories (arXiv.org …): SciPost/arXiv  living reviews (LR in Relativity …)  rankings & tiers (BE Press Economics, Nature X …)  article level metrics (SPARC, PLOS …)  assessment houses & virtual journals: highlight selections, seal/stamp of approval ... (ISI, Scopus, Google Schol., F1000, ScienceOpen …) 1. Pre-publication review & selection selective: magazine vs. journal vs. repository… 2. Public peer review & interactive discussion integrative: referees & community … 3. Peer review completion iterative: quality assurance feedback loops … 4. Post-publication review & evaluation retrospective: citations, ratings, bibliometry, ALM ... Pöschl Front. Comp. Neurosci. 2012, Hyman & Renn, Edition OA 2012 Modular, flexible & transparent new standard of quality assurance.  epistemic web
  18. 18. Multi-Stage Open Peer Review QAIntensity/Level Discussion Paper ACPD, Economics Preprint/e-print (arXiv) Manuscript (author) Effort / Time Pre-Screening Access Review Article v1 F1000 Res. Public Review & Discussion Journal Article ACP, SciPost, Economics Article v2/vX F1000 Res. Editor/ Reviewer Highlight Selection Highlight Magazine SciPost Select, Science/Nature ? Highlight Section ACP, Economics Views, Downloads, Ratings, Citations … plain, context-weighted … Highly Cited / Ranked / Used ISI-WoK, Scopus, Google Scholar, OA-ALM …
  19. 19. Provide access to high quality scientific publications review & revision involving the community  more & better information for scientists & society Document the scientific discourse public record of scientific evidence, arguments & progress  universal & traceable web of knowledge (epistemic web) Demonstrate transparency & rationalism transparent & rational approach to complex questions & problems  role model for societal decision processes Vision Promote societal progress by open access, public review & interactive discussion in global commons of scholarly information.
  20. 20. Propositions 1) Continue & promote experiments with improved forms of peer review  build on existing models & experience rather than re-inventing the wheel  use & expand multi-stage open peer review as flexible construction kit 2) Introduce & demand access to article reviews & pre-publication history  establish new standards & proofs of quality assurance to cope with increase of scholarly articles & journals (incl. predatory OA publishers) 3) Advance & apply new metrics of publication impact & quality  use article level metrics instead of misleading journal impact factors  terminate intolerable, intransparent & unscholarly reliance on citation counting oligopoly (ISI, Scopus, Google Scholar, …) 4) Promote open access publishing as a basis for innovation  continue to support new & improved forms of OA publishing  trust principles of mass/energy conservation: OA publishing costs can be covered by conversion of subscription budgets (offsetting …)  transform majority of today’s scholarly journals from subscription to OA in accordance with community-specific publication preferences.  OA2020 Initiative & Expression of Interest for OA Transformation of Scholarly Journals
  21. 21. Outside Perspectives on ACP/EGU Ho et al., Views on the peer review system of biomedical journals, BMC Med. Res. Method., 2013 Survey of 1300/28000 biomedical academics & conclusions/recommendations: Biomedical journals may consider issuing publication ethics guidelines, offering courses for reviewers, providing authors with channels to expressing their concerns and the adoption of multi-stage open peer review. Bornmann et al., Is Interactive Open Access Publishing Able to Identify High-Impact Submissions?, J. Am Soc. Inform. Sci. Technol., 2013 A Study on the Predictive Validity of Atmospheric Chemistry and Physics by Using Percentile Rank Classes All in all, our results on the predictive validity of the ACP peer review system can support the high expectations that Pöschl (2010), chief executive editor of ACP, has of the new selection process at the journal: “The two-stage publication process stimulates scientists to prove their competence via individual high-quality papers and their discussion, rather than just by pushing as many papers as possible through journals with closed peer review …”
  22. 22. Further References I The following references and links provide orientation about the development and perspectives of open access in general and interactive open access publishing with public peer review and interactive discussion in particular (multi-stage open peer review as practiced at EGU). 1. Open Access Declarations & Initiatives 1.1. Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities http://openaccess.mpg.de/286432/Berlin-Declaration http://openaccess.mpg.de/319790/Signatories http://openaccess.mpg.de/mission-statement_en http://openaccess.mpg.de/1527674/Session_II http://openaccess.mpg.de/1528633/Session-2-Poeschl.pdf 1.2. Bethesda Statement on Open Access Publishing http://legacy.earlham.edu/~peters/fos/bethesda.htm 1.3. Budapest Open Access Initiative http://www.budapestopenaccessinitiative.org/ http://www.budapestopenaccessinitiative.org/boai-10-recommendations http://www.opensocietyfoundations.org/voices/opening-access-research 2. Development & Concepts of Interactive Open Access Publishing & Public Peer Review 2.1. Multi-stage open peer review: scientific evaluation integrating the strengths of traditional peer review with the virtues of transparency and self-regulation http://journal.frontiersin.org/Journal/10.3389/fncom.2012.00033/abstract 2.2. Interactive journal concept for improved scientific publishing and quality assurance http://www.ingentaconnect.com/content/alpsp/lp/2004/00000017/00000002/art00005
  23. 23. Further References II 2.3. A Short History of Interactive Open Access Publishing http://publications.copernicus.org/A_short_History_of_Interactive_Open_Access_Publishing.pdf 2.4. EGU Position Statement on the Status of Discussion Papers Published in EGU Interactive Open Access Journals, European Geosciences Union 2010 http://www.egu.eu/about/statements/position-statement-on-the-status-of-discussion-papers-published-in- egu-interactive-open-access-journals/ 2.5. Further initiatives & visions of open evaluation http://www.economics-ejournal.org/ http://f1000research.com/ https://www.scienceopen.com/ http://www.frontiersin.org/Computational_Neuroscience/researchtopics/Beyond_open_access:_visions_ for_open_evaluation_of_scientific_papers_by_post-publication_peer_review/137

×