Peer review in scholarly journals

Academy of Science of South Africa
Academy of Science of South AfricaAcademy of Science of South Africa
@SAJS_Official @ASSAf_Official
#PeerReview
Peer review in
scholarly journals
PEER
E
V
I
E
W
13 September 2023
@SAJS_Official @ASSAf_Official
#PeerReview
13 September 2023
Peer review in scholarly journals
P E E R
E
V
I
E
W
PROGRAMME
Welcome and introduction
What is peer review and what is it not? Jemma Finch, UKZN Associate Editor: SAJS
What reviewers should know about peer review Nkosinathi Madondo, MUT Associate Editor Mentee:
SAJS
What authors should know about peer review Shane Redelinghuys, Wits Associate Editor Mentee:
SAJS
Q&A
What everyone should know about AI and peer
review
Martin Bekker Wits University
Where to start with peer review Leslie Swartz, SUN Editor-in-Chief: SAJS
Q&A
Thanks and closure
@SAJS_Official @ASSAf_Official
#PeerReview
13 September 2023
Peer review in scholarly journals
P E E R
E
V
I
E
W
Meet the presenters
@SAJS_Official @ASSAf_Official
#PeerReview
13 September 2023
Peer review in scholarly journals
P E E R
E
V
I
E
W
What is peer review
and what is it not?
Jemma Finch
@SAJS_Official @ASSAf_Official
#PeerReview
13 September 2023
Peer review in
scholarly journals
P E E R
E
V
I
E
W
@SAJS_Official @ASSAf_Official
#PeerReview
‘Peer review is secretive. Only those present
in the deliberative chambers know exactly
what happens there’ (Lamont, 2009, p. 2)
@SAJS_Official @ASSAf_Official
#PeerReview
?
?
?
?
?
?
?
?
?
?
? ?
?
?
?
?
?
?
?
?
@SAJS_Official @ASSAf_Official
#PeerReview
Let’s make things more transparent….
• What is peer review?
• What is it not?
• What are we striving for?
What is peer review?
• Check quality, novelty, significance and
presentation
• Constructive
• Collegial and respectful
• Detailed and comprehensive
@SAJS_Official @ASSAf_Official
#PeerReview
What is peer review?
• Independent, unbiased
• Anonymous (depending on the model)
• Confidential
• Professional service
@SAJS_Official @ASSAf_Official
#PeerReview
What is it not?
• Gatekeeping
• Spelling and grammar check
• Opinions unaccompanied by evidence
• Personal
• Paid (usually)
@SAJS_Official @ASSAf_Official
#PeerReview
What is it not?
• A chance to:
• Pull apart other people’s work
• Show how clever you are
• Advance competing interests
• Help your buddies
• Boost your citations
@SAJS_Official @ASSAf_Official
#PeerReview
What are we striving for?
• Diverse reviewers: origin, gender, career
stage
• Transparency
• Fairness
@SAJS_Official @ASSAf_Official
#PeerReview
Further reading
• Small Pond Science Blog ‘Why I don’t have trouble finding peer reviewers’
https://smallpondscience.com/2022/07/20/why-i-dont-have-trouble-finding-peer-
reviewers/#more-27097
• Stephen Heard’s Scientist Sees Squirrel Blog ‘How (as an editor) I choose lists
of reviewers’ https://scientistseessquirrel.wordpress.com/2018/12/20/how-as-an-
editor-i-choose-lists-of-reviewers/
• Lamont 2009: How Professors Think: Inside the Curious World of Academic
Judgment. Harvard University Press.
• Day & Gastel 2012: How to write and publish a scientific paper. Cambridge
University Press.
• Cargill & O’Connor 2013: Writing Scientific Research Articles. Wiley
Blackwell.
@SAJS_Official @ASSAf_Official
#PeerReview
What reviewers need to
know about peer
review?
nathi madondo
13 September 2023
@SAJS_Official @ASSAf_Official
#PeerReview
Peer review in
scholarly journals
P E E R
E
V
I
E
W
Mytalkwill beframedaroundtheidea
oftakingthescholarship forward
@SAJS_Official @ASSAf_Official
#PeerReview
Taking the scholarship forward
@SAJS_Official @ASSAf_Official
#PeerReview
A cursory report cannot
A detailed report can
In a way, not only reviewing the quality of
the manuscript but also your quality as a
reviewer (Boughey, 2022).
What isthe quality of a
manuscript?
Well presented and clear argument
Topic has the potential to inform current
debate
Methodological coherence
Conceptual depth
@SAJS_Official @ASSAf_Official
#PeerReview
What quality is not?
Unsubstantiated claims – sweeping
statements
Some authors try to make an easy
publication to earn subsidy
@SAJS_Official @ASSAf_Official
#PeerReview
The paper to review should be in the
area of expertise of the reviewer to be
able to judge its quality
To avoid cursory report
@SAJS_Official @ASSAf_Official
#PeerReview
To judge quality …
So, to review is to take the scholarship
forward
@SAJS_Official @ASSAf_Official
#PeerReview
Concluding thoughts
A big challenge to take the scholarship
forward - neo-liberal agenda
@SAJS_Official @ASSAf_Official
#PeerReview
References
Boughey, C. (2023, March 31). NRF rating process
with Professor Chrissie Boughey [video file]. You
tube. https://youtu.be/uIqdjb3z1l8
@SAJS_Official @ASSAf_Official
#PeerReview
What authors should know about
peer review
Shane Redelinghuys
@SAJS_Official @ASSAf_Official
#PeerReview
13 September 2023
Peer review in
scholarly journals
P E E R
E
V
I
E
W
Initial assessment
‘Under review’
2-16 weeks
~100 days
Know your publisher’s policies
• Manuscript submission instructions (formatting, referencing
style, word limit)
• Attention to detail will improve the aesthetics of your manuscript
– simplify reading to appreciate the content
• Streamlines the review process, minimise turnaround time
• Publishing agreements (e.g., open access, publication fees) &
ethical principles
• Is it acceptable?
 Yes!
 Won’t affect the handling of your paper or the review process
 Do so respectfully
 First, use the online tracking system of the journal to track status
 Will hear back very soon if manuscript not suited for the journal
 ‘Under review’ – reviewers have been invited / waiting for
reviewers to accept review / waiting for reviewers to submit their
review reports / reviews being assessed by AE
Follow-up on submission
• When is a good time to do so?
 Subject matter – dependent, technicality of your paper
 After 3-6 months
Follow-up on submission
An analysis of ~3 million papers indexed in PubMed from 1981 to 2015
indicated that the median time-until-acceptance has remained consistent at
around 100 days (i.e., approx. 3 months)
• A platform on which researchers tell others about their experiences with a
particular journal
• A tool to help you determine which journal to submit to, based on researcher-
provided statistics:
 Time until first review received
 Total handling time
 Time until desk rejection (rejection without review)
 Number of review reports received
 Quality of reviews (ranked out of 5) (subjective)
 Number of rounds of review
SciRev – An extra layer of transparency?
Peer review in scholarly journals
Peer review in scholarly journals
• Don’t take it personal
• Manuscript rejection – does not mean it is the end of the road for
your paper
• Consider why – take reviews into account – May be too
specialised/general
• Decide if the work is worth resubmitting (most of the time it is)
• Improve work by taking a different approach with your analysis –
brainstorm with peers
Rejection – what to do?
@SAJS_Official @ASSAf_Official
#PeerReview
Sources
• Peer review ethics: Six things every author should know
https://www.apa.org/about/division/digest/publishing/peer-review-ethics
• The history of publishing delays
https://blog.dhimmel.com/history-of-delays/
• The peer review process
https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-
review/the-peer-review-process.html
• Stichting SciRev
https://scirev.org/
Martin Bekker
School of Electronic and Information Engineering, University of the Witwatersrand
Martin Bekker // martin.bekker@wits.ac.za
LLMs tools and you, the peer reviewer
Our time together
How do LLMs work?
What are they good at?
What are they bad at?
How to get the best out of LLMs
What do I do if something smells fishy?
What level of AI-support is acceptable?
How should AI be referenced?
(How) should prompts be shared?
Warning: Strong Language and Sensitive Content
The following presentation contains examples of language and content that may be
offensive, including swearing, violent language, and racist undertones. These examples are
presented solely for the purpose of analysis and discussion, highlighting the impact of such
language in certain contexts. The intention is not to endorse or promote any form of
discrimination, violence, or offensive behaviour.
It is important to approach these examples with sensitivity and understanding, recognizing
that they may cause discomfort or distress. We encourage you to exercise discretion when
considering the appropriateness of this content for your audience.
Please note that the presentation aims to foster awareness, education, and critical thinking
regarding the use of offensive language and its potential effects. We strongly condemn any
form of discrimination, violence, or hate speech, and advocate for respectful and inclusive
communication.
By proceeding with this presentation, you acknowledge that you have been provided with
this warning and disclaimer, and that you understand the purpose and context in which
these examples are being presented.
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Large Language Models
Many available today
A corpus of text
A stochastic model
A base model
Safety layers
Reinforcement Learning through Human Feedback
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Strengths // What it is good at
Repackaging of ideas
Summating text
Style transfer
Fixing grammar
Brainstorming / Dinner Ideas / Essay Ideas
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Weaknesses
Mathematical model ≠ not a linguistic model
Stochastic parrot
Hallucinations
Causal reasoning
Jailbreaking
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Peer review in scholarly journals
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Peer review in scholarly journals
Martin Bekker // martin.bekker@wits.ac.za
Hidden Harms
Human Labour Exploitation
Environmental
Monopolising power
(Fundamental?) Un-understandability
Commodity Fetishism
Martin Bekker // martin.bekker@wits.ac.za
Threats
Claims of IP non-consent (and theft)
Racism, sexism, hate speech (based on training biases)
Poor advice (negligent, dangerous, harmful)
Mass-scale cheating
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
Peer review in scholarly journals
Martin Bekker // martin.bekker@wits.ac.za
Martin Bekker // martin.bekker@wits.ac.za
How to get the best out of LLMs
Only use for what it’s good (narrow!)
Only use once the thinking’s done (generally)
Never let it have the final say
• GTPZero
• Quillbot.com
• Quetext.com
• AIdetector.com
• Scale.ai
• Originality.ai
Martin Bekker // martin.bekker@wits.ac.za
Prompt: Write an essay on {insert topic} in {insert wordcount} in
the style of an 18-year old second langue speaker. Use at least 10
unusual words, and two sentences must be translated directly
from Afrikaans, using Afrikaans grammar.
Academics (ad students!) are human
It’s dangerous to think one can outsmart tech (most of the time)
Critical thinking under threat
What do I do if something smells fishy?
1. Understand where your publication stands vis-à-vis LLMs (default
should be ‘none’
2. Seek acknowledgements
3. Search (a few) citations
4. Talk to author
How should AI-help be referenced?
If usage permitted -
Norms (e.g. APA)
Acknowledgements
Level
Github / Repository
{but consider how would
journal check for
paraphrased plagiarism?}
Prompts
Online repository
Google Drive
Github
Supplemetary documentation
Platform, date, version (if possible)
Use Ban
Grammar and spelling assistance / Proofing
Co-creation / Editor
Brainstorming and planning
‘Over to you’ / No holds barred
Show your prompts
Responsible for your work
Martin Bekker // martin.bekker@wits.ac.za
Happy reviewing
Martin Bekker // martin.bekker@wits.ac.za
Where to start
with peer review
Leslie Swartz
@SAJS_Official @ASSAf_Official
#PeerReview
13 September 2023
Peer review in
scholarly journals
P E E R
E
V
I
E
W
A recap,
and some
principles
Peer review is central to the academic enterprise, and
is as important as writing your own articles
There is currently a crisis in peer reviewing – it is hard
for journals to find peer reviewers, and without peer
review journals cannot continue to function
We are all peers
We all have a responsibility to keep the peer review
system going
A rule of thumb: for every time I send an article for
review, I should be prepared to review another article
What do I want from a peer reviewer?
• I want someone who is
• Competent
• Constructive
• Fair
• Kind
• Clear
• Focused on making my work better
• Not trying to make me a version of themselves
“I am not good enough”
Academics and impostor
syndrome – who do NOT
think that they are
impostors?
The academic hierarchy
(“only professors….”)
What do you really need to
know, and what don’t you
need to know?
It is fine to point out the
limitations of your
knowledge in your review
(the editor may well have
chosen a range of
competencies
You don’t have
to pretend
“I’m not
sure I am
the right
person to
review
this”
• Make a list of what you think you can and
cannot do
• How important is what you can do to
helping the author and the journal?
• How much of a barrier is what you can’t do
for your ability to be helpful?
Read the paper quickly, and then:
• Don’t be shy to contact the editor and raise
any concerns – we editors are grateful to
you and want to work with you!
If still in doubt
Your role
as a
reviewer is
that of a
‘peer
mentor’
(Way, et
al, 2021)
You are not expected to know everything,
but you can make it clear what you do and
don’t know
Four main
areas:
Do I understand the
methods?
Do I have a good idea of
who the audience is for
this journal? Can I stand
as a ‘representative’ of
that audience?
Do I know enough about
the content to
comment?
Have the authors told
me enough about the
context of the work that
I can make an
assessment?
Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM,
Mavis B. Fundamentals of Scholarly Peer Review: A
Workshop for Health Professions Educators on Practicing
Scholarly Citizenship. MedEdPORTAL. 2021 Aug 2;17:11174.
You are not
alone…
• Get help from others and give help
to others, but ONLY ONCE YOU HAVE
ASKED THE EDITOR IF YOU MAY DO
THIS
• Concerns:
• Confidentiality and respect for
authors
• “Ghost reviewing” and
exploitation of more junior
reviewers, hidden from sight
• In this – remember your rights, those
of the authors, and those of the
journal
You are
not
alone…
• Consider some options
• Ask your supervisor/a more
experienced colleague to
include you in peer reviewing
when they get approached
• Approach a more senior
colleague/supervisor to help you
when you get asked to peer
review
• Develop peer review buddy
systems (peer to peer)
Peer review buddy systems (1)
• Link in with “shut up and write” – writing support systems
• Journal clubs also useful
• A good place to start, even before you do peer reviews
yourselves:
• Make a commitment that you and your buddies will “peer
review” reviews you yourselves receive from journals.
• Assess the reviews you receive in terms of competence,
respectfulness, constructiveness, etc; and
• Suggest in your group ways in which the reviews could
have been improved
Peer
review
buddy
systems
(2)
First, establish rules
of confidentiality for
your group
Always get
permission from the
editor to use a peer
system
Step 1: Each
(independently)
sketch out areas to
be covered in review
Step 2: Distribute the
tasks and each do
your task
Step 3: Review what
one another have
done
Step 4: Assemble full
review
• Write in a supportive tone, but be definitive
• Summarize your understanding of the work
• Explain your overall impressions (recommendation /
ratings)
• Base your overall impressions on actual content
• Indicate strengths and weaknesses, providing specific
examples
• Provide suggestions for improvement
• Be detailed and clear
• Explain any descriptors, such as “insufficient method”
Tips for Writing Comments*
* Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisor’s
completed clinical evaluation reports. Med Educ 2008; 42:816-22.
NOTE: THIS SLIDE IN ITS ENTIRETY IS COPIED FROM Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM, Mavis B. Fundamentals of Scholarly Peer Review: A
Workshop for Health Professions Educators on Practicing Scholarly Citizenship. MedEdPORTAL. 2021;17:11174. https://doi.org/10.15766/mep_2374-
8265.11174
A PLEA
FROM A
JOURNAL
EDITOR
Please do not undermine the
peer review process while it is
underway, BUT
Do not
undermine
the process
Please do give journals feedback
on peer review experiences,
good and bad
Do give
feedback
Remember: we need to peer
review peer review!
Remember
1 von 76

Más contenido relacionado

Similar a Peer review in scholarly journals(20)

Peer reviewsPeer reviews
Peer reviews
Alexander Serebrenik1.4K views
Publication StrategyPublication Strategy
Publication Strategy
Smriti Arora85 views
Research Paper Writing[www.writekraft.com]Research Paper Writing[www.writekraft.com]
Research Paper Writing[www.writekraft.com]
WriteKraft Dissertations52 views
Introduction to Library Research Skills Introduction to Library Research Skills
Introduction to Library Research Skills
Northeast Center, Office of Academic Support, SUNY Empire State College6.8K views
Sample Source Evaluation SampleSample Source Evaluation Sample
Sample Source Evaluation Sample
I Don'T Want To Write My Paper 5 views
Week 2 in Research.pdfWeek 2 in Research.pdf
Week 2 in Research.pdf
sdfghj212 views
Klaus-MSKCC-Feb-8-2010.pptKlaus-MSKCC-Feb-8-2010.ppt
Klaus-MSKCC-Feb-8-2010.ppt
JamesBon182 views
2015-03GrantWriting2015-03GrantWriting
2015-03GrantWriting
Institute for Research on Teaching & Learning1.7K views
Behind the scenes of peer reviewBehind the scenes of peer review
Behind the scenes of peer review
Academy of Science of South Africa191 views
Evaluation Essays SamplesEvaluation Essays Samples
Evaluation Essays Samples
Best Paper Writing Services 6 views

Más de Academy of Science of South Africa(20)

Writing for a scholarly journalWriting for a scholarly journal
Writing for a scholarly journal
Academy of Science of South Africa404 views
Writing for a scholarly journalWriting for a scholarly journal
Writing for a scholarly journal
Academy of Science of South Africa1K views
SPU Overview DATAD-R September 2017SPU Overview DATAD-R September 2017
SPU Overview DATAD-R September 2017
Academy of Science of South Africa119 views
Leti Kleyn UPSpace 10 Year OverviewLeti Kleyn UPSpace 10 Year Overview
Leti Kleyn UPSpace 10 Year Overview
Academy of Science of South Africa163 views
DATAD-R African Open Science Platform (AOSP)DATAD-R African Open Science Platform (AOSP)
DATAD-R African Open Science Platform (AOSP)
Academy of Science of South Africa194 views
The South African Journal of Science - iThenticateThe South African Journal of Science - iThenticate
The South African Journal of Science - iThenticate
Academy of Science of South Africa683 views
8-Acknowledging funders8-Acknowledging funders
8-Acknowledging funders
Academy of Science of South Africa368 views
7-Managing errata & retractions with CrossMark7-Managing errata & retractions with CrossMark
7-Managing errata & retractions with CrossMark
Academy of Science of South Africa512 views
6-Screening for plagiarism6-Screening for plagiarism
6-Screening for plagiarism
Academy of Science of South Africa198 views
5-Cited-by-linking5-Cited-by-linking
5-Cited-by-linking
Academy of Science of South Africa169 views
4-Managing CrossRef DOIs4-Managing CrossRef DOIs
4-Managing CrossRef DOIs
Academy of Science of South Africa245 views
3-CrossRef initiatives3-CrossRef initiatives
3-CrossRef initiatives
Academy of Science of South Africa165 views
2-Quality publishing in a digital environment2-Quality publishing in a digital environment
2-Quality publishing in a digital environment
Academy of Science of South Africa174 views
1-Introduction to CrossRef1-Introduction to CrossRef
1-Introduction to CrossRef
Academy of Science of South Africa169 views
The State of SA Journals Project - Johann MoutonThe State of SA Journals Project - Johann Mouton
The State of SA Journals Project - Johann Mouton
Academy of Science of South Africa625 views
SciELO South Africa - Louise van HeerdenSciELO South Africa - Louise van Heerden
SciELO South Africa - Louise van Heerden
Academy of Science of South Africa428 views
Scholarly Publishing Report - Ina SmithScholarly Publishing Report - Ina Smith
Scholarly Publishing Report - Ina Smith
Academy of Science of South Africa220 views

Último(20)

Dance KS5 BreakdownDance KS5 Breakdown
Dance KS5 Breakdown
WestHatch52 views
Narration lesson plan.docxNarration lesson plan.docx
Narration lesson plan.docx
Tariq KHAN90 views
Education and Diversity.pptxEducation and Diversity.pptx
Education and Diversity.pptx
DrHafizKosar56 views
Class 10 English  lesson plansClass 10 English  lesson plans
Class 10 English lesson plans
Tariq KHAN172 views
Narration  ppt.pptxNarration  ppt.pptx
Narration ppt.pptx
Tariq KHAN62 views
Scope of Biochemistry.pptxScope of Biochemistry.pptx
Scope of Biochemistry.pptx
shoba shoba110 views
Chemistry of sex hormones.pptxChemistry of sex hormones.pptx
Chemistry of sex hormones.pptx
RAJ K. MAURYA97 views
CWP_23995_2013_17_11_2023_FINAL_ORDER.pdfCWP_23995_2013_17_11_2023_FINAL_ORDER.pdf
CWP_23995_2013_17_11_2023_FINAL_ORDER.pdf
SukhwinderSingh895865467 views
NS3 Unit 2 Life processes of animals.pptxNS3 Unit 2 Life processes of animals.pptx
NS3 Unit 2 Life processes of animals.pptx
manuelaromero201389 views
ACTIVITY BOOK key water sports.pptxACTIVITY BOOK key water sports.pptx
ACTIVITY BOOK key water sports.pptx
Mar Caston Palacio132 views
SIMPLE PRESENT TENSE_new.pptxSIMPLE PRESENT TENSE_new.pptx
SIMPLE PRESENT TENSE_new.pptx
nisrinamadani2146 views
ICANNICANN
ICANN
RajaulKarim2057 views
Streaming Quiz 2023.pdfStreaming Quiz 2023.pdf
Streaming Quiz 2023.pdf
Quiz Club NITW87 views

Peer review in scholarly journals

  • 1. @SAJS_Official @ASSAf_Official #PeerReview Peer review in scholarly journals PEER E V I E W 13 September 2023
  • 2. @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W PROGRAMME Welcome and introduction What is peer review and what is it not? Jemma Finch, UKZN Associate Editor: SAJS What reviewers should know about peer review Nkosinathi Madondo, MUT Associate Editor Mentee: SAJS What authors should know about peer review Shane Redelinghuys, Wits Associate Editor Mentee: SAJS Q&A What everyone should know about AI and peer review Martin Bekker Wits University Where to start with peer review Leslie Swartz, SUN Editor-in-Chief: SAJS Q&A Thanks and closure
  • 3. @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W Meet the presenters
  • 4. @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W
  • 5. What is peer review and what is it not? Jemma Finch @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W
  • 7. ‘Peer review is secretive. Only those present in the deliberative chambers know exactly what happens there’ (Lamont, 2009, p. 2) @SAJS_Official @ASSAf_Official #PeerReview ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  • 8. @SAJS_Official @ASSAf_Official #PeerReview Let’s make things more transparent…. • What is peer review? • What is it not? • What are we striving for?
  • 9. What is peer review? • Check quality, novelty, significance and presentation • Constructive • Collegial and respectful • Detailed and comprehensive @SAJS_Official @ASSAf_Official #PeerReview
  • 10. What is peer review? • Independent, unbiased • Anonymous (depending on the model) • Confidential • Professional service @SAJS_Official @ASSAf_Official #PeerReview
  • 11. What is it not? • Gatekeeping • Spelling and grammar check • Opinions unaccompanied by evidence • Personal • Paid (usually) @SAJS_Official @ASSAf_Official #PeerReview
  • 12. What is it not? • A chance to: • Pull apart other people’s work • Show how clever you are • Advance competing interests • Help your buddies • Boost your citations @SAJS_Official @ASSAf_Official #PeerReview
  • 13. What are we striving for? • Diverse reviewers: origin, gender, career stage • Transparency • Fairness @SAJS_Official @ASSAf_Official #PeerReview
  • 14. Further reading • Small Pond Science Blog ‘Why I don’t have trouble finding peer reviewers’ https://smallpondscience.com/2022/07/20/why-i-dont-have-trouble-finding-peer- reviewers/#more-27097 • Stephen Heard’s Scientist Sees Squirrel Blog ‘How (as an editor) I choose lists of reviewers’ https://scientistseessquirrel.wordpress.com/2018/12/20/how-as-an- editor-i-choose-lists-of-reviewers/ • Lamont 2009: How Professors Think: Inside the Curious World of Academic Judgment. Harvard University Press. • Day & Gastel 2012: How to write and publish a scientific paper. Cambridge University Press. • Cargill & O’Connor 2013: Writing Scientific Research Articles. Wiley Blackwell. @SAJS_Official @ASSAf_Official #PeerReview
  • 15. What reviewers need to know about peer review? nathi madondo 13 September 2023 @SAJS_Official @ASSAf_Official #PeerReview Peer review in scholarly journals P E E R E V I E W
  • 17. Taking the scholarship forward @SAJS_Official @ASSAf_Official #PeerReview A cursory report cannot A detailed report can In a way, not only reviewing the quality of the manuscript but also your quality as a reviewer (Boughey, 2022).
  • 18. What isthe quality of a manuscript? Well presented and clear argument Topic has the potential to inform current debate Methodological coherence Conceptual depth @SAJS_Official @ASSAf_Official #PeerReview
  • 19. What quality is not? Unsubstantiated claims – sweeping statements Some authors try to make an easy publication to earn subsidy @SAJS_Official @ASSAf_Official #PeerReview
  • 20. The paper to review should be in the area of expertise of the reviewer to be able to judge its quality To avoid cursory report @SAJS_Official @ASSAf_Official #PeerReview To judge quality …
  • 21. So, to review is to take the scholarship forward @SAJS_Official @ASSAf_Official #PeerReview
  • 22. Concluding thoughts A big challenge to take the scholarship forward - neo-liberal agenda @SAJS_Official @ASSAf_Official #PeerReview
  • 23. References Boughey, C. (2023, March 31). NRF rating process with Professor Chrissie Boughey [video file]. You tube. https://youtu.be/uIqdjb3z1l8 @SAJS_Official @ASSAf_Official #PeerReview
  • 24. What authors should know about peer review Shane Redelinghuys @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W
  • 26. Know your publisher’s policies • Manuscript submission instructions (formatting, referencing style, word limit) • Attention to detail will improve the aesthetics of your manuscript – simplify reading to appreciate the content • Streamlines the review process, minimise turnaround time • Publishing agreements (e.g., open access, publication fees) & ethical principles
  • 27. • Is it acceptable?  Yes!  Won’t affect the handling of your paper or the review process  Do so respectfully  First, use the online tracking system of the journal to track status  Will hear back very soon if manuscript not suited for the journal  ‘Under review’ – reviewers have been invited / waiting for reviewers to accept review / waiting for reviewers to submit their review reports / reviews being assessed by AE Follow-up on submission
  • 28. • When is a good time to do so?  Subject matter – dependent, technicality of your paper  After 3-6 months Follow-up on submission An analysis of ~3 million papers indexed in PubMed from 1981 to 2015 indicated that the median time-until-acceptance has remained consistent at around 100 days (i.e., approx. 3 months)
  • 29. • A platform on which researchers tell others about their experiences with a particular journal • A tool to help you determine which journal to submit to, based on researcher- provided statistics:  Time until first review received  Total handling time  Time until desk rejection (rejection without review)  Number of review reports received  Quality of reviews (ranked out of 5) (subjective)  Number of rounds of review SciRev – An extra layer of transparency?
  • 32. • Don’t take it personal • Manuscript rejection – does not mean it is the end of the road for your paper • Consider why – take reviews into account – May be too specialised/general • Decide if the work is worth resubmitting (most of the time it is) • Improve work by taking a different approach with your analysis – brainstorm with peers Rejection – what to do?
  • 33. @SAJS_Official @ASSAf_Official #PeerReview Sources • Peer review ethics: Six things every author should know https://www.apa.org/about/division/digest/publishing/peer-review-ethics • The history of publishing delays https://blog.dhimmel.com/history-of-delays/ • The peer review process https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer- review/the-peer-review-process.html • Stichting SciRev https://scirev.org/
  • 34. Martin Bekker School of Electronic and Information Engineering, University of the Witwatersrand Martin Bekker // martin.bekker@wits.ac.za LLMs tools and you, the peer reviewer
  • 35. Our time together How do LLMs work? What are they good at? What are they bad at? How to get the best out of LLMs What do I do if something smells fishy? What level of AI-support is acceptable? How should AI be referenced? (How) should prompts be shared?
  • 36. Warning: Strong Language and Sensitive Content The following presentation contains examples of language and content that may be offensive, including swearing, violent language, and racist undertones. These examples are presented solely for the purpose of analysis and discussion, highlighting the impact of such language in certain contexts. The intention is not to endorse or promote any form of discrimination, violence, or offensive behaviour. It is important to approach these examples with sensitivity and understanding, recognizing that they may cause discomfort or distress. We encourage you to exercise discretion when considering the appropriateness of this content for your audience. Please note that the presentation aims to foster awareness, education, and critical thinking regarding the use of offensive language and its potential effects. We strongly condemn any form of discrimination, violence, or hate speech, and advocate for respectful and inclusive communication. By proceeding with this presentation, you acknowledge that you have been provided with this warning and disclaimer, and that you understand the purpose and context in which these examples are being presented. Martin Bekker // martin.bekker@wits.ac.za
  • 37. Martin Bekker // martin.bekker@wits.ac.za
  • 38. Large Language Models Many available today A corpus of text A stochastic model A base model Safety layers Reinforcement Learning through Human Feedback Martin Bekker // martin.bekker@wits.ac.za
  • 39. Martin Bekker // martin.bekker@wits.ac.za
  • 40. Strengths // What it is good at Repackaging of ideas Summating text Style transfer Fixing grammar Brainstorming / Dinner Ideas / Essay Ideas Martin Bekker // martin.bekker@wits.ac.za
  • 41. Martin Bekker // martin.bekker@wits.ac.za
  • 42. Martin Bekker // martin.bekker@wits.ac.za
  • 43. Martin Bekker // martin.bekker@wits.ac.za
  • 44. Weaknesses Mathematical model ≠ not a linguistic model Stochastic parrot Hallucinations Causal reasoning Jailbreaking Martin Bekker // martin.bekker@wits.ac.za
  • 45. Martin Bekker // martin.bekker@wits.ac.za
  • 46. Martin Bekker // martin.bekker@wits.ac.za
  • 48. Martin Bekker // martin.bekker@wits.ac.za
  • 49. Martin Bekker // martin.bekker@wits.ac.za
  • 51. Martin Bekker // martin.bekker@wits.ac.za
  • 52. Hidden Harms Human Labour Exploitation Environmental Monopolising power (Fundamental?) Un-understandability Commodity Fetishism Martin Bekker // martin.bekker@wits.ac.za
  • 53. Threats Claims of IP non-consent (and theft) Racism, sexism, hate speech (based on training biases) Poor advice (negligent, dangerous, harmful) Mass-scale cheating Martin Bekker // martin.bekker@wits.ac.za
  • 54. Martin Bekker // martin.bekker@wits.ac.za
  • 56. Martin Bekker // martin.bekker@wits.ac.za
  • 57. Martin Bekker // martin.bekker@wits.ac.za
  • 58. How to get the best out of LLMs Only use for what it’s good (narrow!) Only use once the thinking’s done (generally) Never let it have the final say
  • 59. • GTPZero • Quillbot.com • Quetext.com • AIdetector.com • Scale.ai • Originality.ai Martin Bekker // martin.bekker@wits.ac.za Prompt: Write an essay on {insert topic} in {insert wordcount} in the style of an 18-year old second langue speaker. Use at least 10 unusual words, and two sentences must be translated directly from Afrikaans, using Afrikaans grammar. Academics (ad students!) are human It’s dangerous to think one can outsmart tech (most of the time) Critical thinking under threat
  • 60. What do I do if something smells fishy? 1. Understand where your publication stands vis-à-vis LLMs (default should be ‘none’ 2. Seek acknowledgements 3. Search (a few) citations 4. Talk to author
  • 61. How should AI-help be referenced? If usage permitted - Norms (e.g. APA) Acknowledgements Level Github / Repository {but consider how would journal check for paraphrased plagiarism?}
  • 62. Prompts Online repository Google Drive Github Supplemetary documentation Platform, date, version (if possible)
  • 63. Use Ban Grammar and spelling assistance / Proofing Co-creation / Editor Brainstorming and planning ‘Over to you’ / No holds barred Show your prompts Responsible for your work Martin Bekker // martin.bekker@wits.ac.za
  • 64. Happy reviewing Martin Bekker // martin.bekker@wits.ac.za
  • 65. Where to start with peer review Leslie Swartz @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W
  • 66. A recap, and some principles Peer review is central to the academic enterprise, and is as important as writing your own articles There is currently a crisis in peer reviewing – it is hard for journals to find peer reviewers, and without peer review journals cannot continue to function We are all peers We all have a responsibility to keep the peer review system going A rule of thumb: for every time I send an article for review, I should be prepared to review another article
  • 67. What do I want from a peer reviewer? • I want someone who is • Competent • Constructive • Fair • Kind • Clear • Focused on making my work better • Not trying to make me a version of themselves
  • 68. “I am not good enough” Academics and impostor syndrome – who do NOT think that they are impostors? The academic hierarchy (“only professors….”) What do you really need to know, and what don’t you need to know? It is fine to point out the limitations of your knowledge in your review (the editor may well have chosen a range of competencies You don’t have to pretend
  • 69. “I’m not sure I am the right person to review this” • Make a list of what you think you can and cannot do • How important is what you can do to helping the author and the journal? • How much of a barrier is what you can’t do for your ability to be helpful? Read the paper quickly, and then: • Don’t be shy to contact the editor and raise any concerns – we editors are grateful to you and want to work with you! If still in doubt
  • 70. Your role as a reviewer is that of a ‘peer mentor’ (Way, et al, 2021) You are not expected to know everything, but you can make it clear what you do and don’t know Four main areas: Do I understand the methods? Do I have a good idea of who the audience is for this journal? Can I stand as a ‘representative’ of that audience? Do I know enough about the content to comment? Have the authors told me enough about the context of the work that I can make an assessment? Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM, Mavis B. Fundamentals of Scholarly Peer Review: A Workshop for Health Professions Educators on Practicing Scholarly Citizenship. MedEdPORTAL. 2021 Aug 2;17:11174.
  • 71. You are not alone… • Get help from others and give help to others, but ONLY ONCE YOU HAVE ASKED THE EDITOR IF YOU MAY DO THIS • Concerns: • Confidentiality and respect for authors • “Ghost reviewing” and exploitation of more junior reviewers, hidden from sight • In this – remember your rights, those of the authors, and those of the journal
  • 72. You are not alone… • Consider some options • Ask your supervisor/a more experienced colleague to include you in peer reviewing when they get approached • Approach a more senior colleague/supervisor to help you when you get asked to peer review • Develop peer review buddy systems (peer to peer)
  • 73. Peer review buddy systems (1) • Link in with “shut up and write” – writing support systems • Journal clubs also useful • A good place to start, even before you do peer reviews yourselves: • Make a commitment that you and your buddies will “peer review” reviews you yourselves receive from journals. • Assess the reviews you receive in terms of competence, respectfulness, constructiveness, etc; and • Suggest in your group ways in which the reviews could have been improved
  • 74. Peer review buddy systems (2) First, establish rules of confidentiality for your group Always get permission from the editor to use a peer system Step 1: Each (independently) sketch out areas to be covered in review Step 2: Distribute the tasks and each do your task Step 3: Review what one another have done Step 4: Assemble full review
  • 75. • Write in a supportive tone, but be definitive • Summarize your understanding of the work • Explain your overall impressions (recommendation / ratings) • Base your overall impressions on actual content • Indicate strengths and weaknesses, providing specific examples • Provide suggestions for improvement • Be detailed and clear • Explain any descriptors, such as “insufficient method” Tips for Writing Comments* * Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisor’s completed clinical evaluation reports. Med Educ 2008; 42:816-22. NOTE: THIS SLIDE IN ITS ENTIRETY IS COPIED FROM Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM, Mavis B. Fundamentals of Scholarly Peer Review: A Workshop for Health Professions Educators on Practicing Scholarly Citizenship. MedEdPORTAL. 2021;17:11174. https://doi.org/10.15766/mep_2374- 8265.11174
  • 76. A PLEA FROM A JOURNAL EDITOR Please do not undermine the peer review process while it is underway, BUT Do not undermine the process Please do give journals feedback on peer review experiences, good and bad Do give feedback Remember: we need to peer review peer review! Remember