Peer review in scholarly journals

@SAJS_Official @ASSAf_Official
#PeerReview
Peer review in
scholarly journals
PEER
E
V
I
E
W
13 September 2023

#PeerReview
13 September 2023
Peer review in scholarly journals
P E E R
E
V
I
E
W
PROGRAMME
Welcome and introduction
What is peer review and what is it not? Jemma Finch, UKZN Associate Editor: SAJS
What reviewers should know about peer review Nkosinathi Madondo, MUT Associate Editor Mentee:
SAJS
What authors should know about peer review Shane Redelinghuys, Wits Associate Editor Mentee:
SAJS
Q&A
What everyone should know about AI and peer
review
Martin Bekker Wits University
Where to start with peer review Leslie Swartz, SUN Editor-in-Chief: SAJS
Q&A
Thanks and closure

#PeerReview
13 September 2023
P E E R
E
V
I
E
W
Meet the presenters

#PeerReview
13 September 2023
P E E R
E
V
I
E
W

What is peer review
and what is it not?
Jemma Finch
#PeerReview
13 September 2023
Peer review in
scholarly journals
P E E R
E
V
I
E
W

#PeerReview

‘Peer review is secretive. Only those present
in the deliberative chambers know exactly
what happens there’ (Lamont, 2009, p. 2)
#PeerReview
?
?
?
?
?
?
?
?
?
?
? ?
?
?
?
?
?
?
?
?

#PeerReview
Let’s make things more transparent….
• What is peer review?
• What is it not?
• What are we striving for?

What is peer review?
• Check quality, novelty, significance and
presentation
• Constructive
• Collegial and respectful
• Detailed and comprehensive
#PeerReview

What is peer review?
• Independent, unbiased
• Anonymous (depending on the model)
• Confidential
• Professional service
#PeerReview

What is it not?
• Gatekeeping
• Spelling and grammar check
• Opinions unaccompanied by evidence
• Personal
• Paid (usually)
#PeerReview

What is it not?
• A chance to:
• Pull apart other people’s work
• Show how clever you are
• Advance competing interests
• Help your buddies
• Boost your citations
#PeerReview

What are we striving for?
• Diverse reviewers: origin, gender, career
stage
• Transparency
• Fairness
#PeerReview

Further reading
• Small Pond Science Blog ‘Why I don’t have trouble finding peer reviewers’
https://smallpondscience.com/2022/07/20/why-i-dont-have-trouble-finding-peer-
reviewers/#more-27097
• Stephen Heard’s Scientist Sees Squirrel Blog ‘How (as an editor) I choose lists
of reviewers’ https://scientistseessquirrel.wordpress.com/2018/12/20/how-as-an-
editor-i-choose-lists-of-reviewers/
• Lamont 2009: How Professors Think: Inside the Curious World of Academic
Judgment. Harvard University Press.
• Day & Gastel 2012: How to write and publish a scientific paper. Cambridge
University Press.
• Cargill & O’Connor 2013: Writing Scientific Research Articles. Wiley
Blackwell.
#PeerReview

What reviewers need to
know about peer
review?
nathi madondo
13 September 2023
#PeerReview
Peer review in
scholarly journals
P E E R
E
V
I
E
W

Mytalkwill beframedaroundtheidea
oftakingthescholarship forward
#PeerReview

Taking the scholarship forward
#PeerReview
A cursory report cannot
A detailed report can
In a way, not only reviewing the quality of
the manuscript but also your quality as a
reviewer (Boughey, 2022).

What isthe quality of a
manuscript?
Well presented and clear argument
Topic has the potential to inform current
debate
Methodological coherence
Conceptual depth
#PeerReview

What quality is not?
Unsubstantiated claims – sweeping
statements
Some authors try to make an easy
publication to earn subsidy
#PeerReview

The paper to review should be in the
area of expertise of the reviewer to be
able to judge its quality
To avoid cursory report
#PeerReview
To judge quality …

So, to review is to take the scholarship
forward
#PeerReview

Concluding thoughts
A big challenge to take the scholarship
forward - neo-liberal agenda
#PeerReview

References
Boughey, C. (2023, March 31). NRF rating process
with Professor Chrissie Boughey [video file]. You
tube. https://youtu.be/uIqdjb3z1l8
#PeerReview

What authors should know about
peer review
Shane Redelinghuys
#PeerReview
13 September 2023
Peer review in
scholarly journals
P E E R
E
V
I
E
W

Initial assessment
‘Under review’
2-16 weeks
~100 days

Know your publisher’s policies
• Manuscript submission instructions (formatting, referencing
style, word limit)
• Attention to detail will improve the aesthetics of your manuscript
– simplify reading to appreciate the content
• Streamlines the review process, minimise turnaround time
• Publishing agreements (e.g., open access, publication fees) &
ethical principles

• Is it acceptable?
 Yes!
 Won’t affect the handling of your paper or the review process
 Do so respectfully
 First, use the online tracking system of the journal to track status
 Will hear back very soon if manuscript not suited for the journal
 ‘Under review’ – reviewers have been invited / waiting for
reviewers to accept review / waiting for reviewers to submit their
review reports / reviews being assessed by AE
Follow-up on submission

• When is a good time to do so?
 Subject matter – dependent, technicality of your paper
 After 3-6 months
Follow-up on submission
An analysis of ~3 million papers indexed in PubMed from 1981 to 2015
indicated that the median time-until-acceptance has remained consistent at
around 100 days (i.e., approx. 3 months)

• A platform on which researchers tell others about their experiences with a
particular journal
• A tool to help you determine which journal to submit to, based on researcher-
provided statistics:
 Time until first review received
 Total handling time
 Time until desk rejection (rejection without review)
 Number of review reports received
 Quality of reviews (ranked out of 5) (subjective)
 Number of rounds of review
SciRev – An extra layer of transparency?

• Don’t take it personal
• Manuscript rejection – does not mean it is the end of the road for
your paper
• Consider why – take reviews into account – May be too
specialised/general
• Decide if the work is worth resubmitting (most of the time it is)
• Improve work by taking a different approach with your analysis –
brainstorm with peers
Rejection – what to do?

#PeerReview
Sources
• Peer review ethics: Six things every author should know
https://www.apa.org/about/division/digest/publishing/peer-review-ethics
• The history of publishing delays
https://blog.dhimmel.com/history-of-delays/
• The peer review process
https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-
review/the-peer-review-process.html
• Stichting SciRev
https://scirev.org/

Martin Bekker
School of Electronic and Information Engineering, University of the Witwatersrand
Martin Bekker // martin.bekker@wits.ac.za
LLMs tools and you, the peer reviewer

Our time together
How do LLMs work?
What are they good at?
What are they bad at?
How to get the best out of LLMs
What do I do if something smells fishy?
What level of AI-support is acceptable?
How should AI be referenced?
(How) should prompts be shared?

Warning: Strong Language and Sensitive Content
The following presentation contains examples of language and content that may be
offensive, including swearing, violent language, and racist undertones. These examples are
presented solely for the purpose of analysis and discussion, highlighting the impact of such
language in certain contexts. The intention is not to endorse or promote any form of
discrimination, violence, or offensive behaviour.
It is important to approach these examples with sensitivity and understanding, recognizing
that they may cause discomfort or distress. We encourage you to exercise discretion when
considering the appropriateness of this content for your audience.
Please note that the presentation aims to foster awareness, education, and critical thinking
regarding the use of offensive language and its potential effects. We strongly condemn any
form of discrimination, violence, or hate speech, and advocate for respectful and inclusive
communication.
By proceeding with this presentation, you acknowledge that you have been provided with
this warning and disclaimer, and that you understand the purpose and context in which
these examples are being presented.

Large Language Models
Many available today
A corpus of text
A stochastic model
A base model
Safety layers
Reinforcement Learning through Human Feedback

Strengths // What it is good at
Repackaging of ideas
Summating text
Style transfer
Fixing grammar
Brainstorming / Dinner Ideas / Essay Ideas

Weaknesses
Mathematical model ≠ not a linguistic model
Stochastic parrot
Hallucinations
Causal reasoning
Jailbreaking

Hidden Harms
Human Labour Exploitation
Environmental
Monopolising power
(Fundamental?) Un-understandability
Commodity Fetishism

Threats
Claims of IP non-consent (and theft)
Racism, sexism, hate speech (based on training biases)
Poor advice (negligent, dangerous, harmful)
Mass-scale cheating

How to get the best out of LLMs
Only use for what it’s good (narrow!)
Only use once the thinking’s done (generally)
Never let it have the final say

• GTPZero
• Quillbot.com
• Quetext.com
• AIdetector.com
• Scale.ai
• Originality.ai
Prompt: Write an essay on {insert topic} in {insert wordcount} in
the style of an 18-year old second langue speaker. Use at least 10
unusual words, and two sentences must be translated directly
from Afrikaans, using Afrikaans grammar.
Academics (ad students!) are human
It’s dangerous to think one can outsmart tech (most of the time)
Critical thinking under threat

What do I do if something smells fishy?
1. Understand where your publication stands vis-à-vis LLMs (default
should be ‘none’
2. Seek acknowledgements
3. Search (a few) citations
4. Talk to author

How should AI-help be referenced?
If usage permitted -
Norms (e.g. APA)
Acknowledgements
Level
Github / Repository
{but consider how would
journal check for
paraphrased plagiarism?}

Prompts
Online repository
Google Drive
Github
Supplemetary documentation
Platform, date, version (if possible)

Use Ban
Grammar and spelling assistance / Proofing
Co-creation / Editor
Brainstorming and planning
‘Over to you’ / No holds barred
Show your prompts
Responsible for your work

Happy reviewing

Where to start
with peer review
Leslie Swartz
#PeerReview
13 September 2023
Peer review in
scholarly journals
P E E R
E
V
I
E
W

A recap,
and some
principles
Peer review is central to the academic enterprise, and
is as important as writing your own articles
There is currently a crisis in peer reviewing – it is hard
for journals to find peer reviewers, and without peer
review journals cannot continue to function
We are all peers
We all have a responsibility to keep the peer review
system going
A rule of thumb: for every time I send an article for
review, I should be prepared to review another article

What do I want from a peer reviewer?
• I want someone who is
• Competent
• Constructive
• Fair
• Kind
• Clear
• Focused on making my work better
• Not trying to make me a version of themselves

“I am not good enough”
Academics and impostor
syndrome – who do NOT
think that they are
impostors?
The academic hierarchy
(“only professors….”)
What do you really need to
know, and what don’t you
need to know?
It is fine to point out the
limitations of your
knowledge in your review
(the editor may well have
chosen a range of
competencies
You don’t have
to pretend

“I’m not
sure I am
the right
person to
review
this”
• Make a list of what you think you can and
cannot do
• How important is what you can do to
helping the author and the journal?
• How much of a barrier is what you can’t do
for your ability to be helpful?
Read the paper quickly, and then:
• Don’t be shy to contact the editor and raise
any concerns – we editors are grateful to
you and want to work with you!
If still in doubt

Your role
as a
reviewer is
that of a
‘peer
mentor’
(Way, et
al, 2021)
You are not expected to know everything,
but you can make it clear what you do and
don’t know
Four main
areas:
Do I understand the
methods?
Do I have a good idea of
who the audience is for
this journal? Can I stand
as a ‘representative’ of
that audience?
Do I know enough about
the content to
comment?
Have the authors told
me enough about the
context of the work that
I can make an
assessment?
Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM,
Mavis B. Fundamentals of Scholarly Peer Review: A
Workshop for Health Professions Educators on Practicing
Scholarly Citizenship. MedEdPORTAL. 2021 Aug 2;17:11174.

You are not
alone…
• Get help from others and give help
to others, but ONLY ONCE YOU HAVE
ASKED THE EDITOR IF YOU MAY DO
THIS
• Concerns:
• Confidentiality and respect for
authors
• “Ghost reviewing” and
exploitation of more junior
reviewers, hidden from sight
• In this – remember your rights, those
of the authors, and those of the
journal

You are
not
alone…
• Consider some options
• Ask your supervisor/a more
experienced colleague to
include you in peer reviewing
when they get approached
• Approach a more senior
colleague/supervisor to help you
when you get asked to peer
review
• Develop peer review buddy
systems (peer to peer)

Peer review buddy systems (1)
• Link in with “shut up and write” – writing support systems
• Journal clubs also useful
• A good place to start, even before you do peer reviews
yourselves:
• Make a commitment that you and your buddies will “peer
review” reviews you yourselves receive from journals.
• Assess the reviews you receive in terms of competence,
respectfulness, constructiveness, etc; and
• Suggest in your group ways in which the reviews could
have been improved

Peer
review
buddy
systems
(2)
First, establish rules
of confidentiality for
your group
Always get
permission from the
editor to use a peer
system
Step 1: Each
(independently)
sketch out areas to
be covered in review
Step 2: Distribute the
tasks and each do
your task
Step 3: Review what
one another have
done
Step 4: Assemble full
review

• Write in a supportive tone, but be definitive
• Summarize your understanding of the work
• Explain your overall impressions (recommendation /
ratings)
• Base your overall impressions on actual content
• Indicate strengths and weaknesses, providing specific
examples
• Provide suggestions for improvement
• Be detailed and clear
• Explain any descriptors, such as “insufficient method”
Tips for Writing Comments*
* Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisor’s
completed clinical evaluation reports. Med Educ 2008; 42:816-22.
NOTE: THIS SLIDE IN ITS ENTIRETY IS COPIED FROM Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM, Mavis B. Fundamentals of Scholarly Peer Review: A
Workshop for Health Professions Educators on Practicing Scholarly Citizenship. MedEdPORTAL. 2021;17:11174. https://doi.org/10.15766/mep_2374-
8265.11174

A PLEA
FROM A
JOURNAL
EDITOR
Please do not undermine the
peer review process while it is
underway, BUT
Do not
undermine
the process
Please do give journals feedback
on peer review experiences,
good and bad
Do give
feedback
Remember: we need to peer
review peer review!
Remember

Peer review in scholarly journals

Recommended

Recommended

More Related Content

Similar to Peer review in scholarly journals

Similar to Peer review in scholarly journals (20)

More from Academy of Science of South Africa

More from Academy of Science of South Africa (20)

Recently uploaded

Recently uploaded (20)

Peer review in scholarly journals