A session I gave to the University of Glasgow as part of our CPD events

1. Using Moodle for Peer Review
Sarah Honeychurch
@NomadWarMachine

2. Why use Peer Assessment/Review?
"..if we want students to develop critical thinking, judgement
and autonomy in assignment production they should be
provided with high-level evaluative experiences similar to
those of experts. Peer review, students evaluating and
commenting on each other's work, is one way to achieve
this.”
http://www.reap.ac.uk/PEER.aspx

3. Peer assessment or peer review?
• Definitions
– Assessment – gives grade
– Review – gives feedback but no grade
– Two aspects:
• Writing reviews of others’ work
• Receiving reviews on own work from peers
“… many students dislike being asked to mark other students work and receiving
marks from other students. There are many reasons for this, including a belief by
students that their peers do not have the expertise to mark reliably. In
contrast, students are usually very positive about the benefits of peer review
processes where marking is not involved.”
http://www.reap.ac.uk/PEER/Research.aspx

4. Peer assessment?
• Benefits:
– Students (say that they) want more feedback
– Students (say that they) want prompt feedback
– Does not take (much) extra staff time
• Problems:
– Concerns about the reliability of student marking (quantity over
quality?)
– Students don’t like grading peers

5. Peer Review
• Benefits of receiving peer review
– Accessible language of peers
– Variety of feedback
– Conflict of feedback mirrors professional life (must process and
decide what to accept/reject)
• Benefits of giving peer review
– Constructing feedback requires cognitive engagement
• Knowledge construction
• Engagement with assessment criteria
• Developing discipline specific writing skills
– Variety of approaches can stimulate self-reflection
• Issues
– Summative assessment?

6. Online Tools for peer
assessment/review
• Moodle Workshop
• New LTI tool: ACJ (Adaptive Comparative Judgement)
• Other options
– Moodle Wiki
– Moodle Forums

7. Moodle Workshop
• Similar student interface to Moodle Assignment
• Options for self, peer and tutor review
• Options for review or assessment
• Option to assign tutor mark for submission and for review

8. Moodle Workshop
• Grading Strategy:
– Accumulative – Comments and grades given
– Comments – Comments but no grades
– Number of errors – Comments and yes/no responses
– Rubric - Marked against specified criteria
• If using grades, can set %ages of grade:
– Given to them for their submission
– Given to them for their peer assessment

9. Moodle Workshop
• Submission settings
– Can add clear guidance of how/what to submit
– Can restrict file types
• Assessment settings
– Can include self assessment
– Can include an example assessment
• Feedback settings
– Can ask/require students to give overall comment
– Can allow upload of feedback files
• Standard Moodle options for groups/restricted access

10. Moodle Workshop
“Gotchas” manual selection through phases

11. Moodle Workshop
“Gotchas” – look for greyed out tick marks

12. Moodle Workshop
“Gotchas” – look for red crosses

13. Adaptive Comparative Judgement
• Thurstone (psychophysics); Pollitt
• Students are presented with (random) pairs of
submissions and judge better/worse
• Software uses iterative and adaptive algorithm to sort
submissions
• Staff can “seed” submissions to provide grade boundaries
https://learn.gla.ac.uk/acjdemo/index.php
Pollitt, A (2012) The method of Adaptive Comparative
Judgement. Assessment in Education: Principles, Policy &
Practice. 19: 3, 1-20. DOI:10.1080/0969594X.2012.665354

14. Adaptive Comparative Judgement
• Benefits
– Timely feedback
– Easier than peer assessment
– Little staff time
– Has inter and intra rater reliability
• Issues
– Trust in new system
– Complex criteria
– Rich feedback
http://www.cambridgeassessment.org.uk/Images/232694-
investigating-the-reliability-of-adaptive-comparative-judgment.pdf

15. Adaptive Comparative Judgement
• Learning Tools Interoperability (LTI)
• Add external tool
• ACJ Experimental
• Comparisons:
– Students may submit
– Students must submit
– Teacher uploads
• Teacher uploads review guidelines

16. Adaptive Comparative Judgement
• Submission options:
– Paste web page url
– Paste YouTube url
– Type into HTML editor
– Upload PDF
– Upload image
– Type/paste source code
• Ranking options
– Teachers
– Students (peers)
• Scoring options
– Based on final rank
– Bsed on judgement participation
– Based on rank and participation

17. Discussion
• Do you trust students to peer review responsibly?
– Why/not?
• Do you think peer assessment or review are preferable?
• What about ACJ
– Would you be interested in using it?

18. Using Moodle for Peer Review
Sarah Honeychurch
@NomadWarMachine
ACJ: Niall.Barr@Glasgow.ac.uk