Slides from Jisc webinar 18 Feb 2013

1. Peer Review and the development
of evaluative skills
David Nicol
Emeritus Professor of Higher Education
University of Strathclyde, Scotland
Visiting Professor, University of Ulster
Adjunct Professor, University of Swinburne, Australia
Consultant to JISC: Assessment and Feedback Programme
JISC Webinar: 18 February 2013

2. Plan for Session
 Brief introduction to peer review
 You experience of peer review – produce assignment
(5 mins) and review two peer assignments (10 mins)
 Reflection and discussion
 Drawing on case examples – engineering, sociology
chemistry

3. Engineering Design
Peer Project case study
 DM 100 Design 1: first-year class
 Dr Avril Thomson, Course Leader, Design
Manufacturing and Engineering Management (DMEM),
University of Strathclyde
avril.thomson@strath.ac.uk
 Caroline Breslin, Learning Technology Adviser,
University of Srathclyde
caroline.breslin@strath.ac.uk
Funded by JISC: see www.reap.ac.uk/PEER.aspx

4. Peer review
Definition of peer review
 Peer review is an arrangement whereby students
make evaluative judgements about the work of peers
and provide a written feedback commentary.
 In peer review, students produce as well as receive
feedback.

5. Introduction
 Research on peer review has been confounded by
three factors (i) an over-focus on peer assessment
rather than peer review (ii) a bias towards examining
the benefits of receipt of feedback reviews rather
than the production of feedback reviews and (iii)
studies examining general benefits of peer review
without distinguishing its component parts.
 Today the focus is primarily on producing feedback
reviews

6. Your assignment
 Write a convincing argument for having students review the
work of peers (the reviewing component only). Provide
evidence for your argument (from literature, logical, from you
own experience, convincingly anecdotal) and identify and
respond to any obvious counter-arguments.
 Criteria for good argument are: convincingness of argument (ii)
evidence in support of argument (iii) identification and
responses to obvious counter-arguments.
 Five minutes for this task Normally students can produce about
10-14 lines of text 
 Access to the task and instructions is here in google docs:
http://bit.ly/peerevalform
 Keep your audio on and please keep to time.

7. The peer review task
 Review and provide feedback comments on the work
of two peers using the given criteria.

8. Argument from peer 1
I think that students would gain understanding of their own work in
the process of reviewing the work of peers. When reading another
student’s work, the reviewer would more likely be able to see areas
where improvements could be made. It is often the case that it is
easier to identify others’ weaknesses than one’s own. When
reviewing the work of others, the student would engage in a process
of comparison with their own work. This leads to a form of reflection
otherwise not available. However, it could be argued that students
are not well-qualified to comment on the work of others. They do
not have the knowledge of the subject or the pedagogical training to
make valuable comments. This I do not agree with. Students are
often close to each other in their level of knowledge and writing and
would therefore be able to give constructive criticism. At the same
time, giving criticism would heighten awareness of their own
performance.

9. Argument from peer 2
Having students review the work of peers should be a regular
activity in higher education because if students do this they will see
the way others tackle the same assignment and they will learn and
get ideas from this. Also, when they review they will have to apply
some criteria and this will help them to understand these criteria
better. In my experience students often produce poor assignments
because they do not understand what is expected, not because they
cannot do the work. Indeed, when I organise peer discussion of
criteria before a task this results in better quality work. However, it
is clear that there might be problems of plagiarism as in reading
peers assignments it is likely that students will copy without owning
the ideas themselves. This could be tackled, however, by having
students review and just say what they would do to improve their
own assignment (if they had the opportunity) without actually
getting them to do it. In this way, they would provide evidence of
interpretation rather than copying.

10. Criteria/questions for the peer review
(i) Summarise the core of the argument written by each
peer in one sentence.
(ii) Identify and list what evidence is actually used to
support the argument?
(ii) Make one suggestion that would strengthen the
argument. Give a reason for your suggestion in a
sentence or two.
Tackle one review, then the next and submit
Peer assignment 1 is here: http://bit.ly/peerarg1
Peer assignment 2 is here: http://bit.ly/peerarg2
Time = 10 minutes = 5 minutes each review

11. Discussion and reflection
 You: Reflection on your experience – the learning
from peer review
 Me: presentation of some recent research findings
using ‘student quotes’
 Facilitators – manage chat discussion and highlight
questions and ideas

12. Reflection (1)
 What did you learn from this peer review exercise?

13. What was the most valuable aspect of the
reviewing process?
Rate each of the following on the following scale (where 0 is not valuable,
1 is of some value and 7 is very valuable).
0 1 3 5 7
1.Seeing how peers had approached this task
2.Engagement with the criteria/questions
3.Comparing the peer assignments with your own
4.Making evaluative judgements about others’ work
5.Writing the feedback commentaries
6.Comparing one peer’s work with the other
7.Thinking about changes to your own assignment

14. Research: what do students say?

15. Results: learning from RECEIVING reviews
Please give examples of what you learned from RECEIVING peer reviews
from other students (n=54)
Specific content mentioned: Depth of analysis needed, more numerical
data and figures, stronger rationale, how to structure it better etc.
Receiving peer reviews gave me insight into what others thought of my
work and gave me a direction to improve (reader response)
Where the PDS was confusing to understand (reader response)
Parts that I had previously missed were brought to eye such as market
competition (noticing)
The person who peer reviewed my PDS gave me positive feedback
which helped me a lot (motivational)
Not much, they...[the peer reviews]...weren’t very good (no value)

16. Results: learning from PROVIDING reviews
Please give examples of what you learned from PROVIDING peer reviews
of other’s work (n=47)
How to look at work critically that isn’t your own [critical judgement]
Thinking from a critical point of view [critical judgement]
I was given a greater understanding of the level of the work the
course may be demanding [attention to expectations/criteria]
Allowed me to see from an assessor’s perspective
[expectations/criteria]
When giving advice to people on theirs, it gave me greater perception
when reviewing my own work by listening to my own advice for
example [reflection/transfer]
I had a chance to see other peoples work and aspects of their work
that I felt were lacking in my work, this helped me to improve my work
[reflection/transfer]

17. Results: How you carried out peer review
Could you make any comments about how you carried out the
peer review? How did you evaluate the quality of the work to
provide a response to the peer review questions? (n=37)
I compared it to mine and ...and said how I would improve it
Partly by comparing my work to theirs
I tried to think about what I wrote and whether this product
design specification was better or worse

18. Focus groups
 How did you go about reviewing?
‘I read it through and compared it with what I had done to see if
they had put something I had not done and then I added it in if
they hadn’t. The four questions...[provided by the
teacher]...were useful as they provided a framework for the
review. If we hadn’t had the questions it would have been
difficult. I did the reviews separately and then answered one then
the other. The first was a better standard than the other – so I
used the ideas from the better one to comment on the weaker
one. I also read the guidelines in class when I did the peer review.
There were ideas from the good one that I hadn’t even thought
of in mine’

19. Results: reviewing
In the focus groups the effects of the review questions (criteria)
was probed further. Typical comments were:
You compare it (the other student’s work) to the criteria but
then in the back of your mind you’re comparing it to our own at
the same time.
I went down the questions and compared it to my own..I was
trying to think what has this person done. Have they put in more
effort or knowledge than me.
I went through the questions keeping my own in mind
You’ve got what you’ve done at the back of your mind while
going over theirs so you see where you’ve gone wrong without
anyone pointing it out so you learn it yourself
Reviewing is grounded in comparisons with students’ own work
(Nicol, Thomson and Breslin, 2013)

20. Summary
 Reviewing elicits multiple acts of evaluative judgement
1. Evaluate peer’s work against own
2. Evaluate one peer’s work against another (and own)
3. Evaluate work against given criteria to produce response
 The pre-condition for these effects
1 Students must first have produced an assignment in the ‘same
domain’ as those that they are asked to review
To what extent does your experience resonate with this finding?

21. Some thoughts about criteria/standards
 Students both create and apply evaluative criteria
1 Create criteria as they compare work with own (holistic
judgements)
2 Apply explicit criteria (analytic) to instances of practice when
the produce a written response (analytic judgements)
 ‘Through reviewing students generate richer criteria
than those provided by the teacher but sounder criteria
than those they might be able to formulate on their
own’ (Nicol, Thomson and Breslin, 2013)
Nicol, D., Thomson, A and Breslin, C. 2013. Rethinking feedback in higher education: a peer
review perspective. Submitted to Assessment and Evaluation in Higher Education

22. Reflection
 Why are these findings important in relation to
current debates about feedback in higher education?

23. Purpose of feedback
 Feedback should develop the students’ capacity to
make evaluative judgements about their own and
others work (Boud and Associates, 2010: Cowan,
2010; Sadler, 2010)
 Feedback should serve the function of progressively
enabling students to better monitor, evaluate and
regulate their own learning, independently of the
teacher (Nicol and Macfarlane-Dick, 2006: Nicol,
2009)

24. Limitations with received feedback
1. Can promote learning of scripted responses – students
dependent on teacher (Orsmond and Merry, 2011)
2. Assumes that ‘others are required to identify and
provide the information students need to learn and
that learning is driven by how others go about this’
(Boud and Malloy, 2012)
3. Any use of teacher feedback involves students in acts
of self-assessment (Nicol, 2012: Black and William,
1997)
4. Even if you require students to act on teacher
feedback it is still a transmission model
5. Practicality – teacher workload

25. What do the students say?

26. Focus groups
 What do you think is best for learning – giving or
receiving feedback?
I think when you are reviewing...[the work of peers]...it’s more a
self-learning process, you’re teaching yourself; well, I can see
somebody’s done that and that’s a strength, and I should
maybe try and incorporate that somehow into my work.
Whereas getting...[teacher]... feedback you’re kind of getting
told what to do; you’re getting told this is the way you should
be doing it, and this is the right way to do it. You’re not really
thinking for yourself.... I think...[reviewing]... would help you
not need so much of teacher feedback, if there was more of
this. Whereas, I think if you’re not being able to do...
[reviewing]... then you will always be needing more...[teacher
feedback]...

27. Focus groups
 What do you think is best for learning – giving or
receiving feedback?
‘For me it would probably be to give feedback because I think
seeing what other people have done is more helpful than getting
other people’s comments on what you have already done. By
looking at other people’s work you can see for yourself what you
have forgotten or not even thought about. When people give
feedback on yours they generally just talk about what is there.
They don’t say, well I did this on mine and you could put that in
yours.’

28. Peer review: a new perspective on feedback
 Students construct feedback ‘meanings’ for themselves
while they produce it for others
 Puts feedback processes in the hands of the student
 Reduces their need for teacher feedback
 Suggests another focus for teacher feedback – helping
students calibrate the quality of their own judgements
(reviews)
Nicol, D., Thomson, A and Breslin, C. 2013. Rethinking feedback in higher education: a peer
review perspective. Submitted to Assessment and Evaluation in Higher Education

29. Principles of good peer review practice
1. Ensure an atmosphere of trust and respect
2. Encourage critical engagement with criteria and standards
3. Ensure an assignment has been produced in the same domain as
those to be reviewed
4. Require well-reasoned explanations for reviews (not just marks)
5. Give practice in making holistic as well as analytic judgements
6. Facilitate dialogue around the object and quality of the review
7. Integrate self-reviews into peer review designs
8. Provide signposts that help students calibrate the quality of their
reviews
9. Encourage critical reflection on received reviews
Nicol (2013) available at http://www.reap.ac.uk/PEERToolkit/Design.aspx
www.reap.ac.uk/PEER.aspx

30. Feedback in professional and
workplace settings
1. In the professions, feedback never comes from a
single source: task is usually to evaluate, weigh up
and reconcile and respond to different and
sometimes contradictory feedback perspectives.
2. Professionals are not just ‘consumers’ of feedback
but also ‘producers’
Nicol , D. Thomson, A and Breslin, C. 2013. Rethinking feedback
practices in higher education: a peer review perspective.
Submitted to Assessment and Evaluation in Higher Education

31. Design decisions
1. Target task – factual or open-ended (design, computer
programme, essay, report etc)
2. Unit for task: individual, pair, group
3. Unit for review: individual, pair, group work
4. Matching reviewers: random, by ability, by topic
5. Number of reviews – more is better
6. Privacy: anonymous or known reviewer and/or author
7. Peer review rubric – not-given: guidelines: fixed
format
8. Review focus: holistic v analytic, content or process
9. Use of received reviews: drafts, self-review, new task
10. Requesting and responding to feedback
11. Grading: no marks, marks for participation, for
reviews, marks for self-review after peer review

32. References
Nicol, D., Thomson, A & Breslin, C. (2013). Rethinking feedback in higher education: a peer review
persepective, Submitted to Assessment and Evaluation in Higher Education.
Nicol, D (2013), Resituating feedback from the reactive to the proactive. In D. Boud and L. Malloy (Eds)
Effective Feedback in Higher and Professional Education: understanding it and doing it well, Routledge UK
Boud, D. and Associates (2010) Assessment 2020: Seven propositions for assessment reform in higher
education. Sydney: Australian Learning and Teaching Council (ALTC) Available from
www.assessmentfutures.com
Cowan, J. (2010) Developing the ability for making evaluative judgements, Teaching in Higher Education,
15(3), 323-334.
Nicol, D. and Macfarlane-Dick, D. (2006), Formative assessment and self-regulated learning: A model and
seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218
Nicol, D (2009) Assessment for learner self-regulation: enhancing achievement in the first year using learning
technologies, Assessment and Evaluation in Higher Education, 34(3), 335-352.
Nicol, D (2010) From monologue to dialogue: improving written feedback in mass higher education,
Assessment and Evaluation in Higher Education, 35:5, 501-517
Nicol, D (2010) The foundation for graduate attributes: developing self-regulation through self and peer
assessment, QAA Scotland, Enhancement Themes. Available at:
http://www.enhancementthemes.ac.uk/resources/publications/graduates-for-the-21st-century
Nicol, D (2011) Developing students’ ability to construct feedback, QAA Scotland, Enhancement Themes.
Available at http://www.enhancementthemes.ac.uk/resources/publications/graduates-for-the-21st-century
Roscoe, R. & Chi, M. (2008) Tutor learning: the role of explaining and responding to questions, Instructional
Science, 36, 321-350.
Sadler, D.R (2010) Beyond feedback: Developing student capability in complex appraisal, Assessment and
Evaluation in Higher Education, 35:5, 535-550