Loureiro,Pombo&Moreira_peer assessment_iodl&icem2010

The quality of peer assessment in a wiki based online context

The quality of peer assessment in a wiki
based online context:
a qualitative study in a post-graduation module

Maria João Loureiro

IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira
CIDTFF, DE, Universidade de Aveiro


topics

Context
Course organisation (Distance Education)
Quality of peer assessment (results)
Final thought
Contributions



context of the study - Portugal

Oporto
Aveiro
Coimbra



context (U. Aveiro)
• created in 1973
• 10 to 11.000 students
• courses (grad and pos-grad)
teachers education
sciences, math, …
engineering
health
…
• member of the European
Consortium of Innovative
Universities



context
PhD program in Multimedia in Education module
Distance Education (DE)
blearning regime
face-to-face and at a distance group work
module aim - develop research competences related with
literature reviewing
search, select, systematize, synthesize and literature related with DE;
communication, collaborative work and assessment competences (self-
and peer- assessment)



Context (literature)
focus on learner self-directed and collaborative learning ->
increase attention to alternative/authentic/integrated
assessment methods
HE assessment methods are quite narrow – assessment
students’ outcomes test based – students are not implicated
peer assessment (PA) can empower students (critical thinking,
reflection)
reliability and validity of PA – no consensus
literature on the quality of online (web 2.0) peer assessment
scarce
Boud and Falchikov (2007), Joordens et al. (2009), Li et al. (2009, 2010), Peng (2010),
Rourke et al. (2008), Sluijsmans et al. (2004), Topping (2000, 2008, 2010),
Zundert et al. (2010)


PA benefits:
promote higher order thinking, developing students’ critical
thinking, communication, lifelong learning, problem solving…
increase the amount of feedback, from teachers and from
peers
enhance students’ sense of ownership, responsibility and
students’ motivation, since they find it useful, attractive and
enjoyable
support active and autonomous learning
can avoid the involvement of free-riders in group work, i.e,
facilitate the identification of individual contributions


PA pitfalls:
students’ attitudes toward PA may hinder the above
mentioned benefits –students’ unconfident towards PA -
judging friends – increase students’ anxiety and resistant
toward PA
PA is time-consuming – training, preparation, and
monitoring are needed
PA can be subjective – students might be too hard, critical or
boasting, or not used to assess their peers
PA advantages increase – formative feedback


course organisation and development
1st week online
- Diagnosis of students’ perceptions about how to do literature
review and about “Distance Education” (individual).
- Familiarisation with the module guide and discussion of the
proposed activities, including assessment (individual).
- Individual readings of recommended literature and or other
materials related with theoretical and methodological issues to
be taken into account to do a literature review.
- Use of the synthesis sheet – instrument developed to help
students to analyse and document their readings (individual).



1st face-to-face session
- Teacher, tutor and students’ presentation and definition of
work groups (classroom).
- Negotiation of the module activities and schedule and of the
assessment framework (classroom).
- Session about online literature search tools (made by a
library technician).
- Presentation of tips on how to do a literature review (teach).
- Discussion and syntheses of the students’ readings about how to
do a literature review (groups).
- Information search (groups).


2nd weekonline
- 1st ind.reflection about the ongoing activities and students’
learning
- Definition of the literature review themes related with DE and of
the work plan (groups)
- Formative assessment of the work plan (teachers)
3rd and 4th weeksonline
- 2nd ind.reflection about the activities and the developed
competences
- Development of the 1st version of the literature review (gr.)
- Formative assessment of the ongoing work (teachers and PA)
- Revision of the paper and preparation for its presentation
(groups).


2nd face-to-face session
- Presentation, discussion and assessment (teachers
and peer summative assessment) of the groups work.
- Self and peer assessment of the developed
collaborative competences (intra group) and final
reflection concerning students own learning during the
module (individual)
- Module assessment (individual).


Methodology
Qualitative approach - study recent phenomena (like
the quality of online PA) in their own contexts
Data gathering and analysis techniques:
observation mediated by the wiki (PA was delivered by the
students using the wiki) – content analysis (2 researchers,
reliability 100%, external validation)
inquiry (a questionnaire was applied to evaluate the module
in the final face to face session) – descriptive statistics


Criteria Indicators
Use of the negotiated Students used the negotiated criteria or words with the same
criteria significance explicitly.
Adequacy of the chosen
Concepts are used with rigour and differentiated.
vocabulary
PA text includes criticism (both positive and negative), pointing
criticisms out aspects that could be improved, without making
Provision suggestions.
of PA text presents questions for reflection concerning, for
questions
constructiv example, the theme or the development of the literature review.
e feedback suggestions PA text includes suggestions that can lead to the improvement
for of the literature review, like, new readings, proposing ways to
improvement systematize the corpus of the review....
the PA includes a score which should be in accordance with the
Fairness of the score
given feedback
Adequacy of the final A synthesis of the main positive and negative aspects of the
considerations review is presented. 2010, Anadolu University – Loureiro, Pombo and Moreira
IODL & ICEM


Examples
negotiated criteria
“(…) the topics and sub-topics that underlie the work are well framed
[i.e. the structure is adequate]. Its coherence and rigour are remarkable
and the group demonstrates synthesis competences”.
adequacy of the chosen vocabulary
“the paper does not present significant gaps in what concerns cohesion,
textual coherence or even orthographic or syntactic mistakes.”
constructive feedback
“the work does not include the research question that guides the
development of the work”



Adequacy of the final considerations

Fairness of the score

Provision of constructive feedback
(suggestions for improvement)
(questions)
(criticisms)

Adequacy of the chosen vocabulary

Use of the negotiated criteria

0 20 40 60 80 100 %


Self and peer assessment of develloped competences
(intra-groups) should be published in the module's site

Self and peer assessment of the work groups should be
published in the module's site

The personal reflections should be published in the
module's site

Col. work should be assessed only by teachers

Col. work should be assessed individually, by teachers and
by colleagues

Col. work should be assessed with several components

Col. work should be assessed as a whole with the same
mark for all elements
0 2 4 6 8 10 12 14 16
Num ber of students

completely disagree disagree agree completely agree don't know


Final thoughts
PA could have more quality (suggestions,
questions, …)
Results
In line with the literature – lack of quality can be explained
by some of the pitfalls of PA (students’ attitudes towards PA,
uncomfortable, lack of confidence, unreliability + fear of
being exposed in the social web – in this case while
agreement about publishing PA)
PA needs training


Contributions
framework to assess the quality of PA in web 2.0
online contexts,
for some of the criteria, quality PA can be achieved
reports students’ perceptions and attitudes towards
the assessment of collaborative work, including PA



Thank you for your attention

questions???



courses
organization
Ramos (2004)


Loureiro,Pombo&Moreira_peer assessment_iodl&icem2010

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Mehr von Maria Joao Loureiro

Mehr von Maria Joao Loureiro (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Loureiro,Pombo&Moreira_peer assessment_iodl&icem2010