Building a Bridge Between Theory and Practice in Formative e-Assessment

Building a Bridge Between Theory and
Practice in Formative e-Assessment

e-Assessment in
Practice,
12 November,
Cranfield

Scoping a Vision for
Formative e-Assessment

• Commissioned by JISC
• June 2008 to January 2009
• Norbert Pachler, Harvey
Mellar, Caroline Daly, Yishay
Mor, Dylan Wiliam,
Emmanouil Agianniotakis
• http://feasst.wlecentre.ac.uk/

2
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Methodology
Desk research
Literature review
Comparing frameworks
Five Practical Enquiry Days
Combination of collaborative reflection, report back
from team, and guest plenaries
Launch day, three Pattern workshops, developers' day

3

What is formative e-assessment?
Formative?
‘practice’ assessment /
serial summative assessment synonymous with learning

The e?
The use of digital
means to support

Alternative view:
Formative features of assessment,
which are afforded by specific
features of digital media 4

OK, so what do we mean by
Formative Assessment?
“An assessment functions formatively when
evidence about student achievement elicited by
the assessment is interpreted and used to make
decisions about the next steps in instruction that
are likely to be better, or better founded, than the
decisions that would have been made in the
absence of that evidence”
(Dylan Wiliam)
5

Formative = feedback + moments of
contingency
"... These create "moments of contingency," in
which the direction of the instruction will
depend on student responses. Teachers
provide feedback that engages students, make
time in class for students to work on
improvement, and activate students as
instructional resources for one another."
(Leahy, Lyon, Thompson, and Wiliam 2005)
6

Participatory Methodology for
Practical Design Patterns
• Problem
– Acceleration → need for effective protocols for sharing
of design knowledge

• Context
– interdisciplinary communities of practitioners
engaged in collaborative reflection on a common
theme of their practice.
Son, this was my
dad's mobile.
I want you to 7
have it.

Problem: The Design
Divide

the gap between those who
have the expertise to develop
high-quality tools and
resources and those who
don’t (Mor & Winters, 2008*)

the critical resource is not the capacity to
produce, but the knowledge to do it right.
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 8

Solution...
(in architecture)


What is a pattern?
• At is simplest, it is a
– Generalised solution to a problem
– Follows a specific structure
Cookbook: ingredients,
procedure, expected
What are we trying to
outcomes
achieve / solve?

Problem Solution
Context

When, Where,
Who
10

A burda pattern..

Season: Fall
For: Women
Garment Type: Dress
Style: Classic, Evening Wear, Romantic
Material: Taffeta

“if I copy a dress, I can only create
the same dress. If I have a pattern, I
can create many dresses” (Yim Ping
LENDEN)

11

As for
software

12

In learning / e-learning..

13

The Participatory Pattern
Workshops Methodology


Collaborative reflection workshop
Facilitate on-going design-
level conversation between
designers and practitioners
involved in diverse aspects
of the problem domain.

Open, trusting and convivial.
And at the same time
Critical, focused and output-directed.


Collaborative reflection workshop


Case Stories Workshop
Engender collaborative
reflection among
practitioners by a
structured process of
sharing stories of
successful practice.


Pattern Mining Workshop
Shift from anecdotes to
transferable design
knowledge by identifying
commonalities across
case stories, and
capturing them in a
semi-structured form.


Future Scenarios Workshop
Validate design patterns
by applying them to
novel real problems in
real contexts.


A few cases
• Creature of the week
• CoMo
• Post 16 String Comparison
• Open Mentor
• ...

20

Creature of the week (Judy
Robertson)
Situation
large class (138), first and second year computer science students.
assignment: create a virtual pet in Second Life.

Task
− Engage and motivate the students
− show examples of good work which others could
learn from
− show students their work is valued.
− build a sense of community.
21
http://purl.org/planet/Cases/creatureoftheweek

22

CoMo (Niall Winters, Yishay Mor)
Situation
− Royal Vet College.
− Hospital rotations as part of their training.
Task
− Allow students to capture critical incidents in text
and image.
− Support sharing of clinical experiences and co-
reflection.
23
http://purl.org/planet/Cases/CoMo

24

Post 16 string comparison (Aliy
Fowler)
Situation
− Grammar school been piloting the ‘string comparison’ approach to
language teaching at post-16 for AS and A2 level students.
− Sixth Form level, grammatical consolidation and whole-sentence
translation.

Task
− Allow students to practise written language
independently and receive feedback on errors in
order to improve their language skills.
http://purl.org/planet/Cases/Post16stringcomparison
25

Solution
A bespoke string (sequence) comparator was
designed; uses fine-granularity sequence
comparison to compare correct language
strings to a user’s answer. Students answer
questions and the comparator marks up errors
in their input using colour coding (and font
style) to highlight the different types of error. If
an answer contains errors the student is given
a second attempt in which to correct the
submission based on the feedback received.
26

Open mentor (Denise Whitelock)

http://purl.org/planet/Cases/OpenMentor
27

A few patterns..

• Try Once, Refine Once
• Feedback on Feedback
• Classroom display

28

Try Once,
Refine Once

(Aliy Fowler)

http://pul.org/planet/Patterns/TryOnceRefineOnce

29

Problem
Lack of immediate feedback for students
leads to fossilisation of errors and
misconceptions

providing immediate feedback in an
iterative fashion can also hinder
effective learning since students are
able to "grope their way" step-by-step
to a correct solution without
necessarily having to think about each
answer as a whole.
30

Context
Class size
− Large (30-300)
Content
− Skills facts
Mode of instruction
− Blended / on-line. Computer tested.

31

Feedback
on
Feedback

(Linda
McGuigan)

33
http://purl.org/planet/Patterns/FeedbackonFeedback

Good feedback should - Problem
Alert learners to their weaknesses.
Diagnose the causes and dynamics of these.
Include operational suggestions to improve the learning
experience.
Address socio-emotive factors.

Tutors know this, but are pressed for
time. Or not aware of their feedback
strategies

Large teaching organisations are not equipped to
provide tutors with personal feedback on their
teaching 34

Context
Large scale, technology supported, graded
courses
− many tutors instructing many students.
Feedback is mediated by technology that allows it
to be captured and processed in real time
Topic of study is subject to both grading and
formative feedback.

35

Solution
Embed a mechanism in the learning and teaching system
that regularly captures tutor feedback, analyses it, and
presents them with graphical representation of the types of
feedback they have given. Ideally, this should also include
constructive advice as to how to shift from less to more
effective forms.
In computer supported environments (e.g. VLEs), this
mechanism could be integrated into the system, providing
tutors with immediate analysis of their feedback, as well as
long-term aggregates.

36

Classroom Display

http://purl.org/planet/Patterns/Classroomdisplay 37

Problem
Using learner
generated content..
Rewards participation.
Relates to learner's personal
experiences.
Window on student conceptions.

Need to collate works in a single easy to access
location.
Learners uncomfortable about presenting their work in
public
Legal or other restrictions on sharing work.
38

Context
Class size:
− Small / medium (6-60)
Mode of instruction:
− Blended (preferable)
Time frame
− Continuous, over a period
Pedagogy
− Involves construction / media production
39

Solution

40

Augmented domain map

41

Example scenario
When using Try Once Refine Once there is a risk
Once,
that high-achievers do not receive feedback.
So -
Use Showcase Learning to celebrate students’
work and provoke feedback from peers and
tutors.
Use Feedback on Feedback to alert tutors to the
problem.

42

Conclusions
Tip of the iceberg
Practitioners (educational / software)
acknowledge the value of patterns, when served
with side dishes of cases + scenarios
Collaborative elicitation of patterns from cases
could be a potent form of professional
development.

43

Thank you
The pattern language network project:
http://patternlanguagenetwork.org

The learning patterns project:
http://lp.noe-kaleidoscope.org/

The formative e-assessment project:
http://feasst.wlecentre.ac.uk/
Yishay Mor
This presentation: people.lkl.ac.uk/yishay
http://talks.yishaymor.org/e-assessmentinpractice2009 yishaym@gmail.com
http://www.slideshare.net/yish/eain @yishaym

44

Building a Bridge Between Theory and Practice in Formative e-Assessment

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Building a Bridge Between Theory and Practice in Formative e-Assessment

Similar to Building a Bridge Between Theory and Practice in Formative e-Assessment (20)

More from Yishay Mor

More from Yishay Mor (20)

Recently uploaded

Recently uploaded (20)

Building a Bridge Between Theory and Practice in Formative e-Assessment

Editor's Notes