More Related Content Similar to Learning from US research on assessment and feedback (20) More from University of Strathclyde (20) Learning from US research on assessment and feedback1. Learning from US Research on
Assessment and Feedback
John Kleeman, Questionmark
CETIS Assessment and Feedback : Feb 2nd 2011
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2. Introduction
John Kleeman
Founder and Chairman of Questionmark
www.questionmark.com
blog.questionmark.com
Twitter: @johnkleeman
Experienced e-assessment practitioner
Not an academic nor a psychologist
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 2
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
3. Which learning method will retain
information best a week later?
A. Study a subject, no test
B. Study a subject, take a multiple choice test, no
feedback
C. Study a subject, take a multiple choice test,
feedback after each question
D. Study a subject, take a multiple choice test,
feedback at the end of the test
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 3
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
4. Graph from Roediger & Butler : The critical role of retrieval
practice in long-term retention. Trends in Cognitive
Sciences 2010.
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 4
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
5. Which learning method will retain
information best a week later?
A. Study a subject, no test
B. Study a subject, take a multiple choice test, no
feedback
C. Study a subject, take a multiple choice test,
feedback after each question
D. Study a subject, take a multiple choice
test, feedback at the end of the quiz
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 5
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
6. Retrieval helps you learn
Francis Bacon, 1620
If you read anything
over twenty times, you
will not learn it by
heart so easily as if
you were to read it
only ten, trying to
repeat it between
whiles, and when
memory failed, looking
at the book.
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 6
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
7. Learning
Learning is only
useful if you can use
or retrieve the
learning
Receive Working Long term
information memory memory
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 7
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
8. Retrieval
• Question Research evidence
Cue • Other stimulus shows that retrieval
practice helps future
retrieval
• Retrieve from long
term memory
Search
• Act or respond
Act
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 8
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
9. Two randomly selected groups study a
reading passage. Group 1 studies it for 14
minutes. Group 2 studies it for 7 minutes
and takes a quiz/test
Who learns most?
A. Group 1
B. Group 2
C. Group 1 learn more immediately but retain less after a
week
D. Group 2 learn more immediately but retain less after a
week
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 9
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
10. Graph from Test-Enhanced Learning : Roediger
and Karpicke 2006. Psychological Science.
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 10
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
11. Two randomly selected groups study a
reading passage. Group 1 studies it for 14
minutes. Group 2 studies it for 7 minutes
and takes a quiz/test
Who learns most?
A. Group 1
B. Group 2
C. Group 1 learn more immediately but retain less
after a week
D. Group 2 learn more immediately but retain less after a
week
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 11
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
12. Retrieval practice key findings
Retrieval practice helps retention of learning
Questions/quizzes are a good way of giving
retrieval practice, other routes also possible
Unprompted questions (e.g. short answer) best
Multiple choice questions still useful
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 12
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
13. Learning in a Perfect World (a stylized view)
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 13
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
14. Learning in the Actual World
They don’t get it All!
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 14
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
15. Learning Curve
Purpose:
• Content repetition
• Memory retrieval practice
• Strengthens memory recall
Ouch! The
Forgetting
Curve
Knowledge/Skills
Experience
Experience
Experience
Questions
Questions
Learning
Learning
Time Learning
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 15
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
16. In the Actual World
They Forget It!
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 16
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
17. Slow the Forgetting Curve
Purpose:
• Memory retrieval practice
• Strengthens memory recall
• Reduce Forgetting Curve
Knowledge/Skills
Experience
Experience
Experience
Questions
Questions
Questions
Questions
Learning
Learning
Time Learning
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 17
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
18. Where does feedback fit in?
• Question Feedback most useful
• Other stimulus
Cue
on incorrect answers
• Retrieve from long
Some value on
Search term memory correct answers
• Act or respond
Act
• Correct misconceptions
Feedback • Re-encode
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 18
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
19. Feedback tip #1 – give feedback
Students only know they are wrong if you tell
them
Short answer questions give retrieval practice
without feedback, help learning more with
feedback
Multiple choice questions can be dangerous for
learning without feedback
Do include the correct answer, in most contexts,
just telling people they are wrong doesn’t help
learning
Applies also in summative exams
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 19
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
20. Feedback tip #2 – do retrieval first
Usually force people do the retrieval before you
give them feedback
For instance, don’t allow them to peek ahead to see the
right answer
Otherwise you are depriving them of the retrieval
practice
Exception when building understanding
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 20
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
21. Feedback tip #3 – give retrieval
practice later
If someone gets a question wrong, they are
deprived the positive learning effects of retrieval
So if you can, after giving feedback, give another
opportunity for retrieval practice (e.g. retake
questions on the topic)
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 21
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
22. Feedback tip #4 – get people to pay
attention
Feedback is only useful if learners pay attention
Give feedback in a way that people will attend
Consider attention span
Vary feedback methods
Review whether to give feedback on correct answers
Monitor effect of feedback
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 22
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
23. Feedback tip #5 – delay feedback if
you can
A slight delay on feedback is usually better than
immediate feedback
Probably due to the spacing effect in learning
Research not fully conclusive
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 23
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
24. A reminder of the impact of retrieval
and feedback
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 24
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
25. Some questions for you
Do people retain more by studying or by studying
and answering questions?
Is feedback more important for short answer
questions or multiple choice questions?
Should you allow people to peek ahead at
feedback before answering a question?
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 25
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
26. Where to learn more
Three prominent researchers in the field
Professor Henry Roediger
University of Washington in St. Louis, Memory Lab
http://psych.wustl.edu/memory/publications/
Assistant Professor Jeffrey Karpicke
Purdue University, Indiana
http://memory.psych.purdue.edu/publications/
Assistant Professor Elizabeth Marsh
Duke University, North Carolina
http://pn.aas.duke.edu/sites/marsh/publications.html
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 26
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
27. Dr Will Thalheimer
Expert at translating research into practice
Two free white papers on Questionmark website
http://www.questionmark.com/whitepapers
The Learning Benefit of Questions
Providing Feedback to Learners
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 27
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
28. Learn more from me
Follow my learning in this area
http://blog.questionmark.com
Twitter : @johnkleeman
Any questions or comments or suggestions feel
free to email me at john@questionmark.com
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 28
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
29. Can you give some feedback to me?
Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Slide 29
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.