ONLINE EDUCA 2011

New Research Shows How to Use
Quizzes and Tests to Slow the
Forgetting Curve
Educa Online, December 1st 2011
John Kleeman, Questionmark Chairman
www.questionmark.com

Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Would you like to
reduce learner
forgetting curves?

Slide 2

Retrieval helps you learn
 Francis Bacon, 1620
If you read anything
over twenty times, you
will not learn it by heart
so easily as if you were
to read it only ten, trying
to repeat it between
whiles, and when
memory failed, looking
at the book.

Slide 3

Learning

Receive Working Long term
information memory memory

Slide 4

Learning useful if can retrieve
• Question  Research shows
Cue • Other stimulus retrieval practice
helps future retrieval
• Retrieve from long
term memory  Successful retrieval
Search
modifies memory
making it more
• Act or respond retrievable in future
Act

Slide 5

Experimental methodology
Quiz / retrieval practice
Re-study group group

Study Study

Re-study Quiz

Measure Measure
Slide 6

Roediger & Karpicke experiment 2006
 120 students divided into groups randomly
 Given reading comprehension passages used in
TOEFL
 Some students
 Study for 14 mins
 Some students
 Study for 7 mins
 Recall material for 7 mins
 Compared results after 5 mins and 1 week

Slide 7

How much learners retained
90%
80%
70%
60%
50%
Restudy
40%
Quiz
30%
20%
10%
0%
After 5 mins After 1 week

Slide 8

Marsh & others 2007. Experiments
with multiple choice questions
60%

50%

40%

30% No quiz
Quiz
20%

10%

0%
SAT II Bloom Definition Bloom Application

Slide 9

Roediger & Butler. Feedback on
multiple choice questions
Retention after one week
60%

50%

40%

30%

20%

10%

0%
No quiz Quiz, no feedback Quiz, immediate Quiz, delayed
feedback feedback

Slide 10

Karpicke & Blunt experiment 2011
 80 students in 4 groups, studying a science text
 Study
 Repeated Study
 Concept Mapping (mind maps)
 Retrieval Practice
 Surveyed learners on expectation of retention
 Tested a week later to see how much retained

Slide 11

How much learners retained
Actual retention
Learner expectation of
80%
retention 70%
80%
70% 60%
60% 50%
50% 40%
40% 30%
30% 20%
20% 10%
10%
0% 0%

Slide 12

Larson experiment 2009
Quizzes reduce forgetting curve
 In real-life, not in the laboratory
 Medical residents (doctors in training)
 20 doctors in each group

Quiz group Restudy group
Day 1 Study Study
Day 1 Quiz Re-study
2 weeks later Quiz Re-study
2 weeks later Quiz Re-study
6 months later Final exam Final exam


Part of the results
Quiz group Re-study group

62%

48%
44%
36%

19%

Initial Quiz Quiz 6 months
Slide 14

Retrieval practice key findings
 Retrieval practice helps retention of learning
 Applies to facts
 And also concepts
 Retrieval practice includes
 Recall
 Formative quizzes/tests
 Self-assessment
 Explain or read back of material
 Timing of retrieval
 Works best when spaced out
 Strongest effect when retrieval is difficult
 Feedback very helpful
 Repetition of retrieval
 Research suggests 5-7 spaced repetitions helpful

Slide 15

Slide 16 Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Computer quizzes are effective
retrieval practice because …
 Learners do not realize they will forget
 Quizzes encourage/force retrieval practice
 Quizzes give corrective feedback
 And also:
 Randomize so easy to repeat
 Mobile devices
 Accessible
 Measure that practice has happened

 Computer assessments also have many “indirect“
benefits

Slide 17

Indirect benefits of post-learning
assessments
 Direct to further learning
 Correct misconceptions
 Feed back to improve the course
 Compare to pre-test
 Measure achievement
 Encourage study
 Engage students

Slide 18

If you are a practitioner …
 Give quizzes after learning sessions
 Focus questions on important things to remember
 Make quizzes effortful (delay & difficulty)
 Question types:
 Recall (e.g. short answer) best
 Multiple choice work too
 Include feedback
 Usually the correct answer
 Slight delay if possible
 Encourage retrieval (don’t allow peek ahead)
 Spacing helps significantly

Slide 19

If you are a manager …
 How easy is it for learners to take quizzes?
 How easy is it for your instructors to quiz
learners?
 Could you make it easier?

Slide 20

Potential objections
 “No time”
 “We teach creativity not facts”
 “How sure are you of the research?”
 “Our learners don’t like to be tested”
 “What about privacy?”

Slide 21

In a Perfect World…

Slide 22

In a Perfect World…
we would remember everything we learn!

Slide 23

In the Actual World…

Slide 24

In the Actual World…
we forget some of it!

Slide 25

Questions through Learning Process

Reduce
Forgetting
Create
Intrigue
Knowledge/Skills

Strengthen
Memory
Questions

Questions
Questions
Questions

Questions
Time

Slide 26

References
Who Title Journal Date Description URL
Roegiger & Test-Enhanced Learning Psychological 2006 Widely cited paper Here
Karpicke Science showing benefits of
testing vs studying

Marsh, The memorial consequences Psychonomic 2007 Shows benefits of multiple Here
Roediger, of multiple-choice testing Bulletin & Review choice questions
Bjork&Bjork

Larsen, Test-enhanced learning in Medical Education 2009 Shows experiment on Here
Butler & medical education. medical education and
Roediger retrieval practice

Fazio, Right/wrong feedback: Memory 2010 Research showing just Here
Huelser, Consequences for learning telling students right or
Johnson & wrong without correct
Marsh answer unhelpful
Roediger & The critical role of retrieval Trends in Cognitive 2010 8 page review article on Here
Butler practice in long-term retention Sciences retrieval and retention

Karpicke & Retrieval Practice Produces Science Magazine 2011 Clear paper comparing Select from here
Blunt More Learning than Elaborative retrieval with study and (non-
Studying with Concept Mapping mind maps commercial)

Kleeman Recent cognitive psychology International CAA 2011 Some more detail from Here
research shows strongly that Conference me
quizzes help retain learning.
What does this mean1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Copyright ©
for CAA?

Some questions for you:
 What is the best way to remember this session for
the future?

 What question types seem to give best effect?

 What is the best timing for feedback?

Slide 28

Your questions
 Contact me at john at questionmark.com or
Twitter: @johnkleeman

Slide 29

ONLINE EDUCA 2011

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie ONLINE EDUCA 2011

Ähnlich wie ONLINE EDUCA 2011 (18)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

ONLINE EDUCA 2011

Hinweis der Redaktion