2. THE EVIDENCE STANDARD
Teachers can feel bombarded…
I strive to be a scholarly teacher …
• Apply the rigor we bring to our academic
disciplines to the discipline of teaching.
• Choose teaching methods that are strongly
informed by the best empirical evidence
available.
Contrast teaching your subject with treating a
medical condition like diabetes
3. In your teaching do you have a method for holding
students accountable for preparing for class?
Previous anonymous poll results (compiled):
~17% → I don’t, but I ask/threaten really well
~50% → I use a paper method (quiz, journal…)
~10% → I use a digital method (clickers, etc.)
~5% → I use Just-in-Time Teaching
~18% → I have some other method
(N ~ 200)
4. OVERVIEW
1. Motivation for change
2. Basics of Just-in-Time Teaching
3. Mock example
4. Evidence for effectiveness
5. Summaries
Teaser for tomorrow’s talk
“Just-in-Time Teaching: Making It Shine”
8:45 AM in Wolf 205
9. JUST-IN-TIME TEACHING
Online pre-class assignments
called WarmUps
First half - Students
• Conceptual questions, answered in sentences
• Graded on thoughtful effort
Second half - Instructor
• Responses are read “just in time”
• Instructor modifies that day’s plan accordingly.
• Aggregate and individual (anonymous) responses
are displayed in class.
Learne
r
Teacher
11. JITT STRUCTURE & RESPONSE
RATES
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
%Responsed
Class #
Response Rate by Day
College Physics I, N = 78
Worth 10% of final grade
Due 10 PM the night before class
Assignments available for prior 2-3 days
College Physics I
12. WARMUP QUESTIONS
• Every-day language
• Occasional simple comprehension question
• Mostly “higher level” questions
• Any question is better than none (don’t be precious)
Connections to evidence:
–Pre-class work reduces working memory load
during class.
–Multimodal practice (not learning styles):
JiTT brings reading, writing and discussion as
modes of practice.
13. METACOGNITION
Two questions in every WarmUp:
First:“What aspect of the material did you find
the most difficult or interesting.”
Last: “How much time did you spend on the pre-
class work for tomorrow?”
Connections to evidence:
–Forced practice at metacognition:
Students regularly evaluate their own
interaction with the material.
14. THE JITT FEEDBACK LOOP
Student responses:
• Graded on thoughtful effort
• Sampled and categorized for display
• Quoted anonymously
Closing the loop:
• Respond to some students digitally
• Class time shifts to active engagement.
15. EXAMPLE: WHIRLING BUCKET
A bucket of water can be whirled in a
vertical circle without the water falling
out, even at the top of the circle when the
bucket is upside down. Explain…
~15% → An outward force holds it in
~30% → An inward force holds it in
~20% → Talked (correctly!) about
acceleration & velocity… but
didn't really answer.
~10% → Nailed it! (or close enough)
16. EXAMPLE: WHIRLING BUCKET
“The water doesn't come out because you
twirling the bucket is applying the force of
spinning, and the water just kind of counteracts
that motion.”
“Because the water naturally wants to keep
traveling in the same direction its being
whirled around in the water attempts to
continue going up in a straight line but the
bottom of the bucket forces it to stay in the
bucket, like when you are pushed by the door
of a car while making a turn.”
17. JUST-IN-TIME TEACHING
A different student role:
• Actively prepare for class
(not just reading/watching)
• Actively engage in class
• Compare your progress & plan accordingly
A different instructor role:
• Actively prepare for class with you
(not just going over last year’s notes )
• Modify class accordingly
• Create interactive engagement opportunities
Learne
r
Teacher
18. JITT VS. FINAL GRADE
CORRELATIONS
College Physics I, Fall 2013
0
20
40
60
80
100
0 10 20 30 40 50 60 70 80 90 100
CumulativeScore(withoutwarm-ups)
WarmUp Score
WarmUps vs. Cumulative Score
Correlation r = 0.71
19. PROGRESSIVE EXAMS
CORRELATIONS
College Physics I:
Important disclosure: This was not a hypothesis we were
testing, it appeared as we analyzed the data. Could be
0.18
0.33
0.43
0.54
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Mini Exam
(week 4)
Exam 1
(week 7)
Exam 2
(week 11)
Final Exam
(week 16)
NoneWeakStrongModerate
Correlations between Total WarmUp Score
and Sequence of Exams
22. STUDIED EFFECTIVENESS
Used at hundreds of institutions
Dozens of studies/articles, in many disciplines:
Bio, Art Hist., Econ., Math, Psych., Chem., etc.
–Increase in content knowledge
–Improved student preparation for class
–Improved use of out-of-class time
–Increased attendance & engagement in class
–Improvement in affective measures
23. Mean on 1-5 scale
Preparation for class 4.06
Engagement during
class 3.93
STUDENT SURVEY RESULTS
9% 10%
81%
10%
18%
73%
10%
22%
68%
0%
20%
40%
60%
80%
100%
Harmful Neutral Helpful
How did WarmUps affect your...
Preparation Engagement Learning
N = 781
24. STUDENT SURVEY QUOTES
Physics:
“Initially, it was hard for me to get used to the
warm-ups. It seemed like along with the
homework assignments there was a lot of things
to do. Eventually I got used to it and ultimately
the warmups really helped me to learn the
material and stay caught up with the class.”
“If it weren't for warm ups, the amount of time I
spent reading the book would have dropped by
75%”
25. WHAT MIGHT STOP YOU?
In terms of the technique:
Time, coverage, not doing your part, pushback…
In terms of the technology:
Learning curve, tech. failures, perfectionism…
In any reform of your teaching:
Reinventing, no support, too much at once…
26. MY SUMMARY
JiTT may be among the easiest research-based
instructional strategies that you can consistently
integrate into your teaching.
From an evidence-based perspective, JiTT
addresses often-neglected areas.
Be prepared to find that students know less than
we might hope. (Perhaps freeing?)
27. YOUR SUMMARY
What part of JiTT concept/process is the fuzziest
for you after this talk?
Tomorrow:
“Just-in-Time Teaching: Making It Shine”
8:45 AM in Wolf 205
Email: jeff.loats@gmail.com
Twitter: @JeffLoats
Slides: www.slideshare.net/JeffLoats
28. LEARNING BY DOING
Topics for tomorrow:
Writing good questions
Getting student buy-in
Choosing a tool
Want to do a WarmUp?
If I’ve talked you into attending tomorrow, email
me (jeff.loats@gmail.com) and tell me so.
I will send you a (brief) WarmUp and I will use
your responses in tomorrow’s talk!
29. JITT REFERENCES & RESOURCES
Simkins, Scott and Maier, Mark (Eds.) (2010) Just inTimeTeaching: Across the Disciplines, Across the
Academy, Stylus Publishing.
Gregor M. Novak, Andrew Gavrini, Wolfgang Christian, Evelyn Patterson (1999) Just-in-Time
Teaching: Blending Active Learning with WebTechnology. Prentice Hall. Upper Saddle River NJ.
K. A. Marrs, and G. Novak. (2004). Just-in-Time Teaching in Biology: Creating an Active Learner
Classroom Using the Internet. Cell Biology Education, v. 3, p. 49-61.
Jay R. Howard (2004). Just-in-Time Teaching in Sociology or How I Convinced My Students to
Actually Read the Assignment. Teaching Sociology,Vol. 32 (No. 4 ). pp. 385-390. Published by:
American Sociological Association
Stable URL: http://www.jstor.org/stable/3649666
S. Linneman, T. Plake (2006). Searching for the Difference: A ControlledTest of Just-in-Time
Teaching for Large-Enrollment Introductory Geology Courses. Journal of Geoscience Education, Vol.
54 (No. 1)
Stable URL:http://www.nagt.org/nagt/jge/abstracts/jan06.html#v54p18
ON-DEMAND SLIDES
30. WHAT TOOLS TO USE?
The crucial part:
Daily reading, grading & using responses
• Automatic full credit for any response
• View all responses to a question together
• Grade responses on the same page with
minimal clicks
Wishlist:
Easy (quick!) individual feedback
31. WHAT TOOLS TO USE?
• CMS/LMS (Blackboard, D2L, Moodle, etc.)
Ready to use, tools… imperfect awful
• Free service from JiTTDL.org.
Designed just for JiTT, but extra login, and the
site has not been improved in ~5 years
• Students email responses
Easy! usually overwhelming and awful
• Blogging tools (WordPress)?
• New tools (TopHat, Learning Catalytics)?
Hinweis der Redaktion
“Learning technologies should be designed to increase, and not to reduce, the amount of personal contact between students and faculty on intellectual issues.”Study Group on the Conditions of Excellence in American Higher Education, 1984
Bombarded: hybrid courses, brain-based learning, blended courses, technology in the classroom, learner-centered teaching, etc.
About ~20 years ago, physics teachers began treating education as a research topic!
Their findings were pretty grim
"But the students do fine on my exams!“
It appeared that students had been engaging in “surface learning” allowing them to solve problems algorithmically without actually understanding the concepts.
Was this just at Harvard (silly question)!
Data from H.S., 2-year, 4-year, universities, etc.
0.23 Hake gain on the FCI means that of the newtonian physics they could have learned in physics class, they learned 23% of it.
Conclusion: Traditional physics lectures are all similarly (in)effective in improving conceptual understanding.
Enter Physics Education Research:
An effort to find empirically tested ways to improve the situation.
Jeff’s results: Depending on the class 60-80% of my students do their WarmUps, self-reporting that they spend ~40 minutes reading/responding (very consistent average)
Average = 37%
Total participants 232
Faculty 175
Higher Ed IT 32
Students 25
Questions are about NEW material
Results for time-spent question: A pretty steady average of ~40 minutes across many courses/levels/cohorts
Misconceptions, good efforts, superior explanations, metacognition, etc.
Incorrect or incomplete responses are often particularly useful for classroom discussion.
0.71 represents a quite strong correlation
0.50 is a moderate correlation (fairly strong for educational interventions)
Is this just about new energy being put into an old class?
(This is a difficult confounding factor in assessing new teaching techniques.)