Paper at CHI 2019, PDF at tiny.cc/icepdf. Digital educational resources could enable the use of randomized experiments to answer pedagogical questions that instructors care about, taking academic research out of the laboratory and into the classroom. We take an instructorcentered approach to designing tools for experimentation that lower the barriers for instructors to conduct experiments. We explore this approach through DynamicProblem, a proof-ofconcept system for experimentation on components of digital problems, which provides interfaces for authoring of experiments on explanations, hints, feedback messages, and learning tips. To rapidly turn data from experiments into practical improvements, the system uses an interpretable machine learning algorithm to analyze students’ ratings of which conditions are helpful, and present conditions to future students in proportion to the evidence they are higher rated. We evaluated the system by collaboratively deploying experiments in the courses of three mathematics instructors. They reported benefits in reflecting on their pedagogy, and having a new method for improving online problems for future students.

1. Enhancing Online Problems Through Instructor-
Centered Tools for Randomized Experiments
Joseph Jay Williams
University of Toronto Computer Science ( Nat. U. of Singapore)
www.josephjaywilliams.com/papers, tiny.cc/icepdf
Anna Rafferty, Andrew Ang, Dustin Tingley, Walter Lasecki, Juho Kim
[I’m originally from the Caribbean,
Trinidad and Tobago]
Postdoc at U of T (www.josephjaywilliams.com/postdoc)
Computer Science PhD positions to do Education Research
CHI 2019 subcommittee on “Learning/Families” (Amy Ogan &
I are SCs)

2. How Can We Help Instructors Conduct A/B Experiments?
• Opportunity: Collect data about alternative instructional
approaches, instead of relying on intuition
• Obstacle: Time and effort to program experiments
• Elaboration Messages in online problems
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
A z-score is defined as the number of
standard deviations a specific point is
away from the mean.
Elaboration Messages:
Explanations,
Hints,
Learning Tips

3. Related Work
• Technology for A/B Experimentation (Optimizely,
edX, ASSISTments) (Heffernan & Heffernan, 2014)
• Involving instructors in research (Barab & Squire, 2014)
• Elaboration messages in online problems (Shute, 2008)

4. Overview
• Design goals for instructor-centered
experimentation
• DynamicProblem, an end-user tool (on-campus
courses & MOOCs)
• Reinforcement learning for dynamic
experimentation
• Insights from deployment with 3 instructors

5. Goals for Instructor Centered Experimentation
• 1. Deploy experiments and obtain data with
minimal programming
– Provide end-user plug-in, DynamicProblem
• 2. Use data for practical improvement
– Use reinforcement learning to automatically give
more effective conditions to future students

6. DynamicProblem Plug-In for Courses
• Embed into any Learning Management System (e.g. Canvas)
or MOOC, via Learning Tools Interoperability Standard

7. Student View of DynamicProblem
Linda is training for a marathon, which is a race that is 26
miles long.
Her average training time for the 26 miles is 208 minutes,
but the day of the marathon she was x minutes faster than
her average time.
What was Linda's running speed for the marathon in miles
per minute?
Elaboration Message
Linda's speed is the distance she ran divided by the time it took. The
distance Linda ran was 26 miles. The time it took her was 208 – x.
Linda's speed was 26/(208 - x)
26/(208 - x)
How helpful was the above information for your learning?
Completely Perfectly
Unhelpful Helpful
0 1 2 3 4 5 6 7 8 9 10
A
B
ACM Learning @ Scale 2016

8. Instructor View of DynamicProblem

9. View Elaboration Messages

10. Add Elaboration Messages

11. View Elaboration Messages
Learning Tip

12. Data Dashboard (Instructor 3)
Learning Tip
Probability
of
Message

13. Observations from Deployment with 3 Instructors
• Lowered Barriers: “not aware of any tools that do this
sort of thing”, “even if I found one, wouldn’t have the
technical expertise to incorporate it in my course”
• Reflection on pedagogy: “I never really seriously
considered [testing] multiple versions as we are now
doing. So even if we don't get any significant data, that
will have been a benefit in my mind”
• Making research practical: “a valuable tool. Putting in
the hands of the teacher to understand how their
students learn. Not just in broad terms, but specifically
in their course”….

14. 2. Use Data For Practical Improvement
• Instructor concerns:
– Experiments advance researchers’ goals, but do not
directly help their students
– Ethics of giving students unhelpful conditions
• Approach of Dynamic Experimentation:
– Analyze data in real-time
– Give higher-rated messages to future students

15. Model
Action a
Dynamic Experimentation: Exploration vs Exploitation
• Multi-Armed Bandit (Reinforcement Learning)
A
Reward R
Policy
Elaboration Message A
The probability is 3/7 * 5/8, because the number of
cookies is changing.
Rating
How helpful was the above information for your learning?
0 1 2 3 4 5 6 7 8 9 10
A B
70% 30%
(Probability of Message being Helpful)
(0 to 10 Rating by Student)
Elaboration Message B
The number of cookies is changing..
Randomized Probability
Matching (Thompson Sampling)

16. Dynamically Weighted Randomization
• 50/50 probability of assignment
• 60/40
• 70/30
• … 100/0
• Probability a student assigned to a message =
• Probability of that message being highest rated

17. Instructor 2’s Experiment
Probability
of
Message

18. Observations from Deployment with 3 Instructors
• Directly helping students: “improved the experience
of many of the students by giving them answers that
are more helpful… the earlier ones can help improve
the experience of the later students. That’s pretty
neat”

19. Student Perceptions?
• Students weren’t surprised by, and appreciated the
approach:
• “I assume companies are always A/B testing on me”
• “now the data I provide can help other people learn”

20. Limitations & Future Work
• Conduct better experiments
– You can use the hosted tool: www.josephjaywilliams.com/dynamicproblem
github.com/kunanit/adaptive-mooclet-lti
• Go beyond subjective student judgments
• Instructor-researcher collaboration
• Personalization of Elaboration Messages
–MOOClet WebService for dynamic A/B testing
(www.josephjaywilliams.com/mooclet)

21. Review
• Design goals for instructor experimentation
• DynamicProblem, an end-user tool (Use it at
www.josephjaywilliams.com/dynamicproblem)
• Multi-armed bandits for dynamic experimentation
• Insights from deployment with 3 instructors
• Limitations & Future Work
• PhD students can do education at U of T Comp Sci
• www.josephjaywilliams.com/postdoc
• Learning/Families subcommittee for CHI 2019 (Amy
Ogan & I are SCs)

22. Thank you!
• National University of Singapore Information
Systems & Analytics
• Harvard VPAL Research Group