«On the impact of programming exercise descriptions. Effects of programming exercise descriptions to scores and working times», eMadrid seminar on «MOOCs as part of the future of digital learning», part of the LWMOOCS Conference

1. On the Impact of
Programming Exercise Descriptions
Effects of Programming Exercise Descriptions to Scores and Working Times
Ralf Teusner, Thomas Hille
Research Assistant
Hasso Plattner Institute

2. MOOC students solve programming exercises within the browser.
Required steps:
1. Understand the problem
2. Build on given code snippets
3. Apply the explained concepts to learn
4. Implement the solution
Situation
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 2
Focus
Focus

3. Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 3
CodeOcean –
Code Execution and Grading in the Browser
Run Score

4. 1. Do exercise structure and wording influence success and learning
outcomes?
2. Are findings from classical education transferrable to MOOC
programming education?
Research Questions
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 4

5. ■ N = 2,444 students à huge variety of prior knowledge
■ Four experiment groups (assigned randomly)
■ Identical implementation task, different exercise descriptions
Experiment Setup
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 5

6. ■ Duke wants to build a music device, although his taste of music would be
regarded as highly… questionable.
■ Write a program that counts up with a for loop from 1 to 50
□ In case the number is divisible by both 3 and 7, ding-dong should be
output in a separate line on the screen.
□ If the respective number is only divisible by 3, ding should be output.
□ If the respective number is only divisible by 7, dong should be output.
□ In all other cases, output ping .
■ Note: Checking the divisibility of a number can be done with the modulo
operator.
The Exercises: Variant A: Optimal
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 6
The first 10 lines
should look like this:
ping
ping
ding
ping
ping
ding
dong
ping
ding
ping
Line 21 looks should
like this:
ding-dong

7. ■ Duke wants to build a music device, although his taste of music would be
regarded as highly… questionable.
■ Write a program that counts up with a for loop from 1 to 50
□ In case the number is divisible by both 3 and 7, ding-dong should be
output in a separate line on the screen.
□ If the respective number is only divisible by 3, ding should be the output.
□ If the respective number is only divisible by 7, dong should be the output.
□ In all other cases, output ping.
■ Note: Checking the divisibility of a number can be done with the modulo
operator.
The Exercises: Variant A: Optimal
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 7
The first 10 lines
should look like this:
ping
ping
ding
ping
ping
ding
dong
ping
ding
ping
Line 21 looks should
like this:
ding-dong

8. ■ Write a program that counts up with a for loop from 1 to 50.
□ Each time the current number is divisible by 3, the program should
output ding in a separate line on the screen.
□ If the number is divisible by 7, the program should output dong in a
separate line on the screen.
□ If a number is divisible by both 3 and 7, the output ding-dong should be
generated in one line instead of ding or dong.
□ If the number is not divisible by 3 or 7, ping should be displayed in one
line on the screen.
The Exercises: Variant B: Non-Optimal Order
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 8
The first 10 lines
should look like this:
ping
ping
ding
ping
ping
ding
dong
ping
ding
ping
Line 21 looks should
like this:
ding-dong

9. ■ Write a program that counts up with a for loop from 1 to 50.
□ If the respective number is divisible by 3 (but not by 7), ding should be
printed out.
□ If the respective number is divisible by 7 (but not by 3), dong should be
printed out.
□ In the case that the number is divisible by both 3 and 7,
ding-dong should be printed out.
□ In all other cases, ping should be the output.
■ Note: Consider the order in which the conditions are to be checked in
advance. This might simplify your solution.
The Exercises: Variant C: Incorrect Order & Complicated
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 9

10. ■ Write a program that counts up with a for loop from 1 to 50.
□ For each number divisible by 3, print ding, for each number divisible by
7 print out dong and for each number that is divisible by 3 as well as by
7 print ding-dong.
□ If a number is not divisible by 3 or 7, ping should be the output.
The Exercises: Variant D: Very Short
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 10

11. Results
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 11
Exercise
∅ score,
rel. stdev
∅ working time [min],
rel. stdev
∅ runs,
rel. stdev
A 0.973, 29% 11:40 (base), 79% 7.1 (base), 85%
B 0.976, 29% 12:58 (+11%), 78% 6.8 (- 4%), 83%
C 0.974, 36% 13:10 (+13%), 76% 7.5 (+ 6%), 74%
D 0.967, 26% 13:09 (+13%), 82% 8.8 (+24%), 94%

12. Results
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 12
Exercise
∅ score,
rel. stdev
∅ working time [min],
rel. stdev
∅ runs,
rel. stdev
A 0.973, 29% 11:40 (base), 79% 7.1 (base), 85%
B 0.976, 29% 12:58 (+11%), 78% 6.8 (- 4%), 83%
C 0.974, 36% 13:10 (+13%), 76% 7.5 (+ 6%), 74%
D 0.967, 26% 13:09 (+13%), 82% 8.8 (+24%), 94%

13. ■ Significant differences between exercise A and C (t-test)
■ Short description (exercise D) caused high number of runs
■ Scores shows ceiling effect (students show high diligence)
■ Overall high standard deviations
Discussion
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 13

14. 1. Do exercise structure and wording influence success and learning
outcomes?
à Yes, considerable effect noticeable
2. Are findings from classical education transferrable to MOOC
programming education?
à Yes, in general, further experimentation needed
General Remarks and Learnings
Ralf Teusner,
LWMOOCS 2018,
28.09.2018
Programming
Exercise
Descriptions
Slide 14

15. On the Impact of
Programming Exercise Descriptions
Effects of Programming Exercise Descriptions to Scores and Working Times
ralf.teusner@hpi.de
Ralf Teusner, Thomas Hille
Research Assistant
Hasso Plattner Institute