Presentation by Rebecca Ferguson at Learning and Knowledge 2015 (LAK15), Poughkeepsie, NY, USA. Massive open online courses (MOOCs) are now being used across the world to provide millions of learners with access to education. Many learners complete these courses successfully, or to their own satisfaction, but the high numbers who do not finish remain a subject of concern for platform providers and educators. In 2013, a team from Stanford University analysed engagement patterns on three MOOCs run on the Coursera platform. They found four distinct patterns of engagement that emerged from MOOCs based on videos and assessments. However, not all platforms take this approach to learning design. Courses on the FutureLearn platform are underpinned by a social-constructivist pedagogy, which includes discussion as an important element. In this paper, we analyse engagement patterns on four FutureLearn MOOCs and find that only two clusters identified previously apply in this case. Instead, we see seven distinct patterns of engagement: Samplers, Strong Starters, Returners, Mid-way Dropouts, Nearly There, Late Completers and Keen Completers. This suggests that patterns of engagement in these massive learning environments are influenced by decisions about pedagogy. We also make some observations about approaches to clustering in this context.

1. Examining engagement:
analysing learner subpopulations in
massive open online courses (MOOCs)
Rebecca Ferguson, Doug Clow

2. What are MOOCs?
● Massive
thousands may sign up
● Open
no payment is required
● Online
resources on the Internet
● Courses
time-bounded cohorts
Commonalities of scale, economic/philosophical perspective,
location and structure – but not pedagogy

3. Current context
% complete from: www.katyjordan.com/MOOCproject
Students seek not merely
access, but access to success
“ ”John Daniel, 2012

4. Patterns of engagement: Coursera
● Sampling
learners explored some course materials
● Auditing
learners watched most videos, but
completed assessments rarely, if at all
● Disengaging
learners completed assessments at the
start of the course and then reduced
their engagement
● Completing
learners completed most assessments
Kizilcec, R., Piech, C., and Schneider, E., 2013. Deconstructing disengagement:
analyzing learner subpopulations in massive open online courses. LAK13
MOOC designers can apply this
simple and scalable categorization
to target interventions and develop
adaptive course features
“
”

5. 5
Replication
FutureLearn data

6. 6
Calculating an activity profile
Replicating the method
● T = on track (3)
undertook the assessment on time
● B = behind (2)
submitted the assessment late
● A = auditing (1)
engaged with content but not assessment
● O = out (0)
did not participate

7. 7
Replication
Identifying dissimilarity between engagement patterns
Assigned numerical value to each label
• On track = 3
• Behind = 2
• Auditing = 1
• Out = 0
Calculated L1 norm for each
engagement pattern
Used that as the basis for 1-dimensional
k-means clustering
Repeated clustering 100 times and
selected solution with highest likelihood
Focused on extracting four clusters

8. 8
Replication
Coursera and FutureLearn results were different
● Sampling
learners explored some course materials
● Auditing
learners watched most videos, but completed assessments rarely, if at all
● Disengaging
learners completed assessments at the start of the course and then reduced
their engagement
● Completing
learners completed most assessments
√
√
x
x

9. 9
Exploring the method
Trying different approaches
● Different values for k
explored values for k between 3 and 8 (silhouette width was at a minimum for k=4
– suggesting this was the least suitable value)
● One-dimensional approach
might discard potentially useful information about patterns of engagement before
they can be used by the clustering algorithm
● Ran k-means on the engagement profiles directly
treating them as 6- or 8-dimensional vectors (some courses were six weeks long,
and some courses were eight weeks long).
Explored with k=4 and again found Samplers and Completers
Explored with k=3 to k=8 – less successful than one-dimensional approach

10. 10
FutureLearn is different
Pask, Gordon. (1976).
Conversation Theory:
Applications in Education
and Epistemology.
New York: Elsevier.

11. 11
FutureLearn is different
Conversation is a central feature

12. 12
Revising the numeric values

13. 13
Patterns of engagement
Patterns vary with pedagogy and learning design
On an eight-week MOOC:
● Samplers visit only briefly
● Strong Starters do first assessment
● Returners come back in Week 2
● Mid-way Dropouts drop out mid-way
● Nearly There drop out near the end
● Late Completers finish
● Keen Completers do almost everything

14. 14
Typical engagement profiles
These profiles apply to an eight-week course
● Samplers visit only briefly
[1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content
● Strong Starters do first assessment
[9, 1, 0, 0, 0, 0, 0, 0] – 9 means they visited content and did assessment on time
● Returners come back in Week 2 [9, 9, 0, 0, 0, 0, 0, 0]
● Mid-way Dropouts
[9, 9, 9, 4, 1, 1, 0, 0] – 4 means they submitted assessment late
● Nearly There drop out near the end
[11, 11, 9, 11, 9, 9, 0, 0] – 11 means full engagement, 8 means submission on time
● Late Completers finish
[5, 5, 5, 5, 5, 9, 9, 9] – 5 means they viewed content and submitted late
● Keen Completers do almost everything [11, 11, 9, 9, 11, 11, 9, 9]

15. 15
Samplers
[1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content
● The largest group in all MOOCs
● Typically accounted for 37% – 39% of learners
● Visited the materials, but only briefly
● Active in a small number of weeks
● 25% – 40% joined after Week 1
● Very few Samplers posted comments (6% – 15%)
● Almost no Samplers submitted any assessment
Highly stable across all MOOC and across most values of k

16. 16
Strong starters
[9, 1, 0, 0, 0, 0, 0, 0]
● All Strong Starters submitted the first assignment
● Engagement dropped off sharply after that
● A little over a third of them posted comments
● Typically posted fewer than four comments
Highly stable across all MOOCs and across most values of k

17. 17
Returners
[9, 9, 0, 0, 0, 0, 0, 0] – came back in Week 3
● Completed the assessment in the first week
● Completed the assessment in the second week
● Then dropped out
● Over 97% completed those two assessments, although some submittted late
● No Returner explored all course steps
● Average amount of steps visited varied (23% – 47%)
This cluster did not appear on MOOC3, which had three widely spaced assessments, so this
engagement pattern was not possible

18. 18
Mid-way Dropouts
[9, 9, 9, 4, 1, 1, 0, 0]
● A much smaller cluster (6% of learners on MOOC1, 7% on MOOC4)
● These learners completed three or four assessments
● They dropped out around halfway through the course
● Mid-way dropouts visited about half the steps on the course
● Just under half posted comments
● Posted just over six comments on average
This cluster did not appear on MOOC2 and MOOC3,
because of the spacing of their assessments

19. 19
Nearly There
[11, 11, 9, 11, 9, 9, 0, 0]
● Another small cluster (5% – 6% of learners)
● Consistently completed assessments
● Dropped out just before the end of the course
● Visited around 80% of the course
● Submitted assignments consistently (>90%) and typically on time until Week 5
● Activity then declined steeply
● Few completed the final assessment
● None completed the final assessment on time
This cluster appeared for all four MOOCs, but was variable with varying k

20. 20
Late Completers
[5, 5, 5, 5, 5, 9, 9, 9]
● Submitted the final assessment
● Submitted most other assessments
● However, either submitted late or missed some assessments
● Each week, more than 94% of this cluster submitted their assessments
● More than three-quarters submitted the final assessment on time (78% – 90%)
● Around 40% of them posted comments (76% did so on MOOC3)
This cluster was fairly stable across all MOOCs and across most values of k

21. 21
Keen Completers
[11, 11, 9, 9, 11, 11, 9, 9]
● Accounted for 7% – 23% of learners
● All the Keen Completers submitted all assessments
● More than 80% of these were submitted on time
● Typically, Keen Completers visited more than 90% of course content
● Over two-thirds contributed comments (68% – 73%)
● Mean number of comments varied from 21 to 54
This cluster was highly stable across all MOOCs and across all values of k

22. 22
Improving learning and learning environments
Closing the loop
● Previews of course material would allow Samplers to make a more
informed decision about whether to join the course
● Sign-up pages could draw attention to the problems experienced by
those who are out of step with the cohort
● Discussion steps for latecomers could support those who fall behind at
the start
● Prompts might encourage flagging learners to return and register for a
subsequent presentation
● Bridges between course weeks could stress links and point learners
forward

23. 23
View these slides at www.slideshare.net/R3beccaF
Rebecca
Ferguson
@R3becca
F
Doug
Clow
@dougclow