VII Jornadas eMadrid "Education in exponential times". "Analysing and Altering MOOC Learners' Behaviours at Scale". Claudia Hauff. TU Delft, Países Bajos. 03/07/2017.

1. Claudia Hauff
Joint work with Dan Davis, Guanliang Chen, Markus Krause,
Efthimia Aivaloglou and Geert-Jan Houben
Analysing and altering MOOC
learners’ behaviours at scale

3. ✤ 60+ MOOCs
✤ 1.5 million enrollments
✤ From primary school to PhD level
✤ Lots of user data (click logs)

4. Our goals
Data
Knowledge
Application
to learning
Gain actionable insights into learner
behaviours at scale.
a. Data Science
b. Big data processing
Increase our knowledge about learners by
looking beyond the learning platform
a. Web data analytics
Design technology interventions that
enable adaptive learning at scale.
a. Information retrieval
b. Human-centred design
c. Learning technologies

5. Learner profiling
beyond the MOOC platform
Learning transfer:
does it take place?
From learners to
earners
Learning paths:
desired vs. executed
flickr@herrolsen

6. Learner profiling beyond
the MOOC platform
ACM WebScience 2016
Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, and Geert-Jan Houben. Beyond the MOOC platform:
Gaining Insights about Learners from the Social Web, ACM WebScience, pp. 15-24, 2016.

7. Whythis research?
Learner
Engagement, retention, …
During the MOOC

8. Whythis research?
Learner
Before the MOOC
NOTHING
Engagement, retention, …
During the MOOC

9. Whythis research?
Learner
Before the MOOC
NOTHING
Engagement, retention, …
During the MOOC
NOTHING
After the MOOC

10. Howto solve the problem?
We propose:
a deeper understanding about learners
can be gained by exploring their traces
on the Social Web.

11. Whatresearch questions?
1
On what Social Web platforms can a significant fraction of
MOOC learners be identified?
Are learners who demonstrate specific traits
on the Social Web drawn to certain types of MOOCs? 2
To what extent do Social Web platforms enable us to
observe (specific) user attributes
that are relevant to the online learning experience?
3

12. Learner identification
across Social Web platforms
edX learners
Email Login name Full name+ +

13. Learner identification
across Social Web platforms
edX learners
Email Login name Full name+ +
1. Explicit Matching
Profile images & links
Identification via
emails

14. Learner identification
across Social Web platforms
edX learners
Email Login name Full name+ +
1. Explicit Matching
Profile images & links
Identification via
emails
2. Direct Matching
Identification via profile
links from Step 1

15. Learner identification
across Social Web platforms
edX learners
Email Login name Full name+ +
1. Explicit Matching
Profile images & links
Identification via
emails
2. Direct Matching
Identification via profile
links from Step 1
3. Fuzzy Matching
Search learners by their
login & full names
Compare:
1. profile link
2. profile image
3. login & full names

16. Social Web platforms
involved in our work

18. Matching results
for 18 DelftX MOOCs
Lowest Highest Overall
Gravatar 4.37% 23.49% 7.81%
Twitter 4.99% 17.58% 7.78%
Linkedin 3.90% 11.05% 5.89%
StackExchange 1.23% 21.91% 4.58%
GitHub 3.43% 41.93% 10.92%

19. Matching results
for 18 DelftX MOOCs
Lowest Highest Overall
Gravatar 4.37% 23.49% 7.81%
Twitter 4.99% 17.58% 7.78%
Linkedin 3.90% 11.05% 5.89%
StackExchange 1.23% 21.91% 4.58%
GitHub 3.43% 41.93% 10.92%
On average, 5% of learners can be identified on globally
popular Social Web platforms.

20. Learners on
Linkedin
- Using job titles & skills to characterise learners

21. Learners on
Linkedin
- Using job titles & skills to characterise learners
Spreadsheet MOOC
- Software Engineer
- Business Analyst
- …

22. Learners on
Linkedin
- Using job titles & skills to characterise learners
Spreadsheet MOOC
- Software Engineer
- Business Analyst
- …
Design Approach MOOC
- Co founder
- UX designer
- …

23. Learners on
StackExchange
- Functional
Programming
learners in
StackOverflow
- To what extent do
learners change their
question/answering
behaviour during
and after a MOOC?

24. Take-home
Messages
On average, 5% of learners from 18 DelftX MOOCs
can be identified on 5 globally popular Social Web platforms. 1
Learners with specific traits prefer different types of MOOCs.2
Learners’ post-course behaviour can be investigated
by using their external Social Web traces.3

25. Learning Transfer:
does it take place?
Best Paper Nominee at
ACM Learning At Scale 2016
An Investigation into the Uptake of Functional
Programming in Practice
Guanliang Chen, Dan Davis, Claudia Hauff and Geert-Jan Houben, Learning Transfer: does it take place in
MOOCs?, ACM Learning At Scale, pp. 409-418, 2016.

26. Whatis learning transfer?
Learning transfer is the application
of knowledge or skills gained in a
learning environment to another
context.

27. Whydo we care?
Learning transfer is a more important
measure of learning in MOOCs than
retention, success or engagement.

28. FP101x
@flickr:christiaan_008
Course programming language: Haskell
Run as a typical video-lecture based MOOC
Assessment: 288 Multiple Choice questions
Introduction to Functional Programming

29. FP101x
@flickr:christiaan_008
Course programming language: Haskell
Run as a typical video-lecture based MOOC
Assessment: 288 Multiple Choice questions
Introduction to Functional Programming
37,485 learners registered.
41% engaged with the course.
5% completed the course.
33% were active on GitHub (1.1M events).

30. Whatdid we do?
FP101x
logs
surveys
coding
activities
3 months 2.5 years + 0.5 years
+ +
email address

31. Whatdid we do?
FP101x
logs
surveys
coding
activities
3 months 2.5 years + 0.5 years
+ +
email address
Are changes made in a
functional language?

32. GitHub
hosting, collaboration and organisation
the most popular social coding platform

33. GitHub
hosting, collaboration and organisation
the most popular social coding platform
detailed

34. GitHub
hosting, collaboration and organisation
the most popular social coding platform
founded in 2007
detailed

35. GitHub
hosting, collaboration and organisation
the most popular social coding platform
founded in 2007
long-term
detailed

36. GitHub
10+ million registered users
hosting, collaboration and organisation
the most popular social coding platform
founded in 2007
long-term
detailed

37. GitHub
10+ million registered users
hosting, collaboration and organisation
the most popular social coding platform
founded in 2007
long-term
large-scale
detailed

38. detailed logs
code changes
project meta-data

39. A sanity check

40. Are “GitHub learners” different?
GitHub
learners
Non-GitHub
learners
#Learners 12,415 25,070
Completion rate 7.71% 4.03%
Avg. time watching
videos
49.1 min 27.7 min
Avg. #questions
attempted
31.3 17.5
Avg. accuracy of
learners’ answers
23.4% 12.9%

41. Are “GitHub learners” different?
GitHub
learners
Non-GitHub
learners
#Learners 12,415 25,070
Completion rate 7.71% 4.03%
Avg. time watching
videos
49.1 min 27.7 min
Avg. #questions
attempted
31.3 17.5
Avg. accuracy of
learners’ answers
23.4% 12.9%
GitHub learners are more engaged than non-GitHub learners
and exhibit higher levels of knowledge.

42. Are “Expert learners” different?
Expert GitHub
learners
Novice GitHub
learners
#Learners 1,721 10,694
Completion rate 15% 6.5%
Avg. time watching
videos
78.6 min 44.4 min
Avg. #questions
attempted
57.9 27.0
Avg. accuracy of
learners’ answers
38.0% 21.1%

43. Are “Expert learners” different?
Expert GitHub
learners
Novice GitHub
learners
#Learners 1,721 10,694
Completion rate 15% 6.5%
Avg. time watching
videos
78.6 min 44.4 min
Avg. #questions
attempted
57.9 27.0
Avg. accuracy of
learners’ answers
38.0% 21.1%
Expert learners are more engaged than Novice learners
and exhibit higher levels of knowledge.

44. To what extent do engaged learners
exhibit learning transfer?
5-10% >30%10-30%<5%

45. To what extent do engaged learners
exhibit learning transfer?
5-10%

46. Which type of learner is more likely
to display learning transfer?
flickr@ConalGallagher
Intrinsically motivated Extrinsically motivated

47. Which type of learner is more likely
to display learning transfer?
flickr@ConalGallagher
Intrinsically motivated

48. Which type of learner is more likely
to display learning transfer?
Experienced Inexperienced

49. Which type of learner is more likely
to display learning transfer?
Experienced

50. Which type of learner is more likely
to display learning transfer?
High-spacing
learning routine
Low-spacing
learning routine

51. Which type of learner is more likely
to display learning transfer?
High-spacing
learning routine

52. Learners who transfer
quickly move to Scala
FP101x

53. Conclusions
Most transfer learning findings from the classroom hold.
The observed transfer rate is low: 8.5%.
Learners quickly moved on after the course to
industrially-relevant functional languages.
@flickr:torsten-reuschling

54. From Learners to Earners:
Enabling MOOC Learners to Apply
their Skills and Earn Money in an
Online Market Place
IEEE Transactions on Learning
Technologies
Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff and Geert-Jan Houben. Can
Learners be Earners? Investigating a Design to Enable MOOC Learners to Apply their Skills and Earn Money in
an Online Market Place, IEEE Transactions on Learning Technologies.

55. What
MOOCs aim to educate the world.
Most successful learners are already
highly educated.
Learners from developing countries
are underrepresented.
is the problem?

56. Whatis the problem?
EX101x: Data Analysis to the MAX()

57. How
Pay learner at scale: recommend tasks
from online market places to learners
that are relevant to the course material.
can we tackle it?

58. How
Pay learner at scale: recommend tasks
from online market places to learners
that are relevant to the course material.
can we tackle it?

59. Howcan we tackle it?

60. What
1) To what extent do online market
places contain relevant tasks?
2) Are learners able to solve
real-world tasks with high quality?
do we need to look at?

61. Setup
1) Weekly spreadsheet “bonus
exercises” drawn from Upwork
(manually checked) in EX101x
2) Accuracy check
3) Quality check (code smells)

62. Howare learners doing?
Good accuracy & quality.

63. Built a working
recommender.

64. Gauging MOOC learners’
adherence to the
designed learning path
Educational Data Mining 2016
Dan Davis, Guanliang Chen, Claudia Hauff and Geert-Jan Houben. Gauging MOOC Learners’ Adherence to the
Designed Learning Path, 9th International Conference on Educational Data Mining, pp. 54-61, 2016.

65. WHAT IS A LEARNING
PATH ?
THE SEQUENCE OF EVENTS A STUDENT
TAKES TOWARDS A LEARNING OBJECTIVE

66. RQ
To what extent do learners
adhere to the designed
learning path?

67. VIDEO QUIZ PROGRESS FORUM
WATCH START VIEW START
SUBMIT
END
SUBMIT
END

68. Frame101x
VIDEO
QUIZ
END
FORUM
SUBMIT
FORUM
END
PROGRESS
QUIZ
START
FORUM
START
QUIZ
SUBMIT
3
3
4
6
36
VIDEO
QUIZ
END
FORUM
SUBMIT
FORUM
END
PROGRESS
QUIZ
START
FORUM
START
QUIZ
SUBMIT
1 1
1
121
VIDEO
QUIZ
END
FORUM
SUBMIT
FORUM
END
PROGRESS
QUIZ
START
FORUM
START
QUIZ
SUBMIT
3
7
7
7
6
VIDEO
QUIZ
END
FORUM
SUBMIT
FORUM
END
PROGRESS
QUIZ
START
FORUM
START
QUIZ
SUBMIT
5 1
1
1
1
1
1
101
FP101x
RI101x
EX101x

69. MOOC ENROLLED PASS CHAINS
PASS/NON
QUIZ
QUESTIONSRATE
TRIES VIDEOS
ACCESSED
FP101x
RI101x
Frame101x
EX101x
37k
9k
34k
33k
5.3%
4.3%
2.4%
6.5%
1.06M/
807K
66k/
30k
95k/
141k
1.02M/
855k
288
75
26
136
1
1-3
2
2
67.5%
49.7%
51.0%
45.0%

70. APPROACH
1. VIDEO INTERACTIONS
2. BEHAVIOR PATTERN CHAINS
3. EVENT TYPE TRANSITIONS

71. VIDEO INTERACTIONS
Non-Passing
Passing
Week 1 Week 2
Week 3
FP101x
Designed Lecture Order
Executed Paths

72. 2. BEHAVIOR
PATTERN
CHAINSWHAT ARE THE MOST COMMON SEQUENCES
IN THE EXECUTED LEARNING PATHS?

73. FORUMEND
EIGHT-STEP CHAIN
EXAMPLE
LECTUREWATCH
QUIZSTART
QUIZSUBMIT
QUIZSUBMIT
QUIZEND
PROGRESS
LECTUREWATCH

74. MOOC ENROLLED PASS CHAINS
PASS/NON
QUIZ
QUESTIONSRATE
TRIES VIDEOS
ACCESSED
FP101x
RI101x
Frame101x
EX101x
37k
9k
34k
33k
5.3%
4.3%
2.4%
6.5%
1.06M/
807K
66k/
30k
95k/
141k
1.02M/
855k
288
75
26
136
1
1-3
2
2
67.5%
49.7%
51.0%
45.0%

75. OPEN CARD
SORTING
OPEN CARD
SORTING

76. MOTIF
A RECURRING OR
REPEATED ELEMENT

77. MOTIF FREQ
TOTAL
FREQ
PASS
FREQ
FAIL
QUIZ COMPLETE 552,363
(29.4%)
328,995
(30.8%)
223,368
(27.7%)
1.
BINGE WATCHING 149,784
(8.0%)
59,498
(5.6%)
90,286
(11.2%)
2.
LECTURE -> QUIZ COMPLETE 100,179
(5.3%)
50,415
(4.7%)
49,764
(6.2%)
3.
QUIZ COMPLETE -> FORUM 99,828
(5.3%)
67,722
(6.3%)
32,106
(4.0%)
4.
FP101x
XQUIZ events only with at least 1 X = SUBMIT
WATCH events only
WATCH event(s) followed by XQUIZ
events w/ >1 X = SUBMIT
XQUIZ events with >1 X = SUBMIT followed
by XFORUM events

78. 3. EVENT
TYPE
TRANSITIONSHOW DO LEARNERS NAVIGATE BETWEEN
CERTAIN EVENT TYPES?

79. EVENT TYPE TRANSITIONS
Frame101x
Non-
Passing
Frame101x
Passing
(EXECUTED)

80. Ongoing work
Data
Knowledge
Application
to learning
Gain actionable insights into learner
behaviours at scale.
a. Data Science
b. Big data processing
Increase our knowledge about learners by
looking beyond the learning platform
a. Web data analytics
Design technology interventions that
enable adaptive learning at scale.
a. Information retrieval
b. Human-centred design
c. Learning technologies

81. MOOCs are vital to bring higher
education to the world.
Lots of unexplored potential.
Plenty of data.
Many users.
http://bit.ly/lambda-lab
Overall …