Can predictive learning analytics empower teachers to support students at risk? Can they enhance students' performance? A large scale study @TheOpenUniversity, UK.

1. Implementing Predictive Learning
Analytics on a Large Scale: The
Teacher's Perspective
Dr Christothea Herodotou
Authors: Christothea Herodotou, Bart Rienties, Avinash Boroowa, Zdenek Zdrahal,
Martin Hlosta , Galina Naydenova. Proceedings of LAK2017.

2. Predictive Learning Analytics (PLA)
o PLA identify students at risk + inform teachers
o Mixed-effects of providing PLA to teachers
o Difficulty in understanding and interpreting PLA data
and visualisations
o Difficulty in identifying specific interventions
o Promising outcomes (van Leeuwen et al., 2014;
McKenney and Mor, 2015)
o Identification of participation problems and intervention
o Development of curriculum materials
2

3. Aim of this study
Evaluate whether providing teachers
with PLA data would empower them to
identify and assist students in need for
additional support
3

4. OU Analyse (OUA)
•OUA: Early identification
of students at risk of
failing
•Available to tutors and
student support teams
•Aim: Improve retention of
OU students
https://analyse.kmi.open.ac.uk/
4

5. OUA Dashboard: Module view
5

6. OUA Dashboard: Student view
6

7. OUA Dashboard: Student view
Nearest neighbours, Predictions with real scores, Personalised recommender
7

8. Research Questions
1.How did 240 tutors within 10 modules made
use of OUA predictions and visualisations to
help students at risk?
2.To what extent was there a positive impact on
students' performance and retention when
using OUA predictions?
3.Which factors explain tutors' uses of OUA?
8

9. Methodology
1.Statistical comparisons: 240 tutors OUA
access Vs 613 tutors with no access to OUA
predictions
 formal withdrawal rates by the end of the
module,
 completion and pass rates.
2.Usage data of OUA dashboard (N=70)
3.Qualitative data: Semi-structured interviews
(N=6)
9

10. Results: Withdrawal rates
Formal withdrawal rates by the end of the module
2015 -OUA students 2015 - nonOUA students
count % count %
Arts 55 1728
did not reach the end of the module 15 27.27% 253 14.64%
reached 40 72.73% 1475 85.36%
Technology 239 1809
did not reach the end of the module 56 23.43% 311 17.19%
reached 183 76.57% 1498 82.81%
Law 246 1820
did not reach the end of the module 33 13.41% 359 19.73%
10

11. Results: Completion & pass rates
Module completion and pass rates
2015 -OUA students 2015- nonOUA students
Completion rates count % count %
Education 260 2841
did not complete 69 26.54% 931 32.77%
completed 191 73.46% 1910 67.23%
Law 245 1815
did not complete 79 32.24% 732 40.33%
completed 166 67.76% 1083 59.67%
Pass rates
Law 246 1820
did not passed 86 34.96% 797 43.79%
Passed 160 65.04% 1023 56.21%
11

12. Results: Usage of OUA dashboard
12

13. Results: Interviews (1/2)
Actual uses of OUA:
•Most useful OUA features:
o colour-coded students
o VLE activity
•Teachers define what features to use and how often (time
management)
Usefulness of OUA
•Teachers checking on students often (emails, forums)
•Teachers more proactive/enhance teaching practices
o “on top” of students
o “where I need to put my efforts"
• Predictions: complement teachers’ intuition + additional
insights
13

14. Results: Interviews (2/2)
Approaching students at risk
o Varied interventions - referral to student support
services, sending emails, texting, calling, doing
nothing
o Varied approaches: persistent VS less pro-active
o Personalised approach
Future intentions
o Interested in future use yet with improvements to
the system (e.g., “sensitive” to students’ activities
online)
14

15. Conclusions
o Blurred picture (withdrawal, completion and pass
rates)
o Variation in teachers’ degree and quality of
engagement with learning analytics.
o Lack of consensus about intervention strategies
o Predictive data - enhance and facilitate teaching
practice, especially within distance learning
contexts
o Research directions: Identify how, when, and what
interventions to trigger to support students
adequately
15

16. 16