Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning. The document discusses motivation and goals of learning analytics, challenges, and examples of data that could be analyzed from students, including demographics, academic performance, physical behavior, and online behavior. It also discusses ensuring principles of transparency, alignment with pedagogy, and responsibility in student data use. Examples are provided of diagnostic testing, analyzing VLE access data, and measuring the effects of video use and flipped classrooms.

1. Learning Analytics
- value what we measure or measure
what we value
Dr Mark Glynn
@glynnmark

2. Contact details
• mark.glynn@dcu.ie
• glynnmark
• http://enhancingteaching.com

3. Outline
• Introduction
• Motivation and goals
• Challenges
• Examples
• Technical bits
• Discussion
– What would you like to
analyse
– Collaboration

4. Teaching Enhancement Unit
TEACHING
ENHANCEMENT
UNIT
Online
and Blended Learning
Support
Awards and
Grants
Credit Earning
Modules
Professional
Development
Workshops

5. Data Analytics
Data analytics is the science of
extracting actionable insight from
large amounts of raw data
DIT – MSc in Computing

6. Youtube

7. Tesco

8. Definiton
Learning analytics is the measurement,
collection, analysis and reporting of data
about learners and their contexts,
for purposes of understanding and
optimising learning and the
environments in which it occurs. A related
field is educational data mining.
- wikipedia

9. Discussion
What data do we
already collect?

10. So much student data we could use
Demographics
• Age, home/term address, commuting distance, socio-economic status, family composition, school
attended, census information, home property value, sibling activities, census information
Academic Performance
• CAO and Leaving cert, University exams, course preferences, performance relative to peers in
school
Physical Behaviour
• Library access, sports centre, clubs and societies, eduroam access yielding co-location with others
and peer groupings, lecture/lab attendance,
Online Behaviour
• Mood and emotional analysis of Facebook, Twitter, Instagram activities, friends and their actual
social network, access to VLE (Moodle)

11. Motivation

12. Discussion
What challenges do
you foresee in your
institution?

13. Core Principles – Open University UK
Learning analytics is a moral practise which should align with core organisational principles
The purpose and boundaries regarding the use of learning analytics should be well defined and visible
Students should be engaged as active agents in the implementation of learning analytics
The organisation should aim to be transparent regarding data collection and provide students with the
opportunity to update their own data and consent agreements at regular intervals
Modelling and interventions based on analysis of data should be free from bias and aligned with
appropriate theoretical and pedagogical frameworks wherever possible
Students are not wholly defined by their visible data or our interpretation of that data
Adoption of learning analytics within the organisation requires broad acceptance of the values and
benefits (organisational culture) and the development of appropriate skills
The organisation has a responsibility to all stakeholders to use and extract meaning from student data
for the benefit of students where feasible

14. Amazon

15. Examples

16. The earlier we diagnose, the earlier
we can treat
John Carroll
Mark Glynn
Eabhnat Ni Fhloinn
@glynnmark

17. Maths Diagnostic test

18. Data Analytics on VLE Access Data
How much can we mine from a
mouseclick ?
John Brennan
Owen Corrigan
Aly Egan
Mark Glynn
Alan F. Smeaton
Sinéad Smyth
@glynnmark

19. No significant difference in the entry profiles of
participants vs. non-participants overall
PredictEd Participant Profile

20. Total Moodle Activity – notice the
periodicity

21. One example module – ideal !

22. LGxxx – Predictor confidence (ROC AUC)

23. All modules

24. 0%
20%
40%
60%
80%
100%
LG116 MS136 LG101 HR101 LG127 ES125 BE101 SS103 CA103 CA168
Workshops
Wikis
Forums
Assignments
Quizzes
scorm
lesson
choice
feedback
database
glossary
wiki
url
book
pages
folders
files
Course content
a b c d e f g h i j

25. Study by numbers
• 17 Modules across the
University (first year, high
failure rate, use Loop,
periodicity, stability of
content, Lecturer on-board)
• Offered to students who
opt-in or opt-out, over 18s
only
• 76% of students opted-in,
377 opted-out, no difference
among cohorts
• 10,245 emails sent to 1,184
students who opted-in over
13 weekly email alerts

26. The Interventions – Lecturers’ Experience

27. Student Engagement - individual

28. Student Engagement - individual

29. Regression analysis- class grades

30. What to measure?
Content Tra
nsfer
Interaction Collaboratio
n
Assessment

31. Collaboration

32. Modules which work well …
• Have periodicity (repeatability) in Moodle access
• Confidence of predictor increases over time
• Don't have high pass rates (< 0.95)
• Have large number of students, early-stage

33. LGxxx: law based subject
Students / year = ~110
Pass rate = 0.78

34. Student Interventions: Feedback

35. Relative data

36. Student Experience of PredictED
Students who took part were asked to complete a short
survey at the start of Semester 2 - N=133 (11% response rate)
Question Group 1 (more
detailed email)
Group 2
% of respondents who opted out of
PredictED during the course of the semester 4.5% 4.5%
% who changed their Loop usage as a
result of the weekly emails
43.3% 28.9%
% who would take part again/are offered and
are taking part again
72.2%
(45.6%/ 26.6% )
76.6%
(46% /30.6% )

37. 33% said they changed how they
used Loop. We asked them how?
• Studied more
– “More study”
– “Read some other articles online”
– “Wrote more notes”
– “I tried to apply myself much more, however yielded no results”
– “It proved useful for getting tutorial work done”
• Used Loop more
– “I tried harder to engage with my modules on loop”
– “I think as it is recorded I did not hesitate to go on loop. And loop as become my first
support of study.”
– “I logged on more”
– “I read most of the extra files under each topic, I usually would just look at the lecture
notes.”
– “I looked at more of the links on the course nes pages, which helped me to further my
understanding of the topics”
– “I learnt how often I need to log on to stay caught up.”

38. Did you change Loop usage for
other modules?
• Most who commented used Loop more often for other modules
– “More often”
– “More efficient”
– “Used loop more for other modules when i was logging onto
loop for the module linked to PredictED”
– “Felt more motivated to increase my Loop usage in general
for all subjects”
One realised that Lecturers could see their Loop activity
“I realised that since teachers knew how much i was
using loop, i had to try to mantain pages long on so it
looked as if i used it a lot”

39. Subject Description Non-Participant Participant
BE101 Introduction to Cell Biology and Biochemistry 58.89 62.05
CA103 Computer Systems 70.28 71.34
CA168 Digital World 63.81 65.26
ES125 Social&Personal Dev with Communication Skills 67.00 66.46
HR101 Psychology in Organisations 59.43 63.32
LG101 Introduction to Law 53.33 54.85
LG116 Introduction to Politics 45.68 44.85
LG127 Business Law 60.57 61.82
MS136 Mathematics for Economics and Business 60.78 69.35
SS103 Physiology for Health Sciences 55.27 57.03
Overall Dff in all modules 58.36 61.22
Average scores for participants are higher in 8 of the 10
modules analysed, significantly higher in BE101, and
CA103
Module Average Performance
Participants vs. Non-Participants

40. Measuring the Flipping effect
Patrick Doyle
Mark Glynn
Eveyn Kelleher
@glynnmark

41. Assessment Challenge

42. Logistics
• 200+ students
• 4 assignments each
• 5 minutes per
assignment
• 10 lecturers
• 2 weeks of assessment

43. Marking guide

44. Related research
Comparing students who watched versus not watched video one
Comparing Means [ t-test assuming unequal variances (heteroscedastic) ]
Descriptive Statistics
VAR Sample size Mean Variance
Didn't watch 84 51.86905 691.58505
Watched 102 63.15686 576.74743
Two-tailed distribution
p-level 0.00284t Critical Value (5%) 1.97402
One-tailed distribution
p-level 0.00142t Critical Value (5%) 1.65387

45. Discussion
What would you like
to measure?

46. Selectively Analyzing your Course
data?
@glynnmark
@drjaneholland
Dr Jane Holland, RCSI
Eric Clarke, RCSI
Dr Mark Glynn, DCU
Dr Evelyn Kelleher, DCU

47. Constructive Alignment
Learning Outcomes

48. Particulars
• Attendance
– Tutorials
– labs
• Moodle logs
• Defined times
• Assessment results

49. Excel results Video tracking
0%
10%
20%
30%
40%
50%
60%
70%
Zero One Two Three Four Five Six Seven
What students watched "x" amount of videos
Watched
Watched before

50. All activities

51. One activity in particular

52. Multiple activities

53. Health warning

54. Questions and discussion…

55. Talking to one another
LMS
SRS
CMS
Timetable
Wifi
Library

56. Databridge
MITM
Course
Database
Timetable
ePortfolio
WifiLMS
Library
SRS

57. Additional slides

58. Building classifiers for each
week/each module
Training Data
Testing

59. Notes on model confidence
• Y axis is confidence in AUC ROC (not probability)
• X axis is time in weeks
• 0.5 or below is a poor result
• Most Modules start at 0.5 when we don't have much
information
• 0.6 is acceptable, 0.7 is really good (for this task)
• The model should increase in confidence over time
• Even if confidence overall increases, due to randomness the
confidence may go up and down
• It should trend upwards to be a valid model and viable
module choice

60. BExxx: Intro to Cell Biology
Results / year = ~300
Pass rate = 0.86

61. BExxx

62. SSxx: Health Sciences
Results / Year = ~150
Pass rate = 0.92

63. MSxxx

64. LGxxx

65. HR101

66. CAxxx

67. Some unusable modules
Modules where the ROC AUC increases slowly
(e.g stays below 0.6) e.g. PS122

68. Timescale for Rollout
• Still some issues on Moodle access log data transfer
to be resolved
• Still have to resolve student name / email address /
Moodle ID / student number
• Still to resolve timing of when we can get new
registration data, updates to registrations (late
registrations, change of module, change of course,
etc.) …
• Should we get new, “clean” data each week ?

69. Why did you take part?
• The majority of students
wanted to learn/monitor
their performance
• Many others were curious
• Some were interested in
the Research aspect
• Some were just following
advice
• Others were indifferent

70. How easy was it to understand the
information in the emails ?
(1= not at all easy, 5 = extremely easy)
• Average 3.97 (SD= 1.07)
• Very few had comments to make
(19/133)
– Most who commented wanted more
detail.

71. Week 3
Training Data
Testing

72. Week 4
Training Data
Testing

73. Week 5
Training Data
Testing

74. Week 6
Training Data
Testing

75. Week 7
Training Data
Testing

76. Week 8
Training Data
Testing

77. Week 9
Training Data
Testing