The goal of higher education institutions is to provide quality education to students. Predicting academic success and early intervention to help at-risk students is an important task for this purpose. This talk explores the possibilities of applying machine learning in developing predictive models of academic performance. What factors lead to success at university? Are there differences between students of different generations? Answers are given by applying machine learning algorithms to a data set of 400 students of three generations of IT studies. The results show differences between students with regard to student responsibility and regularity of class attendance and great potential of applying machine learning in developing predictive models.

1. Machine learning algorithms as tools for
student success prediction
Assoc. Prof. Dijana Oreški, PhD
University of Zagreb
Faculty of Organization and Informatics

2. Intro
• This work has been supported by Croatian Science
Foundation under the project UIP-2020-02-6312.
 SIMON – Intelligent system for automatic selection of machine
learning algorithm in social sciences
• Lab Louise
 Laboratory for data mining and intelligent systems
 louise.foi.hr

3. Intro
• “If you torture data long enough they will confess at
the end.”
Ronald Coase

4. Agenda
• Introduction
 Education data & WHY?
 machine learning algorithms
• Research motivation
• Research methodology HOW?
 CRISP DM standard
• Research results WHAT?
 Model evaluation
 Model interpretation
• Conclusion

5. Agenda
• Introduction
 Education data & WHY?
 machine learning algorithms
• Research motivation
• Research methodology HOW?
 CRISP DM standard
• Research results WHAT?
 Model evaluation
 Model interpretation
• Conclusion

6. Introduction
• Huge number of machine learning algorithms
applications in a broad spectrum of domains.
 Crucial role in harnessing the power of the vast amount of data
we produce daily in the digital age.
• The application of algorithms is complex, iterative and
time-consuming.
 There is a need to automate the selection of algorithms for
models development.
• Which algorithm is best to used in a specific situation,
in a particular domain, at a particular dataset?

7. Research motivation

8. Research motivation
Education
data
Supervised
machine
learning
algorithms

9. Agenda
• Introduction
 Education data & WHY?
 machine learning algorithms
• Research motivation
• Research methodology HOW?
 CRISP DM standard
• Research results WHAT?
 Model evaluation
 Model interpretation
• Conclusion

10. CRISP DM standard
Data
preparation
Data
understanding
Evaluation
Modelling
Data
Domain
understanding
Deployment

11. Research papers
• Kliček, B.; Oreški, D.; Divjak, B., Determining individual learning
strategies for students in higher education using neural networks,
International Journal of Arts and Sciences.
• Oreški, D; Konecki, M; Pihir, I: Predictive Modelling of Academic
Performance by Means of Bayesian Networks, 47th International
Scientific Conference on Economic and Social Development.
• Oreški, D; Pihir, I; Konecki, M., CRISP-DM process model in
educational setting, 20th International Scientific Conference on
Economic and Social Development

12. Research papers
• Oreški, D; Konecki, M; Milić, L., Estimating profile of successful IT
student: data mining approach, MIPRO 2017 - 40th International
Convention Proceedings
• Kovač, R; Oreški, D., Educational Data Driven Decision Making:
Early Identification of Students at Risk by Means of Machine
Learning, Proceedings of CECIIS 2018 /
• Oreški, D.; Hajdin, G., Exploring differences in predictors of
academic success between different generations of students,
EDULEARN19 Proceedings, 2019.

13. Research papers
• Oreški, D., Hajdin, G. Development and comparison of predictive
models based on learning management system data // WSEAS
TRANSACTIONS on INFORMATION SCIENCE and APPLICATIONS,
2019
• Oreški, D.; Hajdin, G., A Comparative Study of Machine Learning
Approaches on Learning Management System Data, 2019
Proceedings - 3rd International Conference on Control, Artificial
Intelligence, Robotics & Optimization
• Filipović, D.; Balaban, I.; Oreški, D., Cluster analysis of students’
activities from logs and their success in self-assessment tests,
Proceedings of CECIIS 2018
• Oreški, D.; Kadoić, N., Analysis of ICT students' LMS engagement
and sucess, International Scientific Conference on Economic and
Social Development, 2018.

14. Domain understanding
• Research goals:
 To determine whether data from different sources (surveys, e-
learning systems...) can be a good basis for creating predictive
models of academic success.
 To determine which variables are the best predictors of
academic success.
 To determine whether predictors of success change over time.

15. Domain understanding
• The improvement of the educational system and the
achievement of students optimal learning requires the
data collection and analysis.
 Recent papers deal with this topic from the perspective of:
 (i) the various academic and non-academic factors involved
in the data,
 (ii) the research methodology used in data analysis:
previously focused on advanced statistical approaches,
nowadays on machine learning approaches,
 iii) accuracy and reliability of developed predictive models.

16. Data understanding
• Data sources:
 Survey,
 Learning management system data,
 YouTube analytics.

17. Data understanding
• Data sources:
 Survey,
 Learning management system data,
 YouTube analytics.

18. Data understanding
Students through generations

19. Data understanding
Students by entrance exam results
Bottom 10%-30% Middle Top 10%

20. Data understanding
First grade at the Faculty

21. Data understanding
I manage time well
Completely Disagree Neither agree Agree Completely
disagree or disagree agree

22. Data understanding
I see myself as responsible person
Completely Disagree Neither agree Agree Completely
disagree or disagree agree

23. Data understanding
I find teamwork useful
Completely Disagree Neither agree Agree Completely
disagree or disagree agree

24. Data understanding
I have prepared for the classes
Completely Disagree Neither agree Agree Completely
disagree or disagree agree

25. Data understanding
• Data sources:
 Survey,
 Learning management system data,
 YouTube analytics.

26. Data understanding

27. Data understanding

28. Data understanding

29. Data understanding

30. Data understanding
• Data sources:
 Survey,
 Learning management system data,
 YouTube analytics.

31. Data understanding
Variable Variable Correlation
Video duration Percentage of video views -0,78
Variable Variable Correlation
Complexity
Percentage of video
views
-0,35

32. Modelling
• Information based machine learning
 Decision tree
• Similarity based machine learning
 K-nearest neighbours
 “When I see a bird that walks like a duck and swims like a duck
and quacks like a duck, I call that bird a duck.” James W. Riley
• Error based machine learning
 Neural networks
 “Success is stumbling from failure to failure with no loss of
enthusiasm.” Winston Churchill
• Probability based machine learning
 Bayesian networks
 “When my information changes, I alter my conclusions. What
do you do, sir?” John Maynard Keynes

33. Agenda
• Introduction
 Education data & WHY?
 machine learning algorithms
• Research motivation
• Research methodology HOW?
 CRISP DM standard
• Research results WHAT?
 Model evaluation
 Model interpretation
• Conclusion

34. Research results
rang A rang B rang C
Highschool grade
average 1 1 4
Lecture attendance 2 4 10
First grade at the
Faculty 3 3 3
I manage my time
well 4 15 13
Entrance exam
results 5 2 1
I find myself as
responsible person 8 10 5
Seminars attendance 9 11 7
I have prepared for
the classes 12 13 12
I find teamwork
useful 15 8 9
Gender 16 7 16

35. Research results
• Strong positive correlation between predictors of academic success
in Generations B and C(r=0.652941176, p<0,01).
• Correlations in predictors between generations A and B
(r=0.370588235, p<0,01), A and C(r=0.388235294, p<0,01) are
smaller.
• Conclusion:
 Predictors of student success change over time, but only when
there are significant changes in the education system.

36. Research results
• Female students are more active on the e-learning
system and complete the course more successfully than
male colleagues.
• Activity on the e-learning system is a significant
predictor of student success.
 Students were most active during the weeks of the colloquium,
and especially the day before the colloquium.
 Students can be characterized as "last minute" students,
because they fulfill their obligations as late as possible in terms
of the deadline.
 They are active in the "late" hours.

37. Conclusion
• Machine learning algorithms provide accurate and
reliable predictive models.
• However, results are not perfect:
 Predicting students at risk will always suffer from classification
errors - false positives and false negatives.
 The error affects the allocation of available resources.
 Potentially negative effects on the individual.

38. Conclusion
• To determine whether data from different sources
(surveys, e-learning systems...) can be a good basis for
creating predictive models of academic success.
 Yes! Integration contributes to successful prediction.
• To determine which variables are the best predictors of
academic success.
 First grade at the faculty, previous knowledge..
 LMS activity..
• To determine whether predictors of success change
over time.
 Change according to change in educational system.

39. Thank you!