Join Bart Rienties, Professor of Learning Analytics at the second LTI Series event Most institutions, including the OU, are exploring how data can better inform teaching and learning. What can we learn from data, and learning analytics in particular? Should we be afraid about being monitored? Or should we embrace this? Bart’s research focuses on how the OU can use the power of learning analytics to enhance teaching and learning, and what the potential limitations are for social interaction, cultural discourse, and practice. This seminar will look at the different models being adopted globally, and use a framework to consider what might be the best approach for the OU. DATE AND TIME: Thu 25 October 2018, 14:00 – 15:00 LOCATION: The Hub Theatre, Walton Hall, Milton Keynes

1. Unpacking Six Myths
at the Open University
LTI Series
Thursday 25 October 2018
The Hub Lecture Theatre
Join the vote by logging into:
https://pollev.com/bartrienties552
@DrBartRienties

2. Unpacking some OU myths?
• Myth: “a widely held but false belief or
idea”
• What evidence is there?
• What works (and what not)?
• Test and Learn Evidence Hub
https://openuniv.sharepoint.com/sites/qual-enhance/test-learn-
evidence/Pages/Home.aspx

3. https://www.lostateminor.com/2018/01/09/unbelievable-photo-shows-flock-birds-forming-shape-giant-bird/

4. https://www.rac.co.uk/drive/car-reviews/daewoo/matiz/matiz-1998-2005/
Mum’s Daewoo Matiz Effect

5. Confirmation bias, also called confirmatory bias or myside bias, is the tendency to search
for, interpret, favour, and recall information in a way that confirms one's pre-existing beliefs or
hypotheses. It is a type of cognitive bias and a systematic error of inductive reasoning. People
display this bias when they gather or remember information selectively, or when they interpret it
in a biased way. The effect is stronger for emotionally charged issues and for deeply entrenched
beliefs.
https://en.wikipedia.org/wiki/Confirmation_bias

6. So can you get all Six questions
right?
• Myth busting????  results are
representative for large groups of OU students
(but not all)
• Results based upon large quantitative data
analysis, which might miss nuance and
context
• Of course there could be exceptions to these
results (e.g., disciplinary, “special sub-groups”)
• Remember “Daowoo Matiz Effect”
https://pollev.com/bartrienties552

7. Myth Sample size (n = )
1. OU students love to work together 116,646
2. OU student satisfaction is positively related to teaching quality,
and success in learning outcomes (e.g., pass rates, retention)
111,526
3. Most OU students are making positive learning gains over time
(i.e., as measured by the grades they get)
4,222 & 18,329
4. The grades that OU students get are mostly related to their
abilities, effort, cognition, etc. (i.e., what students do to study)
4,222 & 18,329
5. OU Student engagement in Moodle is primarily determined by
students (abilities, effort, cognition, time availability, etc.)
45,190
6. Most OU students follow the module schedule when studying 45,190 & 387
https://pollev.com/bartrienties552

10. Li, N., Marsh, V., Rienties, B., Whitelock, D. (2017). Online learning experiences of new versus continuing learners: a large scale replication study. Assessment &
Evaluation in Higher Education, 42(4), 657-672. Impact factor: 1.243

11. Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151

12. Constructivist
Learning Design
Assessment
Learning Design
Productive
Learning Design
Socio-construct.
Learning Design
VLE Engagement
Student
Satisfaction
Student
retention
150+ modules
Week 1 Week 2 Week30
+
Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151
modules. Computers in Human Behavior, 60 (2016), 333-341
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student
engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028.
Assimilative activities

14. How does student satisfaction relate to module performance?Satisfaction
Students who successfully completed module

15. Ullmann, T., Lay, S., Rienties, B. (2017). Data wranglers’ key metric report. IET Data Wranglers, Open

16. Constructivist
Learning Design
Assessment
Learning Design
Productive
Learning Design
Socio-construct.
Learning Design
VLE Engagement
Student
Satisfaction
Student
retention
150+ modules
Week 1 Week 2 Week30
+
Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151
modules. Computers in Human Behavior, 60 (2016), 333-341
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student
engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028.
Communication

17. Hessler, M., Pöpping, D. M., Hollstein, H., Ohlenburg, H., Arnemann, P. H., Massoth, C., . . . Wenk, M. (2018). Availability of cookies during an academic course
session affects evaluation of teaching. Medical Education, 52(10), 1064-1072. doi: doi:10.1111/medu.13627

19. Cognitive learning gains
 Cognitive learning gains were
measured in five ways:
 1. Cognitive learning gains within
modules
 2. Cognitive learning gains from
first to second module
 3. Cognitive learning gains within a
qualification
 4. Cognitive learning gains across
different qualifications
 5. Cognitive learning gains between
institutions
Rogaten, J., & Rienties, B. (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1), 161-172. doi: 10.1080/23752696.2018.1484671.

20. Cognitive learning gains
 Cognitive learning gains were
measured in five ways:
 1. Cognitive learning gains within
modules
 2. Cognitive learning gains from
first to second module
 3. Cognitive learning gains within a
qualification
 4. Cognitive learning gains across
different qualifications
 5. Cognitive learning gains between
institutions
Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret
Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.

21. Cognitive learning gains
 Cognitive learning gains were
measured in five ways:
 1. Cognitive learning gains within
modules
 2. Cognitive learning gains from
first to second module
 3. Cognitive learning gains within a
qualification
 4. Cognitive learning gains across
different qualifications
 5. Cognitive learning gains between
institutions
Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.

22. Cognitive learning gains
 Cognitive learning gains were
measured in five ways:
 1. Cognitive learning gains within
modules
 2. Cognitive learning gains from
first to second module
 3. Cognitive learning gains within a
qualification
 4. Cognitive learning gains across
different qualifications
 5. Cognitive learning gains between
institutions
Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.

23. Cognitive learning gains
 Cognitive learning gains were
measured in five ways:
 1. Cognitive learning gains within
modules
 2. Cognitive learning gains from
first to second module
 3. Cognitive learning gains within a
qualification
 4. Cognitive learning gains across
different qualifications
 5. Cognitive learning gains between
institutions
Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.

25. Estimating learning trajectories
Level 1
Level 2
Level 3
Grade1
Student1
Grade3 Grade1Grade2Grade3Grade1Grade2Grade3Grade2
Student2 Student3
Course1 Course2
Grade1Grade2Grade3
Student4
Grade1Grade2Grade3
Student5
Course3
Rogaten, J., & Rienties, B. (2018). Which first-year students are making most learning gains in STEM subjects? Higher Education Pedagogies, 3(1), 161-172. doi: 10.1080/23752696.2018.1484671.

26. Cognitive learning gains
 Cognitive learning gains were
measured in five ways:
 1. Cognitive learning gains within
modules
 2. Cognitive learning gains from
first to second module
 3. Cognitive learning gains within a
qualification
 4. Cognitive learning gains across
different qualifications
 5. Cognitive learning gains between
institutions
Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., . . . Ullmann, T. (2017). Scholarly insight Spring 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
The proportion of variance due to
differences
OU OB
Level 3: Between qualifications 12% 8%
Level 2: Between students 45% 67%
Level 1 Between modules (i.e., within-
student level between modules any
one student completed)
43% 25%
Number of students (n) 18329 1990
Table 1 Proportion of variance explained by qualification, student
characteristics, and across modules (OU, OB, US)

27. abclearninggains.com/

30. Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student
engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028.
69% of what students are
doing in a week is
determined by us, teachers!

32. Click to edit Master
title style
Excellent group
In advance Catching up
Nguyen, Q., Hupych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper
presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award.

33. Click to edit Master
title style
Passed group
In advance Catching up
Nguyen, Q., Huptych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper
presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award.

34. Click to edit Master
title style
Failed group
In advance Catching up
Nguyen, Q., Huptych, M., Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance: A longitudinal study. Paper
presented at the Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK’18), Sydney, Australia, pp. 141-150. Best-paper award.
Vast majority of students do
not follow the course schedule


35. Myth Supported Sample size (n
= )
Published in
1. OU students love to work together No 116,646 Assessment & Evaluation in Higher
Education 2017, Computers in
Human Behavior 2016
2. OU student satisfaction is positively related to
teaching quality, and success in learning outcomes
(e.g., pass rates, retention)
No 111,526 Computers in Human Behavior
2016, 2017
3. Most OU students are making positive learning gains
over time (i.e., as measured by the grades they get)
No 4,222 & 18,329 Higher Education Practices,
Scholarly Insight Report 2017
Spring and Autumn
4. The grades that OU students get are mostly related
to their abilities, effort, cognition, etc. (i.e., what
students do to study)
No 4,222 & 18,329 Higher Education Practices,
Scholarly Insight Report 2017
Spring and Autumn
5. OU Student engagement in Moodle is primarily
determined by students (abilities, effort, cognition, time
availability, etc.)
No 45,190 Computers in Human Behavior
2017
6. Most OU students follow the module schedule when
studying
No 45,190 & 387 Computers in Human Behavior
2017, LAK 2018

37. Implications for practice
• Substantial freedom for students to select “unique” pathways:
some programmes and qualifications have relatively fixed and
structured pathways. Other programmes and qualifications offer
students wide and far reaching freedom to choose (one
qualification had 84 potential pathways to complete a degree).
However, institutions provide limited to no structural support
which pathways would fit students’ needs and abilities.
Recommendation 1: Institutions needs to improve how we communicate to our
students which modules fit with their needs and abilities, and be more explicit
about successful pathways for students to obtain a qualification.
Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson,
Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.

38. Implications for practice
• Alignment of modules within a qualification: students
experience substantially different learning designs, assessment
practices, and workload fluctuations when transitioning from
one module to another.
Recommendation 2: Institutions need to improve how we communicate and
manage the students’ expectations of the learning designs and assessment
practices from one module to another.
Recommendation 3: In the longer term, it would be beneficial to align the module
designs across a qualification based upon evidence-based practice and what
works, thereby allowing smooth transitions from one module to another in a
qualification.
Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson,
Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.

39. Implications for practice
• Alignment of marking within and across modules within and across qualifications:
“embedded expectations”, norms and practice influence marking practices. Across
some qualifications there appears to be a widespread deliberate approach of
making early assessment relatively easy, both within modules (particularly the first
assignment) and within qualifications (particularly the first module). This approach
is intended to reduce drop-out, but may have unintended consequences.
• Furthermore, given that in most modules teachers are marking relatively small
numbers of students, potential misalignments might be present which may not be
immediately apparent when just looking at average grades and the normal
distribution curves.
• Another potential explanation is the increasing difficulty of the material being
assessed may not be completely accounted for in the marks awarded. Final-year-
equivalent modules rightly contain much more difficult material than entry
modules.
Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson,
Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.

40. Implications for practice
Recommendation 4: It is essential that grades are aligned not only within a
module but also across a qualification. For exam boards we recommend to include
cross-checks of previous performance of students (e.g., correlation analyses) and
longitudinal analyses of historical data to determine whether previously
successful students were again successful, and whether they maintained a
successful learning journey after a respective module.
Recommendation 5: We recommend that clearer guidelines and grade descriptors
across a qualification are developed, which are clearly communicated to staff and
students.
Recommendation 6: Given that many students follow modules from different
qualifications, it is important to develop coherent university-wide grade
descriptors and align marking across qualifications.
Rogaten, J., Clow, D., Edwards, C., Gaved, M., Rienties, B. (Accepted with minor revision: 12-07-2018). Do we need to re-imagine university assessment in a digital world? A big data exploration. Margaret Bearman, Phillip Dawson,
Rola Ajjawi, Joanna Tai, David Boud (Eds). Re-imagining University Assessment in a Digital World. Springer.

41. https://www.toyota-global.com/innovation/environmental_technology/electric_vehicle/

42. Where to find “good” evidence
• Test and Learn Evidence Hub
https://openuniv.sharepoint.com/sites/qual-enhance/test-learn-
evidence/Pages/Home.aspx
• Data wranglers website: Scholarly
Insight reports
• http://intranet6.open.ac.uk/learning-teaching-
innovation/main/data-wrangling
• LATIS: Learning and Teaching Innovation and Scholarship
• http://intranet6.open.ac.uk/learning-teaching-
innovation/main/welcome-latis-learningand-teaching-
innovation-and-scholarship

43. abclearninggains.com/

44. Unpacking Six Myths
at the Open University
LTI Series
Thursday 25 October 2018
The Hub Lecture Theatre
@DrBartRienties