Technology mediation allows to capture comprehensive data sets about interactions occurring in learning experiences. Although these data sets have the potential of increasing the insight on how learning occurs, their use strongly depends on two aspects: the data has to be properly situated in the learning design, and the insights derived need to be translated into actions. In this talk we will explore how to establish this connection for the case of the provision of feedback. We will approach the problem from the point of view of intelligence amplification, that is, how data can support instructors to provide better support to learners through feedback. The talk will discuss some preliminary results from the Ontasklearning.org project.

1. Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
The role of data in
the provision of
feedback
at scale
WallyGobetzflickr.com
ASCILITE Webinar
11 April 2017

2. Abelardo Pardo The role of data in the provision of feedback at scale 2
Learning
Analytics
Situated
Actions
Feedback Examples
theilrflickr.com

3. Abelardo Pardo The role of data in the provision of feedback at scale 3
Learning
Analytics
Situated
Actions
theilrflickr.com

4. Abelardo Pardo The role of data in the provision of feedback at scale 4
DmitryGrigorievflickr.com
Learning Analytics: measure, collect, analyse data
about learners to understand and improve their
learning and the environment in which it occurs

5. Abelardo Pardo The role of data in the provision of feedback at scale 5
DerekBruffflickr.com
Understanding and
Improving Learning
Algorithms
Dashboards
Clickstreams
Multimodal
SNA
Writing analytics
Learning
Designs

6. Abelardo Pardo The role of data in the provision of feedback at scale 6
JamesRhodesflickr.com
Let’s not forget:
Learning analytics are
about learning
Gašević, D., Dawson, S., & Siemens, G. (2015). Let's not forget: Learning analytics are about learning. TechTrends, 59(1), 64-75.

7. Abelardo Pardo The role of data in the provision of feedback at scale 7
DerekBruffflickr.com
Learning is situated in
interactions with others and
relationships to the environment
Ellis, R. A., & Goodyear, P. (2013). Students' Experiences of E-learning in Higher Education. New York and London: Routledge

8. Abelardo Pardo The role of data in the provision of feedback at scale 8
ChrisMooreflickr.com
Learning environment
has a profound effect
on how data is analysed

9. Abelardo Pardo The role of data in the provision of feedback at scale 9
senscienceflickr.com
Marzouk, Z., Rokovic, M., Liaqat, A., Vytasek, J., Samadi, D., Stewart-Alonso, J., . . . Nesbit, J. C. (2016). What if learning analytics
were based on learning science? Australasian Journal of Educational Technology, 32(6).
• Setting goals & monitoring progress
• Distributed practice
• Retrieval practice
• Prior knowledge for reading
• Comparative evaluation of writing
• Collaborative Learning

10. Abelardo Pardo The role of data in the provision of feedback at scale 10
JoshuaRappenekerflickr.com
Data Understanding
Actions
Improving

11. Abelardo Pardo The role of data in the provision of feedback at scale 11
Learning Analytics Model

12. Abelardo Pardo The role of data in the provision of feedback at scale 12
Learning
Analytics
Situated
Actions
Feedback
theilrflickr.com

13. Abelardo Pardo The role of data in the provision of feedback at scale 13
Hattie, J. A. (1999). Influences on student learning. Inaugural professorial address, University of Auckland, New Zealand
If you could choose one…
• More than 500 meta-analyses
of student achievements
• 100 factors with potential
influence
• Feedback in top five
• (74 meta-analyses) Most
effective form: video, audio,
computer-assisted
instructional feedback, and/or
related goals

14. Abelardo Pardo The role of data in the provision of feedback at scale 14
PaulaGrubbflickr.com
"Feedback is a process whereby learners obtain
information about their work in order to appreciate the
similarities and differences between the appropriate
standards for any given work, and the qualities of the
work itself, in order to generate improved work"
Boud, D., & Molloy, E. (Eds.). (2013). Feedback in Higher and Professional Education: Understanding it and doing it well.
London and New York: Routledge.

15. Abelardo Pardo The role of data in the provision of feedback at scale 15
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112.
doi:10.3102/003465430298487
ChrisBallardflickr.com
Feedback Levels
1. Task Level (understanding, performance)
2. Process Level (what to do to understand, perform)
3. Self-regulation level (detecting and directing effort)
4. Self level (personal evaluation and affect)

16. Abelardo Pardo The role of data in the provision of feedback at scale 16
Nicol, D., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good
feedback practice. Studies in Higher Education, 31(2), 199-218. doi:10.1080/03075070600572090

17. Abelardo Pardo The role of data in the provision of feedback at scale
Good feedback practice
1. Clarify what good performance is
2. Facilitate self-assessment
3. Deliver high quality feedback information
4. Encourage teacher and peer dialogue
5. Encourage positive motivation and self-esteem
6. Provide opportunities to close the gap
7. Use information to improve teaching
17
Nicol, D., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good
feedback practice. Studies in Higher Education, 31(2), 199-218. doi:10.1080/03075070600572090

18. Abelardo Pardo The role of data in the provision of feedback at scale 18
Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans. International Journal of Artificial Intelligence in Education. doi:
10.1007/s40593-016-0105-0
Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: A Review of
Evidence-Based Learning Analytics Interventions at the Open University UK. Journal of Interactive Media in Education, 2016(1). doi:
10.5334/jime.394
Change in focus, from artificial intelligence to
intelligence amplification
Tutors are informed about students struggling but it is up to
them to decide if/when to act

19. Abelardo Pardo The role of data in the provision of feedback at scale 19
Learning
Analytics
Situated
Actions
Feedback Examples
theilrflickr.com

20. Abelardo Pardo The role of data in the provision of feedback at scale 20
Tracking

21. Abelardo Pardo Can data help me learn? 21
Jovanovic, J., Gašević, D., Pardo, A., Dawson, S., & Mirriahi, N. (In Press. Accepted 20/Feb/2017). Learning Analytics to Unveil Learning
Strategies in a Flipped Classroom. Internet and Higher Education. doi:10.1016/j.iheduc.2017.02.001
Analyse Learning Strategies

22. Abelardo Pardo The role of data in the provision of feedback at scale 22
Report to students
Kahn, I., & Pardo, A. (2016). Data2U: Scalable Real time Student Feedback in Active Learning Environments. Paper presented at the
International Conference on Learning Analytics and Knowledge, Edinburgh, UK. doi:10.1145/2883851.2883911

23. Abelardo Pardo The role of data in the provision of feedback at scale 23
No statistically significant difference in the rating of feedback (2013 edition,
M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17

24. Abelardo Pardo The role of data in the provision of feedback at scale 24
You should take a more
careful look at how symbols
are encoded in the video.
Would you be able to encode/
decode UAL symbols without
looking at the video?
Good initial work. However,
did you understand the trick
to handle encoding with a
variable number of bits?
Would you be able to provide
an example?
Good work. Would you be
able to come up with your
own machine language and
your encoding scheme?
Remember that it has to be
unambiguous.
Thorough work with the task
about machine language
encoding. Give it a quick
review before the midterm.
Q1 Q2 Q3 Q4
Instructor

25. Abelardo Pardo The role of data in the provision of feedback at scale 25
Algorithm

26. Abelardo Pardo The role of data in the provision of feedback at scale 26
Automatic
Email

27. Abelardo Pardo The role of data in the provision of feedback at scale 27
Helpful feedback
Effect size (Cohen’s d) = 0.49.
Medium positive effect
Midterm Scores
Effect size (Cohen’s d) = 0.21.
Small positive effect
Pardo, A., Jovanovic, J., Dawson, S., Gašević, D. & Mirriahi, N. (2017). Using Learning Analytics to Scale the Provision of Personalised
Feedback. Manuscript in preparation

28. Abelardo Pardo The role of data in the provision of feedback at scale 28
What if I tell you…

29. Abelardo Pardo The role of data in the provision of feedback at scale 29
ontasklearning.org

30. Abelardo Pardo The role of data in the provision of feedback at scale 30
IF [video 3.9.5 not watched] THEN “…."
ontasklearning.org

31. Abelardo Pardo The role of data in the provision of feedback at scale 31
• Support instructors to
create personalised
feedback
• Simple rule-base
knowledge encoding
• Provide appropriate
view of data sources
• Scale to large and highly
diverse cohorts
• Will be released as
open-source project
Q3/4 2017
• First pilots in Q1/2 2017
• Tutorial in LAK 2017
• Contact us if interested
ontasklearning.org

32. Abelardo Pardo The role of data in the provision of feedback at scale 32
BillTyneflickr.com
• Data to understand and improve learning needs to
be situated and connected to actions
• Feedback is a powerful focal point to drive data-
supported decisions
• Explore the design of theory-driven learning
analytics tools that connect data with support
actions

33. Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
The role of data in
the provision of
feedback
at scale
WallyGobetzflickr.com
ASCILITE Webinar
11 April 2017