The provision of student feedback is a challenging and resource intensive task for any instructor but at the same time it has the potential of significantly improve the overall quality of a learning experience. This challenge is magnified even further in the context of large student cohorts. Current initiatives such as the one captured by the OnTask project have explored how to use data about student engagement to support instructors of large student cohorts in this process. But despite the use of technology there are still important aspects to consider. What is the ideal tone of the message? Should they focus on the material? Assessments? Strategies? How often is idea to send these messages? In this talk we will cover some principles and examples of how instructors are addressing the problem.

1. Using OnTask for Student Coaching
in Large Student Cohorts
Abelardo Pardo (@abelardopardo)
Division of Information Technology, Engineering and the Environment
slideshare.net/abelardo_pardo
GARNET
Teaching and Learning Breakfast Series
Teaching Innovation Unit
28 February 2018

2. Centre for
Change and
Complexity in
Learning
Explore human and artificial cognition to
understand knowledge processes and their
impact on society
Seek practical application and impact

3. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 3
Scaling
Feedback
MatthewCabret
Action
Sensemaking
3

4. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 4
Scaling
Feedback
MatthewCabret

5. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 5
Pardo, A., Bartimote-Aufflick, K., Buckingham Shum, S., Dawson, S., Gao, J., Gašević, D., . . . Vigentini, L. (2018).
OnTask: Delivering Data-Informed Personalized Learning Support Actions. Journal of Learning Analytics, 5(3), 235-249.
ontasklearning.org

6. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 6
Student reactions
Lim, Lisa et al. (2018) “Influence of technology-mediated feedback on students’ self-regulated learning”.
Manuscript in preparation

7. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 7
Thaler, R. H., & Sunstein, C. R. (2008). Nudge. Great Britain: Yale University Press.
“People make good choices in contexts in
which they have experience, good
information, and prompt feedback”
DerekBruffflickr.com

8. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 8
Too much
information
Overstimulation
A.HAS

9. Abelardo Pardo Providing personalised student support in flipped classrooms at scale 9
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: beliefs, techniques, and illusions. Annu Rev Psychol, 64,
417-444. doi:10.1146/annurev-psych-113011-143823
• Understand human
memory and learning
• Know useful
techniques to study
• Know how to monitor
• Understand existing
biases
AlexMihis

10. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 10
JacksonLavarnway

11. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 11
EdDunen

12. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 12
Scaling
Feedback
MatthewCabret
Sensemaking

13. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 13
I'mnik
Sensemaking
Learning
Design
Data

14. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 14
Large number of events per user
2016-08-09 00:05:43.124199+00,989,129.78.56.144,"{""outcome"": ""incorrect"", ""assessment"":
""summative"", ""score"": 44.4444444444444, ""exercise"": ""/data2u/static/exco_exc/DRM/
problem_13_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196619,exco-answer
2016-08-09 00:05:44.140307+00,989,129.78.56.144,"{""score"": 44.4444444444444, ""exercise"": ""/data2u/
static/exco_exc/DRM/problem_03_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196620,exco-view
2016-08-09 00:05:47.76122+00,968,49.182.128.186,"{""question_id"":""COD-numberofwires-eqt_1"",""answer"":
1}",abelardopardo,196621,embedded-question
2016-08-09 00:05:48.861036+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":0}",abelardopardo,196622,embedded-question
2016-08-09 00:05:48.959791+00,989,129.78.56.144,"{""outcome"": ""correct"", ""assessment"":
""summative"", ""score"": 50.0, ""exercise"": ""/data2u/static/exco_exc/DRM/problem_03_c.html"",
""sequence"": ""DRM-exco-C""}",abelardopardo,196623,exco-answer
2016-08-09 00:05:49.539162+00,989,129.78.56.144,"{""score"": 50.0, ""exercise"": ""/data2u/static/
exco_exc/DRM/problem_14_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196624,exco-view
2016-08-09 00:05:49.571138+00,1311,129.78.56.159,"{""url"":""https://flip.ee.usyd.edu.au/elec1601/
Material/COD/index.html""}",abelardopardo,196625,resource-view
2016-08-09 00:05:50.050035+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":0}",abelardopardo,196626,embedded-question
2016-08-09 00:05:51.709295+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":0}",abelardopardo,196627,embedded-question
2016-08-09 00:05:51.962474+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":0}",abelardopardo,196628,embedded-question
2016-08-09 00:05:52.0819+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":""-1""}",abelardopardo,196629,embedded-question
2016-08-09 00:05:53.025356+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":1}",abelardopardo,196630,embedded-question
2016-08-09 00:05:54.756229+00,1311,129.78.56.159,"{""url"":""https://flip.ee.usyd.edu.au/elec1601/
Material/COD/COD_notes.html#range-accuracy-and-precision-of-the-floating-point-
representation""}",abelardopardo,196631,resource-view
2016-08-09 00:05:54.856333+00,989,129.78.56.144,"{""outcome"": ""incorrect"", ""assessment"":
""summative"", ""score"": 50.0, ""exercise"": “"/data2u/static/exco_exc/DRM/problem_03_c.html"",
""sequence"": ""DRM-exco-C""}",abelardopardo,196623,exco-answer

15. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 15
4187,
2016-07-14 00:56:46.341946+00,
{“time”":0,"id":"xEJtdMQMcrs","event":"PLAY"},
https://mycourse.com/Material/HLP/HLP_notes.html,
embedded-video
4187, 2016-07-14 00:56:46.341946+00, VIDEO, xEJtdMQMcrs, PLAY, TOP, W2
Combine logs with design
Learner played a video about topic TOP during Week 2

16. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 16
a) Learner played TOP video, first time, during Week 2
b) Learner played TOP video, first time, during Week 6
c) Learner played TOP video, fifth time, during Week 6
What would you say?

17. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 17
Philhearingflickr.com
Gawronski, B., & Creighton, L. A. (2013). Dual Process Theories. In D. E. Carlston (Ed.), The Oxford handbook of social cognition
(pp. 282-312). New York, NY: Oxford University Press.
CarstenTolkmitflickr.com
Rational
System 1
fast, intuitive,
implicit,
subconscious
System 2
slower, deliberative,
explicit (controlled),
conscious
Automatic

18. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 18
a) Learner played TOP video and created annotations
in Week 2
b) Learner played TOP video and did not create
annotations in Week 2
c) Learner played TOP video, first time, no annotations,
during Week 6
d) Learner played TOP video, fifth time, added
annotations
Annotate the video

19. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 19
I'mnik
Sensemaking
Learning
Design
Data

20. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 20
Hartwig, M. K., & Dunlosky, J. (2012). Study strategies of college students: are self-testing and scheduling related to
achievement? Psychon Bull Rev, 19(1), 126-134. doi:10.3758/s13423-011-0181-y
BenMullins
“self-testing, rereading, and scheduling
of study play important roles in real-
world student achievement.”

21. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 21
a) Learner read the to-do list, played TOP video,
created annotations, ticked the to-do list in Week 2
b) Learner played TOP video, did not create
annotations, didn’t read the to-do list in Week 2
c) Learner played TOP video, read the to-do list for the
first time, no annotations, during Week 6
d) Learner played TOP video for the fifth time, reviewed
the to-do list several times, added annotations…
Create a to-do/to-know page

22. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 22
Data encoding events

23. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 23
If THIS then THAT
Learner played TOP video, did not create annotations,
didn’t read the to-do list in Week 2
THIS: TOP Video Play W2 = 0 AND Annotations W2 = 0
AND Read To-Do W2 == 0
THAT: “It would be good for you to check the video as it
relates to [TOPIC] and we need this to then tackle
[OTHER TOPIC]. Also, check the To-Do list, it may be
handy to…

24. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 24
EdDunen

25. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 25
I'mnik
Sensemaking
Learning
Design
Data

26. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 26
TineIvanic
Sensemaking
Learning
Design
Data

27. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 27
Scaling
Feedback
MatthewCabret
Action
Sensemaking

28. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 28
Biology course
Indicator: Level of persistency
VLP: very log persistency
LP: low persistency
MP: Medium persistency
VHP: Very high persistency

29. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 29
Biology course — Very low persistency
“You may want to try to make some headway in this unit.
The concepts of specialization and evolution are very
important for the course. Charles Darwin’s famous
voyage in the HMS Beagle involved a visit to the now
famous Galapagos Islands. See if you can piece
together some of the clues that inspired Charles Darwin.
It’s important for you to get some understanding of these
ideas as they are essential concepts in biology. Where
do species come from?”

30. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 30
Biology course — Very low persistency
“You may want to try to make some headway in this
unit. The concepts of specialization and evolution are
very important for the course. Charles Darwin’s famous
voyage in the HMS Beagle involved a visit to the now
famous Galapagos Islands. See if you can piece
together some of the clues that inspired Charles
Darwin. It’s important for you to get some
understanding of these ideas as they are essential
concepts in biology. Where do species come from?”

31. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 31
Biology course — Low persistency
“Good initial work with this module. You should probably
take another look at the units such as ‘The Galapagos
Exploration’ and ‘The Birds and the moths’. This unit will
help you develop some understanding of evolution
through natural selection –which is one of the most
important processes (perhaps the most important) in the
natural world.”

32. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 32
Biology course — Low persistency
“Good initial work with this module. You should
probably take another look at the units such as ‘The
Galapagos Exploration’ and ‘The Birds and the moths’.
This unit will help you develop some understanding
of evolution through natural selection –which is one of
the most important processes (perhaps the most
important) in the natural world.”

33. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 33
Biology course — Medium Persistency
“Good work with this module. Would you be able to
anticipate the influence of air pollution on Kettlewell’s
moths? How important are relationships (such as
symbiosis) in the natural world? Lessons such as ‘Peer
pressure in nature’ will help us understand how energy
flows in food webs and the different types of
relationships that provide a cornerstone of the natural
world.”

34. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 34
Biology course — Medium Persistency
“Good work with this module. Would you be able to
anticipate the influence of air pollution on
Kettlewell’s moths? How important are relationships
(such as symbiosis) in the natural world? Lessons
such as ‘Peer pressure in nature’ will help us
understand how energy flows in food webs and the
different types of relationships that provide a
cornerstone of the natural world.”

35. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 35
Biology course — Very High Persistency
“Thorough work with the unit! Would you be able to
describe the relationship between clownfish and their
sea anemone host as a commensalism or a mutualism?
Could you explain why hemophilia seems only to afflict
males (such as in the British royal family of the 19 th
century)? Great work!”

36. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 36
Biology course — Very High Persistency
“Thorough work with the unit! Would you be able to
describe the relationship between clownfish and their
sea anemone host as a commensalism or a mutualism?
Could you explain why haemophilia seems only to
afflict males (such as in the British royal family of the
19th century)? Great work!”

37. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 37
Biology course — Week 5 announcement
“The concepts of specialization and evolution are very important for the course. Charles Darwin’s
famous voyage in the HMS Beagle involved a visit to the now famous Galapagos Islands. See if
you can piece together some of the clues that inspired Charles Darwin. It’s important for you to
get some understanding of these ideas as they are essential concepts in biology. Where do
species come from?
You should probably take another look at the units such as ‘The Galapagos Exploration’ and ‘The
Birds and the moths’. This unit will help you develop some understanding of evolution through
natural selection –which is one of the most important processes (perhaps the most important) in
the natural world.
Would you be able to anticipate the influence of air pollution on Kettlewell’s moths? How important
are relationships (such as symbiosis) in the natural world? Lessons such as ‘Peer pressure in
nature’ will help us understand how energy flows in food webs and the different types of
relationships that provide a cornerstone of the natural world.
Would you be able to describe the relationship between clownfish and their sea anemone host as
a commensalism or a mutualism? Could you explain why haemophilia seems only to afflict males
(such as in the British royal family of the 19th century)?”

38. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 38
Too much
information
Overstimulation
A.HAS

39. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 39
Cone of engagement

40. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 40
Call to action
HenrikDonnestad

41. Abelardo Pardo Using OnTask for Student Coaching in Large Student Cohorts 41
VladKutepov
• OnTask offer the possibility of tackling
feedback for large student cohorts
• Move from “information available” to “THIS is
your information”
• Let the design drive the data, and vice versa
• Think like a coach
• Reach out to TIU for expertise
Conclusions

42. Using OnTask for Student Coaching
in Large Student Cohorts
Abelardo Pardo (@abelardopardo)
Division of Information Technology, Engineering and the Environment
slideshare.net/abelardo_pardo
GARNET
Teaching and Learning Breakfast Series
Teaching Innovation Unit
28 February 2018