Talk exploring how to use data to provide scalable feedback in learning experiences. The solutions explored propose the use of algorithms to enhance how humans instructors provide feedback to students more effectively

1. Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Queen’sUniversityflickr.com
Feedback at scale with a little help
from my algorithms
SusanaFernandezflickr.com
Research Seminar
Universitat Pompeu Fabra
Barcelona Spain
5 May 2016

2. Feedback at scale with a little help from my algorithmsAbelardo Pardo 2
RishiSflickr.com
About me
Teaching
First year engineering course
Computer systems
350 students
Plenary 2 hour lecture +
2 hour tutorial +
3 hour laboratory session
Active learning
Heavy use of technology
Research
Educational technology
Learning analytics
Data-guided feedback
Naturalistic experiments

3. Feedback at scale with a little help from my algorithmsAbelardo Pardo 3
gingiberflickr.com
Feedback and Data
Decision Trees
Data driven rubric
Design and
production

4. Feedback at scale with a little help from my algorithmsAbelardo Pardo 4
gingiberflickr.com
Feedback and Data

5. Feedback at scale with a little help from my algorithmsAbelardo Pardo 5
Ericflickr.com
Systematically low ratings of
the feedback provided to the students

6. Feedback at scale with a little help from my algorithmsAbelardo Pardo 6
Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a
decade of National Studies. University of Melbourne: Centre for the Study of Higher Education.
Eleafflickr.com
The feedback question gets systematically
lower values in student surveys

7. Feedback at scale with a little help from my algorithmsAbelardo Pardo 7
Huxham, M. (2007). Fast and effective feedback: are model answers the answer? Assessment & Evaluation in Higher Education,
32(6), 601-611. doi:10.1080/02602930601116946
• Lateness
• Uncertainty about criteria and
contexts
Loonyhikerflickr.com
• Ambiguity or opacity (What do you
mean?)
• Negativity
Loonyhikerflickr.com

8. Feedback at scale with a little help from my algorithmsAbelardo Pardo 8
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in
Higher Education: Learning for the Longer Term. London and New York: Routledge.
Perceived as an administrative chore
instead of a pedagogical necessity
MarcinWicharyflickr.com

9. Feedback at scale with a little help from my algorithmsAbelardo Pardo 9
JoelSageflickr.com
“sustainability of feedback is under threat… pervasive
student concerns about the provision of feedback in
an era of mass education”
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher
Education: Learning for the Longer Term. London and New York: Routledge.

10. Feedback at scale with a little help from my algorithmsAbelardo Pardo 10
Hounsell, D. (2008). The Trouble with Feedback. New Challenges, Emerging Strategies. Interchange(2).
Feedback has a Cinderella status
Bitslammerflickr.com

11. Feedback at scale with a little help from my algorithmsAbelardo Pardo 11
Active Learning Works

12. Feedback at scale with a little help from my algorithmsAbelardo Pardo 12
AllenLaiflickr.com
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher
Education: Learning for the Longer Term. London and New York: Routledge.
Downward spiral: Student disenchantment mounts
when feedback is uninformative and still,
less feedback is provided
Solution: provide high-value feedback, rethink the role of
student, enhance relation between guidance and feedback

13. Feedback at scale with a little help from my algorithmsAbelardo Pardo 13
BrandonMartin-Andersonflickr.com
• Can data guide the provision of effective feedback
• Can this provision be done at scale?

14. Feedback at scale with a little help from my algorithmsAbelardo Pardo 14
DmitryGrigorievflickr.com
Learning Analytics: measure, collect, analyse data
about learners to understand and improve their
learning and the environment in which it occurs

15. Feedback at scale with a little help from my algorithmsAbelardo Pardo 15
EustaquioSantimanoflickr.com
Collect
Report
Analyze
Act
Refine

16. Feedback at scale with a little help from my algorithmsAbelardo Pardo 16

17. Feedback at scale with a little help from my algorithmsAbelardo Pardo 17
EustaquioSantimanoflickr.com
Collect
Report
Analyze
Act
Refine

18. Feedback at scale with a little help from my algorithmsAbelardo Pardo 18
gingiberflickr.com
Feedback and Data
Decision Trees

19. Feedback at scale with a little help from my algorithmsAbelardo Pardo 19
WilliamMurphyflickr.com
• 13 Week first year Engineering
• Weekly activities (formative/summative)
• Videos, MCQ, Exercises, dashboard
• n = 272, Weeks 2-5 and 7-13
Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable
Predictive Models of Academic Performance. Paper presented at the International Conference on Learning Analytics and
Knowledge, Edinburgh. doi:10.1145/2883851.2883870

20. Feedback at scale with a little help from my algorithmsAbelardo Pardo 20
OliverBraubachflickr.com
Objective
1. Data indicators close to
learning design
2. Predictive model
3. Bridge between model and
application
4. Straightforward delivery method

21. Feedback at scale with a little help from my algorithmsAbelardo Pardo 21
LouishPixelflickr.com
• Event counts from interactive
course material
• Midterm/final exam scores
• Recursive partitioning
• Divide cohort into performance
categories

22. Feedback at scale with a little help from my algorithmsAbelardo Pardo 22
Data collected
• Indicators are directly connected with learning design
• Data structure shaped by the schedule (weeks)
• Data available in a per-week basis
• What is the expected midterm/final score in week n?

23. Feedback at scale with a little help from my algorithmsAbelardo Pardo 23
Result Example
• Week 10
• Predicted score at
leaves (out of 40)
• Conditions at nodes
• If (EXC.in >=22) and
(VID.PL < 8.5) then
score = 6

24. Feedback at scale with a little help from my algorithmsAbelardo Pardo 24
• Each leaf node represents a group of students
with their estimated score.
• Example: 6, 8.3, 8.4, 9.4, 9.9, 10, 15 (out of 40)
• Intervention: suggested work before exam
Result interpretation

25. Feedback at scale with a little help from my algorithmsAbelardo Pardo 25
shabnammayetFlickr.com
RMSE: Root mean square error, MAE: Mean absolute error
Performance

26. Feedback at scale with a little help from my algorithmsAbelardo Pardo 26
gingiberflickr.com
Feedback and Data
Decision Trees
Data driven rubric

27. Feedback at scale with a little help from my algorithmsAbelardo Pardo 27
2 Hours
2 Hours
3 Hours
Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C.
Reidsema, L. Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.
2014 Edition

28. Feedback at scale with a little help from my algorithmsAbelardo Pardo 28

29. Feedback at scale with a little help from my algorithmsAbelardo Pardo 29
1. realtime feedback
2. convey engagement
3. compare with rest of
cohort
4. reset weekly
5. one click away from notes
Approach 1 (2014)
SeanDreilingerflickr.com

30. Feedback at scale with a little help from my algorithmsAbelardo Pardo 30
Tracking

31. Feedback at scale with a little help from my algorithmsAbelardo Pardo 31
Technology
Technology
Week N

32. Feedback at scale with a little help from my algorithmsAbelardo Pardo 32
Weekly real time feedback

33. Feedback at scale with a little help from my algorithmsAbelardo Pardo 33
No statistically significant difference in the rating of feedback (2013 edition,
M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17

34. Feedback at scale with a little help from my algorithmsAbelardo Pardo 34
1. encourage contact
between student and
faculty (forum)
2. uses active learning
techniques
3. gives prompt feedback
4. emphasizes time on task
5. kind, specific, helpful
Approach 2 (2015)Ben Manson flickr.com
BenMansonflickr.com

35. Feedback at scale with a little help from my algorithmsAbelardo Pardo 35
Week N
Technology
Human

36. Feedback at scale with a little help from my algorithmsAbelardo Pardo 36
You should take a more
careful look at how symbols
are encoded in the video.
Would you be able to encode/
decode UAL symbols without
looking at the video?
Good initial work. However,
did you understand the trick
to handle encoding with a
variable number of bits?
Would you be able to provide
an example?
Good work. Would you be
able to come up with your
own machine language and
your encoding scheme?
Remember that it has to be
unambiguous.
Thorough work with the task
about machine language
encoding. Give it a quick
review before the midterm.
Q1 Q2 Q3 Q4
Instructor

37. Feedback at scale with a little help from my algorithmsAbelardo Pardo 37
Algorithm

38. Feedback at scale with a little help from my algorithmsAbelardo Pardo 38
Automatic
Email

39. Feedback at scale with a little help from my algorithmsAbelardo Pardo 39
1. Data collected weekly
2. Email sent at end of week
3. 4 weeks before the midterm
nateOneflickr.com

40. Feedback at scale with a little help from my algorithmsAbelardo Pardo 40
Significant difference in the rating of feedback (2014 edition, M=3.35, SD=1.03;
2015 edition, M=3.82, SD=0.90); t(389.78) = -4.88, p <10-6
Effect size (Cohen’s d) = 0.49. Medium positive effect

41. Feedback at scale with a little help from my algorithmsAbelardo Pardo 41
Significant difference in the midterm score (2014 edition, M=12.80, SD=4.79;
2015 edition, M=13.83, SD=4.89); t(684.5) = -2.86, p < 0.002
Effect size (Cohen’s d) = 0.21. Small positive effect

42. Feedback at scale with a little help from my algorithmsAbelardo Pardo 42
gingiberflickr.com
Feedback and Data
Decision Trees
Data driven rubric
Design and
production

43. Feedback at scale with a little help from my algorithmsAbelardo Pardo 43
DepartmentforBusinessInnovation&Skillflickr.com
• Content creation
• Interactive
• Multi-view (tutor, bilingual)
• Focus on content (not style)
• Platform agnostic
• Design-embedded analytics

44. Feedback at scale with a little help from my algorithmsAbelardo Pardo 44
DavidMichalczukflickr.com
• Markup language
• Batch processing
• Sphinx-doc
• Webdav gateway for publishing
• Self-contained web site
• Extensible language: macros
for design elements

45. Feedback at scale with a little help from my algorithmsAbelardo Pardo 45

46. Feedback at scale with a little help from my algorithmsAbelardo Pardo 46

47. Feedback at scale with a little help from my algorithmsAbelardo Pardo 47
Single source approach
•Plain text
•Version control
•Distributed production
•Suitable for ed designers

48. Feedback at scale with a little help from my algorithmsAbelardo Pardo 48
Back annotation

49. Feedback at scale with a little help from my algorithmsAbelardo Pardo 49
“The interactive quizzes during
class are excellent”
“I love the quizzes”
‘Learning was supported by useful
electronic learning resources’
Agreement increased from 78% to
98%
Professional Practice of Radiography (PG). A/Prof. Mark McEntee

50. Feedback at scale with a little help from my algorithmsAbelardo Pardo 50
https://bitbucket.org/abelardopardo/reauthoring
•Activity duration
•MCQ
•Video embedding
•Difficulty/Usefulness 2D grid
•Tutor view
•Tutor feedback
•Coupled with tracking
• Automatic back-annotation

51. Feedback at scale with a little help from my algorithmsAbelardo Pardo 51
FabienCAMBIflickr.com
• Feedback is delicate and messy
• Data can help scaling
• Good focal point for analytics
• Provide intuitive tools to
instructors
• Authoring is a challenge

52. Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Queen’sUniversityflickr.com
Feedback at scale with a little help
from my algorithms
SusanaFernandezflickr.com
Research Seminar
Universitat Pompeu Fabra
Barcelona Spain
5 May 2016