Assessment and feedback re-designs for the generative AI era

1. Assessment and feedback
re-designs for the
generative AI era
@CarlessDavid
Faculty of Education, HKU
CUHK Generative AI Conference
June 8, 2023
The University of Hong Kong

2. Designing learning environments
Assessment and feedback designs are
firmly interlinked
(After Boud & Molloy, 2013)
The University of Hong Kong

3. Key proposition
Assessment adjustments & principles
that are positive in their own right
&
Cater for the realities of the GenAI era
The University of Hong Kong

4. Assessment in higher education …
An impossible mission? (Barnett, 2007)
The University of Hong Kong

5. REDUCE ASSESSMENT
OVERLOAD
The University of Hong Kong

6. ChatGPT & Assessment reform
Students struggle with assessment overload
The University of Hong Kong

7. Tackling the assessment arms-race
Proliferation of assessment coursework
Teachers compete for student attention,
using grades as control & reward (Harland
et al. 2015; Harland & Wald, 2021)
The University of Hong Kong

8. Barriers to reducing assessment
Reluctance to relinquish power
Covering and assessing content
The University of Hong Kong

9. Less can be more
Reducing content & quantity of assessment
is not lowering standards
The greatest enemy of understanding is
content coverage (Howard Gardner)
The University of Hong Kong

10. MAKING TIME & SPACE FOR
ASSESSMENT RENEWAL
The University of Hong Kong

11. Assessment re-designs
Process as well as product
Assessment co-design with students
New feedback possibilities
The University of Hong Kong

12. Cumulative assessment designs
Scaffolded series of tasks focused on students’
thinking processes in developing artefacts
(Lodge et al. 2023)
The University of Hong Kong
Week 4 Week 7 Week 9
Task 1 Task 2 Task 3

13. Process & product
Evidence of iterative cycles of drafting & re-
drafting
Digital traces e.g. Google Drive ‘version
history’ (Sayers, 2023)
The University of Hong Kong
Feedback
spirals

14. Co-design of assessment
Students as co-learners
Students as co-creators
Students as partners
The University of Hong Kong

15. Digital oral assessment
Podcasts
Vlogs
Oral assessment should play a larger role in
(engineering) assessment (Nikolic et al. 2023)
The University of Hong Kong

16. ChatGPT + group assessment
Pairs or trios working with GenAI
Complex learning cannot be accomplished
in isolation (Boud, 2000)
The University of Hong Kong

17. Programmatic assessment
Assessment mapping:
Cumulative … coherent
The University of Hong Kong

18. AUTOMATED FEEDBACK
LITERACIES
The University of Hong Kong

19. Student feedback literacy
Understandings, capacities & dispositions to
make the most of feedback opportunities of
different kinds (Carless & Boud, 2018)
The University of Hong Kong

20. My embryonic research
Enhancing synergies between effective
feedback & automated feedback practices
Case studies: education, medicine, science
Internally funded Teaching Development
Grant
The University of Hong Kong

21. Main research goal
The development of a framework for student
automated feedback literacies
Cf. automated feedback literacy (Shibani,
Knight & Buckingham Shum, 2022)
The University of Hong Kong

22. Defining automated feedback literacies
Capacities to engage in dialogue with
automated systems, critically evaluate
outputs, and utilize them appropriately to
enhance work, knowledge or thinking.
The University of Hong Kong

23. Automated feedback literacies
What are student feedback literacies for
principled use of automated feedback?
The University of Hong Kong

24. Student automated feedback literacies (draft)
1. Appropriate prompts & continuing dialogue …
2. Critical engagement with AI outputs …
3. Co-learning with others …
4. Reflection and self-assessment …
5. Principled follow-up actions …
The University of Hong Kong

25. Implications
The University of Hong Kong

26. Need to rebuild trust
How is trust developed across an institution?
Partnerships of mutual respect
The University of Hong Kong

27. Integrity is important but …
“Focusing on catching cheating is misplaced
effort”
Sir Tim O’Shea
How are HE leaders responding to
generative AI? – HEPI
The University of Hong Kong

28. Tackling cheating concerns
The University of Hong Kong

29. What leadership is needed?
Agile learning for unknown futures
Quiet leadership
Humility & flexibility
The University of Hong Kong

30. Concluding summary
+ reduce assessment overload
+ design for quality learning
+ build trust
+ partner with students in AI
The University of Hong Kong

31. References
Barnett, R. (2007). Assessment in higher education: An impossible mission? In Boud, D. & Falchikov, N. (Eds).
Rethinking assessment in higher education. Routledge.
Boud, D. (2000). Sustainable assessment: Rethinking assessment for the learning society. Studies in Continuing
Education, 22(2), 151-167.
Boud, D. & Molloy, E. (2013). Rethinking models of feedback for learning: The challenge of design. Assessment &
Evaluation in Higher Education, 38(6), 698-712.
Carless, D. (2019). Feedback loops and the longer-term: Towards feedback spirals. Assessment and Evaluation in
Higher Education, 44(5), 705-714. https://doi.org/10.1080/02602938.2018.1531108
Carless, D. and Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback.
Assessment and Evaluation in Higher Education, 43(8), 1315-1325.
https://doi.org/10.1080/02602938.2018.1463354.
Harland, T. & Wald, N. (2021) The assessment arms race and the evolution of a university’s assessment
practices, Assessment & Evaluation in Higher Education, 46:1, 105-117, DOI: 10.1080/02602938.2020.1745753
Lodge, J., Thompson, K. & Corrin, L. (2023) Mapping out a research agenda for generative AI in tertiary education,
AJET, 39(1), 1-8. https://ajet.org.au/index.php/AJET/article/view/8695
Nikolic, S. et al. (2023). ChatGPT versus engineering education assessment. European Journal of Engineering
Education, https://doi.org/10.1080/03043797.2023.2213169
Sayers, D. (2023). A simple hack to ChatGPT-proof assignments using Google Drive.
https://www.timeshighereducation.com/campus/simple-hack-chatgptproof-assignments-using-google-drive
Shibani, A., Knight, S. & Buckingham Shum, S. (2022) Questioning learning analytics? Cultivating critical engagement
as student automated feedback literacy. LAK 22, 12th International Learning Analytics and Knowledge Conference,
https://dl.acm.org/doi/10.1145/3506860.3506912
The University of Hong Kong

32. QUESTIONS
&
COMMENTS
The University of Hong Kong

33. Criticality
“One of the strengths of ChatGPT is that you
don’t know if what it’s telling you is true. We
can use ChatGPT to enable students to
think critically.”
Tim O’Shea, HEPI blog
The University of Hong Kong

34. Staged assessment example
Stage 1: abstract or elevator pitch
Stage 2: annotated bibliography
Stage 3: draft for peer review & AI review
Stage 4: revise & submit
The University of Hong Kong

35. Teacher automated feedback literacy
Modelling & coaching effective use of AI
Motivating or incentivizing appropriate &
ethical use of AI
The University of Hong Kong

37. Alternative assessment & exams
Exams have a long history but are they still
fit for purpose?
“I don’t want to memorize for an exam: I’ve
spent 15 years doing that in school”.
(Business student, Carless, 2015, p. 125)
The University of Hong Kong

38. The University of Hong Kong

39. Concept map of Authentic Assessment
Adapted from Eddy & Lawrence (2013)
The University of Hong Kong
Assessment as
Process
Contextualised
Tasks
Peer & self-
evaluation
Choice and
Flexibility
Students as
Creators

40. The University of Hong Kong