An invited presentation for the International Educational Assessment Network, 26th September, 2023.

1. The Mini-Summit: AI in Assessment
Professor Mark Brown
Institute of Education
Dublin City University
26th September 2023

2. Two Truths and a Lie
I’m no expert but I have a growing
research interest in AI in education
I have a strong track record of research
on digital forms of Assessment
I haven’t taught in the schooling
sector for over 25 years

4. https://jime.open.ac.uk/articles/10.5334/jime.798

5. Ubiquitous

6. A few health warnings
Learning the art of brushing both ways
Untubing AI in Assessment:
Developing positive habits for healthy minds

7. Key messages
1. Avoid the trap of AI-centric thinking.
2. Proactively own, shape, and influence the narrative regarding
the assessment challenges that AI is attempting to solve.
3. Recognise that all was not well with Assessment (and
Feedback) practices before the emergence of Generative
Artificial Intelligence.
Educators need to…

8. • Training memories
• Limited relevance
• Lack of challenge
• Promoting transfer
• Academic integrity
• Culture of banking
• Playing the game
A
c
q
u
i
s
i
t
i
o
n
P
a
r
t
i
c
i
p
a
t
i
o
n
Knowledge Creation

9. 1. A few health warnings

10. 1. A few health warnings
• 92% of OER content in English
• Only 10 of 7,000 languages have
been used by GenAI
• Latest GenAI tools are less prone to
fake knowledge
• AI generated text can’t always be
easily detected
• Adoption of authentic assessment is
not AI proof
• Bad for the environment with
ChatGPT-3 generating 85,000kg of
CO2 emissions
• GenAI is not freely accessible
• ChatGPT is not just a big database
• ChatGPT tends to pose itself falsely
as a real person
• Few of us understand the ‘black box’
and can keep up with developments
• There are serious biases,
discrimination, privacy and ethical
issues
• Risk of propagating inequity and
marginalising those with already
marginalised voices

11. 2. Learning the art of brushing both ways
https://unsplash.com/photos/IW1uAv88Z0o

12. 2. Learning the art of brushing both ways
‘Design In’ ‘Design Out’
• Students can use GenAI tools in their
assessments
• Clear guidelines must be provided to
students (and parents and
caregivers)
• Potential biases, discrimination and
ethical issues have been considered
as well as they can be
• Students should NOT use GenAI
tools in their assessments
• Some assessments may need to be
redesigned to minimize their
vulnerability
• Students should not be asked to
undertake things that can be easily
done with GenAI tools

13. https://www.independent.ie/irish-news/education/ai-fears-force-education-minister-to-abandon-shake-up-of-leaving-cert-assessments/a1200968142.html
https://www.irishtimes.com/ireland/education/2023/09/20/leaving-cert-norma-foley-to-accelerate-reforms-without-teacher-based-assessment/
Stop the Press

14. https://www.bbc.co.uk/newsround/66107886
Risk of short-termism
Evidence of knee-jerk reactions
Photo by Henrik Lagercrantz on Unsplash

15. • In the face of AI, should we try to defend our existing Assessment
practices?
Straw poll…
• Is it possible to avoid spillage across ‘design in’ and ‘design out’
approaches, especially given AI’s ubiquity?
• To what extent do we risk a potential loss of confidence
locally and nationally in existing assessment practices if
we fail to appropriately respond to AI?

16. Design in…
2022
DOI: 10.1111/ejed.12533

17. • From onerous to feasible
• From discrete to continuous
• From uniform to adaptive
• From inauthentic to authentic
• From antiquated to modern
What are the assessment
problems we face?
https://doi.org/10.1016/j.caeai.2022.100075

18. Henderson (2023) asks,
“What labours are involved that can be offloaded or co-constructed with AI?”
https://www2.ed.gov/documents/ai-report/ai-report.pdf

19. “If our educators’ evaluative expertise through the application of
the assessment criteria and standards cannot distinguish
between raw AI outputs compared with what students bring to
the task (with or without those AI) – then we have a problem
with our assessment designs – not with AI.
(Professor Michael Henderson, 2023)
“WHY SHOULD WE ASSESS SOMETHING THAT
AI CAN DO JUST AS WELL OR BETTER?”
https://der.monash.edu/defending-assessment-from-ai

20. Designing assessment for an AI world…
• Re-visiting what we are trying to assess
• Emphasising the process - product is a poor proxy of learning
• Finding ways of assessing the uniqueness of human processing
• Empowering learners to own the assessment process
• Focusing more on creative assessment designs – that elicit
evidence of learning that matters for ‘a life worth living’

21. https://creativehecommunity.wordpress.com/2023/06/23/oa-book-101-creative-ideas-to-use-ai-in-education/

22. Augmenting
Rather than Automating
• Learning to ask the right questions
• Learning how to make good use of
a devil’s advocate
• Learning to make judgments of
what counts as quality
• Learning that effortlessness is not
always the goal to strive for
• Learning to find the light through
the gaps/cracks
https://unesdoc.unesco.org/ark:/48223/pf0000385146

23. 3. Developing positive habits for healthy minds
Photo by Jelleke Vanooteghem on Unsplash

24. 3. Developing positive habits for healthy minds
https://lx.uts.edu.au/collections/artificial-intelligence-in-learning-and-teaching/resources/five-principles-for-effective-ethical-use-generative-ai/
Five principles…

25. Assessment
https://op.europa.eu/en/publication-detail/-/publication/d81a0d54-5348-11ed-92ed-01aa75ed71a1/language-en

27. https://www.unesco.org/en/digital-education/ai-future-learning/competency-frameworks

28. Actions…
• Shaping regulations and policies
• Critical review of Assessment practices
• Actively engaging in communities of practice
• Supporting teachers’ professional development
• Developing, implementing and researching pilot initiatives
How?

29. Assessment
in
Change
ENTANGLED
ASSESSMENT
FUTURES
Assessment
for
Change
Final remarks…

32. Two Truths and a Lie
I’m no expert in AI but I have a
growing research interest in the area
I have a strong track record of research
on digital forms of Assessment
I haven’t taught in the schooling
sector for over 25 years

33. Go raibh maith agaibh!
Thank you