1. Assessment of Higher Education
Learning Outcomes
Hamish Coates
coatesh@acer.edu.au
acer.edu.au/highereducation

2. Rationales for
change
Building Building
capacity and assessment
improvement collaborations

3. Rationales for
change

4. What is the “best”
university in your
country? How do you
know?

5. An experiment
95%
65%

6. “Standards”, not “standardisation”
Generalising the
Performance
principles of good
assessment will
encourage
diversification and
excellence
Minimum standard
Score distributions for eight groups
(institutions, faculties, subjects, etc.)

7. Commitment to
effectiveness
Effectiveness
Little data Happiness data
data
Elite Mass Universal

8. Data for
transparency

9. Plan Data for
leadership
Improve Act
Evaluate
Hunch

10. Enhanced management, evaluation and quality
monitoring – to do more, better, with less
Higher education growing in significance and Shaping
scale: clear rationales for increasing output complexities
and ensuring quality outcomes
Huge cost and competitive pressures – evidence-based
management helps expand provision with quality
Further reliance on flawed, simplistic rankings will
constrain growth and prosperity – robust
multidimensional perspectives required
Internationalisation pervades all facets of
teaching and learning, and graduate outcomes –
international perspectives are vital

11. Building
assessment
collaborations

13. Are valid and reliable cross-cultural/-institutional/-
linguistic comparisons of HE learning outcomes
scientifically and practically feasible?
AHELO at
a glance
An international feasibility study – not a ‘pilot’
Conceived ~2005, announced 06, scoping 06-09, work 10-13
>300 individuals involved
directly
~270 HEIs
For students near end of ‘bachelor degree’, also faculty, national
managers, institutional coordinators, policymakers, expert advisors, etc.
Up to 30,000 leaders,
faculty, students
Design, develop, validate and evaluate Many thousands of public
frameworks,
tests, infrastructure and processes stakeholders
~$30m total cost
HEI reports (not for systems), along with international reports and
materials, and recommendations for a main study

14. „Strands‟ of testing
Generic Skills Economics Engineering
Contextual dimension
National Context Information
Institution Context Instrument
Faculty Context Instrument
Student Context Instrument

15. Framework and
Tuning AHELO / QAA frameworks
Curriculum documents item development
Accreditation systems
Discipline research
Item creation
• Outcome specification
• Document analysis • Gather existing materials • Qualitative testing
• Consultation • Item workshops • Quantitative testing
• Synthesis, review • Technical review • Operationalisation
• Framework mapping
Instrument
• Adaptation, translation validation
Framework
creation • Verification
Authentic, hybrid item types
‘Above content’ reasoning
Focus on ‘common core’

16. Linguistically diverse…
AHELO Arabic Dutch English Finish
world map Flemish Italian Japanese Korean
Norwegian Russian Slovak Spanish
+ Abu Dhabi…

17. Building
capacity and
improvement

18. Institutional Change
positioning
horizons
System Faculty
monitoring improvement
Assessment
collaborations
Stakeholder Policy
engagement research
Learner
feedback

19. Building
communities

20. Building
awareness

21. Reporting results
http://oecdbetterlifeindex.org

22. Assessment
Build frameworks collaborations
Produce items
Analyse and
report
Select and use
Types of
Definitions Tasks Processes Results
sharing

23. How
Preparation
Establish assessment
coordinating centre
Translate and collaborations
validate instruments work
Assessment
Select students
Engage institutions
Administer secure
Train institution test
coordinators
Score responses
Prepare for testing
Verify and provide Prepare multilevel
Reporting
data benchmarking
reports
Distribute reports
to institutions
Interpretation for
monitoring and
improvement

24. Assessment of Higher Education
Learning Outcomes
Hamish Coates
coatesh@acer.edu.au
acer.edu.au/highereducation