The Australian Government recently made an ongoing commitment to a suite of innovative, integrated surveys that collect data about students’ experiences of their higher education from the commencement of their qualification to employment. The Quality Indicators for Learning and Teaching (QILT) survey program includes the Students Experience Survey, the Graduate Outcomes Survey, and the Employer Satisfaction Survey. All higher education institutions offering undergraduate and postgraduate courses in Australia, which includes 40 universities and around 105 private providers, are in-scope for the collection. The QILT measures will work together to provide a coherent insight into student engagement, the student experience and post-study outcomes. The challenges of meeting this broad range of requirements to deliver an indicator framework that provides timely evidence for institutions to improve the experiences of current and future students and to position themselves in the higher education landscape will be discussed.

1. Creating a coherent performance indicator
framework for the higher education student
lifecycle in Australia
Sonia Whiteley
The Social Research Centre & The Australian Centre for Applied
Social Research Methods
EAIR 37th Annual Forum in Krems, Austrla
30 August to 2 September 2015

2. About the Social Research Centre
• We are a private, for-profit company owned by
ANU Enterprise (Australian National University)
• Our services include: survey design and execution,
qualitative research, survey data management,
statistical consulting and analytical reporting.
EAIR 2015 2

3. About the Australian Centre for Applied
Social Research Methods
• The Australian Centre for Applied Social Research
Methods (AusCen) provides national leadership in social
research methods and training by:
 Building a world-class team of researchers and graduate students
in social research methodology, applications and techniques
 Developing and validating new and cost-effective data collection
methods
 Increasing the availability and access to secondary data for
research across Australia, and
 Producing a more sophisticated Australian skills base via training
and educational activities.
EAIR 2015 3

4. Overview
1. Higher education performance indicators in Australia
(a very short history)
2. The Quality Indicators for Learning and Teaching
Survey Program
3. Proposed indicator framework & measures
EAIR 2015 4

5. Quality in higher education
• Until the 1980’s debates about the quality of higher
education were essentially invisible with QA processes
managed by each institution
• The first wave of ‘questions about quality’ resulted from
sectoral diversification and public funding
• The second wave is occurring in the context of
massification, regulation and an orientation towards market
assessments of quality
• The focus has shifted from internal peer review to external
performance indicators
EAIR 2015 5

6. Higher education performance indicators‘
EAIR 2015 6
Linke 1991
Perceived Teaching Quality
Graduate Employment Status
Chalmers 2007
Graduate Satisfaction
Student Involvement and Engagement
Employer Satisfaction
Graduate Full-time Employment
Graduate Participation in Further Studies
Graduate Starting Salaries
DEEWR 2009
Student Experience
Quality of Learning Outcomes

7. Higher education performance indicators (2)
Unclear why there have been difficulties creating a
sustainable set of learning and teaching performance
indicators
 Lack of consensus on the nature of the indicators?
 Retro-fitting of existing data against measures?
 Resistance to increased reporting burden?
EAIR 2015 7

8. Quality Indicators for Learning and Teaching
• QILT is the outcome of 20 years of work on higher
education performance indicators
• The most recent review suggested the indicators should
be:
 Fit for purpose
 Consistent
 Auditable
 Transparent
 Timely
…to provide a robust and reliable measure of teaching
performance throughout the Student Life Cycle.
EAIR 2015 8

9. The Student Life Cycle
EAIR 2015 9

10. Our role in QILT
• The Social Research Centre was commissioned by the
Department of Education and Training as the independent
administrator of QILT.
This involves:
 Collecting data
 Reporting on survey outcomes
 Creating, monitoring and updating the QILT website.
EAIR 2015 10

11. QILT Website
EAIR 2015 11

12. QILT Website
EAIR 2015 12

13. QILT Website
EAIR 2015 13

14. What are the QILT surveys?
• The QILT program consists of:
 Student Experience Survey (SES) - measuring the engagement of
current students with the higher education system
 Graduate Outcomes Survey (GOS) – examining graduates’ labour
market outcomes, and
 Employer Satisfaction Survey (ESS) – assessing the employer’s
opinion of the graduates’ generic skills and work readiness.
EAIR 2015 14

15. What are the QILT surveys? (2)
• The QILT survey suite focuses on:
 Commencing and completing undergraduate university students –
Student Experience Survey (SES)
 University graduates – Graduate Outcomes Survey (GOS)
 Employers of recent university graduates – Employers Satisfaction
Survey (ESS)
• All are cross sectional, point-in-time surveys except
the GOS, which is longitudinal.
EAIR 2015 15

16. QILT in the context of the student life cycle
EAIR 2015 16

17. QILT outcomes
For the first time in Australia we will have:
• an integrated and coherent set of surveys providing
feedback at key junctures of the student experience and
beyond
• a centralised and independent approach to survey
administration
• a program of work that encompasses the entire higher
education sector, rather than just focusing on the
universities and
• a comprehensive and current website containing headline
indicators from each of the surveys.
EAIR 2015 17

18. The QILT Survey Program
The Australian Government’s desire to have
objective KPIs that demonstrate students’ engagement
with and experience of courses, and the quality of those
courses from the students’ perspective.
The three QILT surveys work together to provide
a holistic and coherent picture across the entire
student life cycle.
EAIR 2015 18

19. QILT indicator framwork
There are three sets of indicators underlying the Quality in
Learning and Teaching framework:
• Student engagement
• Graduate destinations, and
• Graduate readiness.
EAIR 2015 19

20. QILT indicator sets
EAIR 2015 20
Indicator set Assessment
Stage of
cycle
Benchmarking
SES
Student
engagement
Formative Process Yes
GOS
Graduate
destinations
Summative Outcome Yes
GOS - L
Graduate
destinations
Summative
Secondary
outcome
Yes
ESS
Graduate
readiness
Summative Output No

21. Student Experience Survey - engagement
The Student Experience Questionnaire (SEQ) focuses on
aspects of the university experience that are:
 Measureable;
 Linked to learning and development outcomes; and
 Able to be influenced by universities.
And measures five conceptual domains:
 Teaching Quality
 Learner Engagement
 Student Support
 Learning Resources, and
 Skills Development.
EAIR 2015 21

22. Student engagement indicators
EAIR 2015 22
Measure
Headline indicator
Overall quality of
educational experience
% of students satisfied with the overall quality of
their educational experience
Sub-indicator
Teaching Quality
% of students satisfied with the quality of the
teaching they have experienced
Learner Engagement
% of students satisfied with their engagement with
learning at their institution
Student Support
% of students satisfied with the support they
received at their institution
Learning Resources
% of students satisfied with the learning resources
provided by their institution
Skills Development
% of students satisfied with the skills development
they experienced through university studies

23. Graduate Outcomes Survey - destinations
The higher education sector has collected data on graduate
destinations since 1972 which has been used to:
 Provide a detailed picture of the labour market
 Support continuous improvement through internal and
external benchmarking, and
 Contribute to performance measurement frameworks and
key performance indicators.
The Graduate Outcomes Questionnaire will examine
destinations and provide a more contextualised
understanding of graduate outcomes.
EAIR 2015 23

24. Graduate Destinations ‘model’
EAIR 2015 24

25. Graduate destination indicators
EAIR 2015 25
Measure
Headline indicator
Employment % of graduates working one hour per week or more
Study % of graduates in full-time study
Sub-indicator
Time-related
underemployment
% of graduates that are working less than 35 hours per week,
want more work and are ready to start work immediately
Qualification-related
underemployment
% of graduates involuntarily employed in an occupation with
an AQF level below their attained AQF level
Contextual indicator
Fully employed
% of graduates that are employed and do not want to work
more hours
Qualification-related
underemployment
% of graduates reporting a mismatch between their education
and the education required for their current position
% of graduates reporting a mismatch between their skills and
the skills required for their current position

26. Graduate readiness
• Graduate readiness from an employers perspective is one
of the most creative components of QILT
• Graduates rate their readiness and then provide contact
information for their supervisor to offer similar feedback
• Recruitment for the employer survey is ‘challenging’ and
provides many opportunities for innovation
• The Employer Satisfaction Questionnaire (ESQ) is in the
early stages of development and validation
EAIR 2015 26

27. Graduate readiness indicators
EAIR 2015 27
Measure
Headline indicator
Employer satisfaction % of employers who agree that the graduate was
prepared for their job
Sub-indicator
Foundation skills % of employers who agree that the graduate
demonstrates foundation skills
Adaptive Skills % of employers who agree that the graduate
demonstrates adaptive skills
Teamwork & Interpersonal
Skills
% of employers who agree that the graduate
demonstrates teamwork and interpersonal skills
Technical Skills & Domain
Specific Knowledge
% of employers who agree that the graduate
demonstrates technical skills and domain specific
knowledge
Employability Skills % of employers who agree that the graduate
demonstrates employability and enterprise skills

28. Graduate readiness indicators (2)
Additional contextual indicators – readiness expectations
 % match between employers and graduates agreeing
that the graduate demonstrates foundation skills
 % match between employers and graduates agreeing
that the graduate demonstrates adaptive skills
 % match between employers and graduates agreeing
that the graduate demonstrates teamwork and
interpersonal skills
EAIR 2015 28

29. Integration of indicator sets
EAIR 2015 29
Student
engagement
Graduate readiness
Graduate’s perspective
Graduate readiness
Employer’s perspective
Skills
Development
Foundation Skills
Adaptive Skills
Teamwork &
Interpersonal Skills
Foundation Skills
Adaptive Skills
Teamwork &
Interpersonal Skills

30. QILT data collection periods
EAIR 2015 30
Domain 2015 2016 2017
Student Experience Survey
August –
September
August –
September
August –
September
Graduate Outcomes Survey
April (trial) and
October rounds
April and
October rounds
April and
October rounds
Employer Satisfaction Survey
April (instrument
trial) and
October rounds
April and
October rounds
April and
October rounds

31. Key challenges
Key challenges include (but are not limited to):
• effectively managing a data collection program that is
active for 10 months of the year, covering an in-scope
population of up to 1 million potential respondents
• including new and diverse stakeholders in unfamiliar
quality and data management processes,
• centralising disparate legacy approaches to collecting
graduate feedback, and
• supporting institutions to effectively engage with and
analyse the data.
EAIR 2015 31

32. To summarise…
QILT is an ambitious, national program of centralised surveys
to support the measures underpinning the indicator
framework. The ability to link views from key stakeholders
during and after contact with the higher education system
provides an unprecedented opportunity to understand
student experiences and outcomes.
EAIR 2015 32

33. Thank you for listening!
Questions?
sonia.whiteley@srcentre.com.au
EAIR 2015 33