The document discusses the need for evidence-based strategies to improve organizational performance in teaching and learning. It notes there is significant variation in learner outcomes across organizations. Developing practices supported by robust evidence is important for achieving synergies between individual practice improvement and organizational change that benefits learners. The document advocates establishing projects focused on collecting evidence of learner benefit to drive sustainable changes in practice.

1. Developing evidence-based strategies to support improved
organisational performance in teaching and learning
Building Futures: Hindsight – Insight – Foresight
National Learning and Teaching Conference, Nelson October 2012
Peter Coolbear

2. The challenge
Achieving synergies
between
individual improvement of practice
and
organisational change
for
the benefit of tertiary learners
October 2012

3. Plan of presentation
• The context – an inefficient tertiary system which works
well for many learners but very poorly for others
• The need for evidence-based improvement
• What constitutes good evidence?
• Ensuring our projects contribute to sustainable change
• Changing expectations of 21st century vocational
educators
October 2012

4. The context …
• Significant advances in course and qualification
completion rates, but still wide variation:
o Course completions <50% - 100%
o Qualifications 12% - 100%
• Lack of parity of success for Māori and Pacific
learners – what’s the gap in your organisation?
• Low progression to success at higher levels of study
• In creasing lack of confidence about consistent
academic standards
• The inevitability of continued change
October 2012

5. Since its establishment in 2007, Ako Aotearoa has
provided $5m + funding for 190 projects to
improve tertiary teaching and learning
• Projects designed to support evidence-based
improvement of practice
We’ve learnt a lot!
October 2012

6. What we’ve learnt …
• We work within a system that allows great teaching and
learning to happen
• We are not very strategic about improving tertiary teaching
• We do not do enough to share good practice
• We work within a system that allows the mediocre (or worse)
• We work in a system that is highly fragmented
• The NZ research base in tertiary education is still quite weak
and has limited impact on practice
October 2012

7. Variability of learner outcomes is the immediate problem,
but is not the only fix
Excellence
Led by the profession and
organisations, enabled by
government
Led by
government
Threshold of
What also needs to be done
acceptability
What the Minister sees
as the present state
October 2012

8. Why is it so important?
It’s about moral purpose ......
• Tertiary education offers opportunities for empowerment
• for individuals
• for whanau
• for communities
• for New Zealand as a whole
It’s also about New Zealand’s international reputation
October 2012

9. Why is evidence of learner benefit fundamental?
October 2012

10. Variation in qualification completion rates for Level 1-3
students within public tertiary providers
90%
80%
Qulaification Completion Rate
70%
60%
50%
40%
30%
20%
10%
0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Providers
Data for 2009 (TEC)
October 2012

11. The policy - research - practice disconnect
Practice
Policy
October 2012

12. Evidence-based enhancement of teaching and
learning
Partnership and continuing dialogue between
professional practitioners and their organisations
• High performing practitioners and tertiary education
organisations continually test their assumptions
about how and how effectively their learners are
learning
October 2012

13. Testing assumptions and effective dialogue relies on
robust, easily understood evidence
• For 70% + traditional practice will not change
without evidence that something else is better
[aid theory]
• Organisational decisions to redirect resources will
(hopefully!) not be made without evidence
[no theory]
October 2012

14. Characteristics of sound evidence
• Focussed on student achievement and outcomes
o For the individual learner
o For the employer
o For the community
• Uses robust, transparent, purposeful and well
validated data collection techniques
• Uses mixed methods (qualitative + quantitative)
• Is action focussed
October 2012

15. Meaningful critical self-reflection
Experienced practitioners often do this intuitively and
informally….
“ Well that went well: there was some really great interaction in
that class and the formative assessment demonstrated that
the group really understood those key concepts”
“ There’s some feedback here from their employers that Joe and
Lisa had some difficulties setting the levels on that formwork
correctly, yet both breezed through the class exercise. Clearly
I need to do something different to help them translate theory
into practice.”
October 2012

16. Evidence-based Enhancement of Teaching and
Learning
A Tertiary Practitioner’s Guide
to Collecting Evidence of
Learner Benefit
Individuals
Organisations
October 2012

17. Organisations need to do this more purposefully and
formally
What works?
What works most effectively?
Can we make it business as usual and sustainable?
• What’s the RoI?
• Is there anything we can stop doing if we do this
instead?
• What are my levers to ensure this happens in the way I
intend?
October 2012

18. So what do we do?
• Systematise the informal evidence
– Students’ stories have validity
– Student contributions to class discussion
– Student preferences for different activities
– Employer feedback
– Alumni feedback
October 2012

19. So what do we do?
• Develop new sources of evidence
– Student engagement – e.g. AUSSE
– Critical incident questionnaires
– Research based in cognitive psychology
– Graduate surveys
– Employer surveys
October 2012

20. An example of robust and rich evidence collection:
• Collects data from students about how students are learning
• Has a very strong research base
• Used by all Australian Universities and now all New Zealand
Universities + pilots in NZ ITPs and PEPs
• International benchmarking against the NSSE used in North
America
• Most highly validated, data-rich student survey instrument
available to tertiary education
• Student satisfaction is just one dimension of thirteen others
• Specifically designed to start organisational conversations.
October 2012

21. October 2012

22. So what do we do?
• Locate evidence and projects firmly in an
organisational context.
October 2012

23. Guide to change projects
• Creating sustainable
change to improve
outcomes for tertiary
learners
(Alkema, 2012)
October 2012

24. Changes to our project funding model
• Ako Aotearoa funds change projects
• i.e. we fund evidence-based change projects with a
high potential to benefit learners.
• Co-funding model - 50% contribution (or more) by the
organisation
• i.e. we require organisational commitment to change
practice
• We will measure the impact of projects
• We expect to create PD resources from high impact
projects
October 2012

25. Logic sequence for sustainable improvement
SHARE Identify
EXTERNALLY need / Investigate options
opportunity
Change
becomes Student success Create
BAU on course(s) intervention
Better Student
student response: do Practice change
outcomes they like it?
October 2012

26. So do Ako Aotearoa projects benefit learners?
October 2012

27. Impact evaluation of our own projects in providing
learner benefit:
• 63 successfully completed small projects
• Interest generated
• Impact on practice
• Evidence of learner benefit associated with the work
• Impact on project team
• Outcomes hierarchy.... direction of travel
October 2012

28. Pragmatic choices
Evidence that identifies the most effective
interventions when providers need to make
choices
October 2012

29. Exemplar project: Dedicated education unit
• Project: Piloted 2 DEUs at
Middlemore hospital using action
research
• http://akoaotearoa.ac.nz/nursing-deu
Vision was to change
the way nurses
were supported
as they learnt
October 2012

30. Exemplar project: Dedicated education unit
• What’s changed? They are now business as usual
• 9 DEUs established, 2 more in 2013 (covers
between 80-90% of nursing students at MIT)
• exploring possibility of multi-disciplinary unit
• Key features
• Robust design, strong organisational buy-
in, focused on change for learners, broad
influence
October 2012

31. The key output from the dedicated education
unit project is not the research report
It’s the practice guide
October 2012

32. Exemplar project: ITO assessment systems
• Project: examined assessment systems for industry
training
• http://akoaotearoa.ac.nz/projects/ito-workplace-
assessment-structures
Vision was to create
system level
change in
assessment
practice
October 2012

33. Exemplar projects: ITO assessment systems
• What’s changed?
• Principles have contributed to organisational wide
practice in 5 ITOs (impacts ~23,000 learners)
• Workshops and new project with BCITO illustrate
how change can be achieved
• Key features
• Issue of critical importance, research based
principles, structure created to embed within
organisational practice
October 2012

34. Professional development workshops and
resources
October 2012

35. Organisation
Explicit aspirations for their learners
Mutual
Mutually
preparedness to Evidence
congruent
innovate and about what
expectations
change practice works
Practitioners
Individual (and team) aspirations for their
own practice and for their learners
October 2012

36. The last big question:
Are we supporting change
to improve what we do now
or
are we fundamentally changing
the way we do things?
October 2012

37. The changing nature of 21st century tertiary education
October 2012

38. So what are the characteristics of a high
performing professional tertiary teacher?
These will stay the same:
• Empathy
• Reflection-in-action
• Critical self reflection-on-action
• Emancipation
October 2012

39. Plus what for the future?
October 2012

40. What are the critical shifts?
• A focus beyond the boundaries of courses and their
programmes – tighter integration between programmes of
study and work
• Changing expectations about Return on Investment
• Changing expectations about inclusivity
• New technologies allow greater learner autonomy and much
greater expectations around meaningful student centricity
• Open source materials will change the dynamics of delivery
October 2012

41. Where is this thinking coming from?
Wide range of different conversations about 21st
education and educators, but more recently:
• DEANZ Conference
• Metro ITPs Graduate Diploma in Tertiary Education
• HERDSA Graduate Attributes
• Te Ara Whakamana
• Tertiary Teaching Excellence Awards portfolios
• Ako Aotearoa’s work on its own business model
October 2012

42. What are the norms of 21st century education?
• Focus on successful outcomes for all learners in their
classes
• Highly culturally competent
• Capability focus rather than content / competence
focus:
“ I teach less content so that my students learn more”
Zoe Jordens
October 2012

43. What are the changes in skill set for the faculty?
More specialist (e.g. learning design / assessment /
curriculum development / open source retrieval)
The 21st Century tertiary educator will spend much less
time developing resources and much more time facilitating
learning and providing facilitative feedback to learners
The 21st Century tertiary educator’s practice will be
increasingly framed by how their students use on-line
technologies
The 21st century educator’s practice will be informed by the
latest understandings in cognitive psychology
October 2012

44. What are the changes in skill set for the tertiary
education organisation?
It will host, support and develop 21st century educators
who are dual professionals
It will own (and, in turn, expect employee ownership of)
successful learner outcomes
It will seek to disestablish the resource management –
educational performance divide
October 2012

45. What are the changes in skill set for the tertiary
education organisation?
It will become a learning organisation
October 2012

46. Organisation
Explicit aspirations for their learners
Mutual
Mutually
preparedness to Evidence
congruent
innovate and about what
expectations
change practice works
Practitioners
Individual (and team) aspirations for their
own practice and for their learners
October 2012

47. Thank you
www.akoaotearoa.ac.nz
October 2012