Draf

1. Success Criteria for Virtual Schools
- Pick&Mix
Professor Paul Bacsich
Sero Consulting Ltd and Matic Media Ltd
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 1

2. Topics
1. Introduction, disclaimers and
acknowledgements
2. History of Pick&Mix in universities
3. Why is he telling us this?
4. Pick&Mix
5. More recent history: application to
colleges and high schools
6. Reflections on this process
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 2

3. 1. Introduction, disclaimers
and acknowledgements
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 3

4. Who is this talk for?
Want to know about comparing your virtual
school with other ones?
Want to know about the “tradecraft” of
benchmarking and quality reviews?
Want to adapt or update a benchmarking
system?
Want to learn some of the underlying
principles of such schemes?
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 4

5. Pick&Mix is an “Open Educational Methodology”
for benchmarking online learning, developed
by Paul Bacsich and available for all to
develop and modify, but the names
“Pick&Mix” and “ELDDA” are reserved for
the “main sequence” of development
Thanks to many, including UK HE Academy,
JISC, EU Lifelong Learning Programme
(Re.ViCa and VISCED), Manchester
Business School, University of Leicester and
Sero Consulting Ltd for support
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 5

6. 2. History of Pick&Mix in
universities
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 6

7. Benchmarking online learning
 At national level, started in 2004-05 in UK and
New Zealand
– Soon spread to Australia
– Not closely linked initially to quality agenda
 At European level, developments include
E-xcellence and UNIQUe
– Some earlier work from OBHE, ESMU etc
– Later, developments in other projects
– Increasingly, links made to quality agenda and to
critical success factors
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 7

8. Pick&Mix history
 Initial version developed in early 2005 in response
to a request from Manchester Business School for
an international (largely US) competitor study
 Since then, refined by literature search, discussion,
feedback, presentations, workshops, concordance
studies and four phases of use – sixth and seventh
phases now
 Forms the basis of the wording of the Critical
Success Factors scheme for the EU Re.ViCa
project and now being used to develop a similar
scheme for Virtual Schools in VISCED
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 8

9. Benchmarking e-learning (UK)
 Foreseen in HEFCE 2005 e-learning strategy for
universities in England (later for Wales via
HEFCW)
 Higher Education Academy (HEA) oversaw it
 Four phases – 82 institutions – 5 methodologies
 Two consultant teams – one run by myself
 My team benchmarked over 40 institutions using
4 methodologies
 Including 24 using Pick&Mix – now well over 30
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 9

10. What HEFCE wanted
“Possibly more important is for us [HEFCE] to
help individual institutions
understand their own positions on e-learning,
to set their aspirations and goals for
embedding e-learning – and then to
benchmark themselves and their progress
against institutions with similar goals,
and across the sector”
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 10

11. Methodologies in UK HE
 Of the five methodologies used in the UK from 2005 on, only
one survives that is actively under development, refined
annually, public domain and available for supervised or self-
applied use in institutions, via funded projects and
commercially:
Pick&Mix
 In other countries’ HE systems:
– eMM is in a similar situation in New Zealand
– Quality Matters is widespread in US
– ACODE is used somewhat in Australia
– On the continent of Europe there are a few methodologies but
fostered purely by EU-funded projects
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 11

12. Pick&Mix overview
 Focussed on online learning, not general pedagogy
 Draws on several sources and methodologies – UK and
internationally (including US – especially Quality on the
Line) and from college sector
 Not linked to any particular style of online learning (e.g.
distance or on-campus or blended)
 Oriented to institutions with notable activity in online
learning
 Suitable for desk research as well as “in-depth” studies
 Suitable for single- and multi-institution studies
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 12

13. 3. But why is he telling us
this?
BECAUSE IT CAN BE USED FOR
COLLEGES AND SCHOOLS ALSO
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 13

14. 4. Pick&Mix
Criteria and metrics
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 14

15. Criteria
 Criteria are “statements of practice” which are
scored into a number of performance levels from
bad/nil to excellent
 It is crucial that these statements are in the public
domain – to allow analysis & refinement
 The number of criteria is also crucial:  24
 Pick&Mix originally had a core of 20 – based on
analysis from the literature (ABC, BS etc) and
experience in many senior mgt scoring meetings
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 15

16. Pick&Mix: 20 core criteria
 Removed any not specific to online learning
– Including those in general quality/accreditation schemes
 Careful about any which are not provably success factors
 Left out of the core were some criteria where there was
not yet UK consensus
 Institutions will wish to add some to monitor their KPIs and
objectives. Recommended no more than 6.
– Pick&Mix now has over 70 supplementary criteria to choose from
– more can be constructed or taken from other schemes
 These 20 have stood the test of four phases of
benchmarking with only minor changes of wording
– originally 18 - two were split to make 20
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 16

17. Pick&Mix Scoring
Use a 6-point scale (1-6)
– 5 (cf Likert, MIT90s levels) plus 1 more for
“excellence”
Contextualised by “scoring commentary”
There are always issues of judging
progress especially “best practice”
The 6 levels are mapped to 4 colours in a
“traffic lights” system
– red, amber, olive, green
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 17

18. Pick&Mix System: summary
Has taken account of “best of breed”
schemes
Output and student-oriented aspects
Methodology-agnostic but uses underlying
approaches where useful (e.g. Chickering
& Gamson, Quality on the Line, MIT90s)
Requires no long training course to
understand
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 18

19. P01 “Adoption” (Rogers)
1. Innovators only
2. Early adopters taking it up
3. Early adopters adopted; early majority
taking it up
4. Early majority adopted; late majority taking
it up
5. All taken up except laggards, who are now
taking it up (or retiring or leaving)
6. First wave embedded, second wave under
way (e.g. BYOD-learning after e-learning)
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 19

20. P10 “Training”
1. No systematic training for e-learning
2. Some systematic training, e.g. in some projects
and departments
3. Institution-wide training programme but little
monitoring of attendance or encouragement to go
4. Institution-wide training programme, monitored
and incentivised
5. All staff trained in VLE use, training appropriate to
job type – and retrained when needed
6. Staff increasingly keep themselves up to date in a
“just in time, just for me” fashion except in
situations of discontinuous change
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 20

21. Supplementary criteria - examples
 IT reliability
 Market research
 Competitor research
 IPR
 Help Desk
 Management of student expectations
 Student satisfaction
 Web 2.0 pedagogy
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 21

22. 5. More recent history
Adaptation to colleges and high
schools (both virtual and blended)
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 22

23. Virtual post-secondary institutions
The EU Re.ViCa project 2007-09 (Review
of Virtual Campuses) did a great deal of
work to refine and “Europeanise” the
Pick&Mix criteria, focussing on
– Critical Success Factors for virtual campuses
A slightly revised scheme was produced,
oriented to institutions where distance
learning was the prevalent approach
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 23

24. Distance Learning
Benchmarking Club 2009-11
In summary: Universities, encouraged by a
UK-funded project, formed a group to
benchmark themselves:
– University of Leicester (UK)
– Royal Swedish Institute of Technology (KTH)
– Lund University (Sweden)
– University College Gotland (Sweden)
– Thompson Rivers University (Canada)
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 24

25. VISCED 2011-12
A further iteration is in final stages to
produce a scheme of Critical Success
Factors for virtual schools
– With initial focus to European virtual schools
– Since these are smaller and few have ever
failed, this is a hard task
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 25

26. Draft key success factors for
European virtual schools
1. Usability (for all participants)
2. Strategy (for online learning)
3. Recruitment and training (fused?)
4. Evaluation (of programmes) – often informal but effective
5. Reliability (of system)
6. Leadership (with knowhow, down and up – fused?)
7. Organisation (formal and informal)
8. Learning outcomes (often individualised)d
9. Use of resources (not key in universities)
10. Market research?
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 26

27. 6. Reflections on this process
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 27

28. Too many concepts
(Critical)
Benchmarking Success
Factors
Accreditation
Standards
/approval
Quality
/kitemarking
Online learning is only a small part of the quality process –
how can agencies and assessors handle five variants of the concept
across many separate methodologies?
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 28

29. My view for ENQA – the pyramid
 Critical Success Leadership level
Factors, a selection of:
 Benchmarking, which Senior managers
split into sub-criteria:
 Quality, split into Criteria are placed
at different layers
in the pyramid
 Detailed pedagogic depending on their “level”
guidelines ----------
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 29

30. Adaptability
The transition of a basically university-
originated methodology to schools is
feasible, particularly for larger high schools
(virtual or blended)
The sectors are not as different as territorial
experts like to believe
But the methodology has to be theoretically
sound and well-researched
There are other extensions - eg for OER !
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 30

31. Merging US and EU practice?
 But could we cross the transatlantic divide?
 There were some discussions between EU
institutions and QM over a common HE system
(UNESCO would love that)
 For schooIs, we need to look at iNACOL
guidelines – meeting after the Symposium
 Since Pick&Mix was heavily influenced by US
experience (Ehrmann; Quality on the Line) there
should be hope – at first sight it looks promising
 A hierarchical (pyramid) approach will be key
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 31

32. A view after 7 years of Pick&Mix
 Methodologies do not survive without regular
updating by a design authority
– this is difficult in a leaderless group context
 Forking of methodologies needs dealt with by
folding updates back to the core system
– otherwise survival is affected
 Complex methodologies do not survive well
 A public criterion system allows confidence,
transparency, and grounding in institutions
 Sectoral boundaries can be overcome
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 32

33. References
The Pick&Mix system is detailed at
http://www.matic-media.co.uk/benchmarking/PnM-2pt6-beta3-full.xlsx
A key paper on the international aspects is:
“Benchmarking E-Learning in UK Universities: Lessons from and for the International
Context”, in Proceedings of the ICDE conference M-2009, Open Praxis –
http://www.openpraxis.com/files/Bacsich%20et%20al..pdf
A specific chapter on the UK HE benchmarking programme methodologies is:
“Benchmarking e-learning in UK universities – the methodologies”, in
Mayes, J.T., Morrison, D., Bullen, P., Mellar, H., and Oliver, M.(Eds.)
Transformation in Higher Education through Technology-Enhanced Learning,
York: Higher Education Academy, 2009 –
http://www.heacademy.ac.uk/assets/documents/learningandtech/Transforming-07.pdf
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 33