The document discusses various concepts related to quality, accreditation, and regulation for virtual learning including critical success factors, benchmarking, and quality standards. It proposes organizing these concepts into a pyramid structure with critical success factors at the top level set by leadership and more detailed guidelines and criteria at lower levels. It also suggests countries write in with information on how virtual schools are accredited, inspected, and the criteria used in their regulatory processes. Finally, it outlines the Pick&Mix benchmarking system used to evaluate virtual learning initiatives according to a set of criteria scored on a scale and organized using a traffic light system.

1. Critical Success Factors,
Benchmarking and
Quality in Virtual Learning
(and Accreditation and Inspection!)
Paul Bacsich
European Virtual Schools Colloquium, Sheffield, 22 May 2012 1

2. Too many concepts
Critical
Benchmarking Success
Factors
Accreditation
Standards?
/approval
Quality
/kitemarking
E-learning (and OER) is only a small part of the quality process –
how can agencies and assessors handle five variants of the concept
across many separate methodologies?
European Virtual Schools Colloquium, Sheffield, 22 May 2012 2

3. Proposal to ENQA - the pyramid
Critical Success Leadership level
Factors -------------
Benchmarking ---- Senior managers
Quality --------------
Criteria are placed
at different layers
in the pyramid
Detailed pedagogic depending on their “level”
guidelines ----------
European Virtual Schools Colloquium, Sheffield, 22 May 2012 3

4. The regulatory pyramid above
depending on their “level”
in the pyramid
at different layers
Criteria are placed
European Virtual Schools Colloquium, Sheffield, 22 May 2012 4

5. Re.ViCa 2007-09
(Review of Virtual Campuses)
 Project supported by the European Union under the Lifelong
Learning Programme - Erasmus/Virtual Campus
– With International Advisory Committee
 Database of countries, agencies and Programmes (500)
 Nine case studies
 Set of 17 Critical Success Factors developed after wide
international consultation – embedded in Pick&Mix scheme
 Organised post-secondary e-learning initiatives are found
across the “G-100” (all except the Least Developed Countries)
European Virtual Schools Colloquium, Sheffield, 22 May 2012 5

6. VISCED
Uses the Re.ViCa wiki
See Susan’s and Barry’s summaries
And our Newsletter and wiki !
Need to modify the Re.ViCa Critical
Success Factors for virtual schools
AND thus need to understand the
accreditation, inspection, quality and
regulatory context round virtual schools
European Virtual Schools Colloquium, Sheffield, 22 May 2012 6

7. A write-in exercise:
For each country
1. Who Accredits virtual schools?
2. Who Inspects virtual schools – and how
often?
3. What criteria do they use?
4. Are there specific criteria for virtual
schools? (like US, Australia, New
Zealand, England in the past for schools –
and UK, Sweden in the past for unis)
European Virtual Schools Colloquium, Sheffield, 22 May 2012 7

8. Criteria: Pick&Mix overview
 Focussed on e-learning, not general pedagogy
 Draws on several sources and methodologies –
UK and internationally (including US) and from
college sector – not yet schools
 Not linked to any particular style of e-learning
(e.g. distance or on-campus or blended)
 Suitable for desk research as well as “in-depth”
studies
 Suitable for single- and multi-institution studies
European Virtual Schools Colloquium, Sheffield, 22 May 2012 8

9. Pick&Mix history
 Initial version developed in early 2005
 Since then, refined by literature search, discussion,
feedback, presentations, workshops, concordance
studies and four phases of use – fifth and sixth
phases now – over 30 institutions benchmarked
 Formed the basis of the wording of the Critical
Success Factors scheme for the EU Re.ViCa
project – now reincorporated in Pick&Mix
 And the basis for the draft Critical Success Factors
for VISCED – but YOU can help update these
European Virtual Schools Colloquium, Sheffield, 22 May 2012 9

10. Pick&Mix
Criteria and metrics
European Virtual Schools Colloquium, Sheffield, 22 May 2012 10

11. Criteria
 Criteria are “statements of practice” which are
scored into a number of performance levels from
bad/nil to excellent
 It is wisest if these statements are in the public
domain – to allow analysis & refinement
 The number of criteria is crucial
 Pick&Mix currently has a core of 20 – based on
analysis from the literature (ABC, BS etc) and
experience in many senior mgt scoring meetings
European Virtual Schools Colloquium, Sheffield, 22 May 2012 11

12. Pick&Mix Scoring
Use a 6-point scale (1-6)
– 5 (cf Likert, MIT90s levels) plus 1 more for
“excellence”
Contextualised by “scoring commentary”
There are always issues of judging
progress especially “best practice”
The 6 levels are mapped to 4 colours in a
“traffic lights” system
– red, amber, olive, green
European Virtual Schools Colloquium, Sheffield, 22 May 2012 12

13. Pick&Mix System: summary
Has taken account of “best of breed”
schemes
Output and student-oriented aspects
Methodology-agnostic but uses underlying
approaches where useful (e.g. Chickering
& Gamson, Quality on the Line, MIT90s)
Requires no long training course to
understand
European Virtual Schools Colloquium, Sheffield, 22 May 2012 13

14. Pick&Mix
Two sample criteria
European Virtual Schools Colloquium, Sheffield, 22 May 2012 14

15. P10 “Training”
1. No systematic training for e-learning
2. Some systematic training, e.g. in some projects
and departments
3. School-wide training programme but little
monitoring of attendance or encouragement to go
4. School-wide training programme, monitored and
incentivised
5. All staff trained in VLE use, training appropriate to
job type – and retrained when needed
6. Staff increasingly keep themselves up to date in a
“just in time, just for me” fashion except in
situations of discontinuous change
European Virtual Schools Colloquium, Sheffield, 22 May 2012 15

16. P05 “Accessibility”
1. VLE and e-learning material are not accessible
2. VLE and much e-learning material conform to
minimum standards of accessibility
3. VLE and almost all e-learning material conform to
minimum standards of accessibility
4. VLE and all e-learning material conform to at least
minimum standards of accessibility, much to higher
standards
5. VLE and e-learning material are accessible, and key
components validated by external agencies
6. Strong evidence of conformance with letter & spirit
of accessibility in all countries where students study
European Virtual Schools Colloquium, Sheffield, 22 May 2012 16