1. LINQ 2013
Innovations and Quality: The Future of Digital Resources
Rome, May 16-17, 2013
EVALUATING THE QUALITY OF
DIGITAL RESOURCES
Dr. Tony Bates,
Tony Bates Associates Ltd,
Vancouver, BC, Canada
1

2. Overview
1. Open education: a brief history
2. Criteria: the ACTIONS model
3. Applying the model to:
• MOOCs
• OERs
• Open Universities
• Online credit programs
4. Making comparisons
5. Conclusions

3. Landmarks in the history of open
education
• 1870: compulsory state-funded
public education up to 13 years
old (England)
• 1944: GI Bill: access to higher
education: 20 million (USA)

4. Landmarks in the history of open
education
• 1963: Robbins Report: ‘all
qualified by ability and
attainment’ (UK)
• 1969: The Open University
(UK): ‘open to people, places,
methods, ideas’: 1.7 million
University of East Anglia
Walton Hall

5. Landmarks in the history of open
education
• 1998: open educational
resources: ‘resources that
reside in the public domain’
• 2001:Wikipedia
• 2002: MIT OpenCourseWare

6. Landmarks in the history of open
education
• 2007: iTunes U (1 billion
downloads, 2013; UK OU: 50
million)
• 2008: MOOCs (Coursera: 3.4
million)
Dave Cormier’s YouTube description of cMOOCs

7. How do we evaluate open digital
resources?
What do we mean by ‘open’? Free?
Accessible? Flexible?
Is being ‘open’ enough? Don’t
forget the ‘education’ part
‘Open’ is an emotionally charged
word
Need for criteria that are VALID
and RELIABLE
7

8. How do we evaluate open digital
resources?
A C T I O N S model for evaluating
educational media (Bates, 1995)
A ccess
C ost
T eaching
I nteraction
O rganization
N ovelty
S ecurity
8

9. Models of innovative open education
Four models:
• MOOCs: Coursera (Duke University)
• Open educational resources: MERLOT
• Open university: UK
• Online credit program: UBC, Canada
‘quality assessment’ using the A C T I O N S
model
9

10. Model 1: MOOC
Duke University (Coursera):
bioelectricity
• open access (no formal qualifications)
• fully online (lecture capture, CMAs)
• free
• top private research university in the
USA
• statement of accomplishment
10

11. Model 2: OER
MERLOT: collection of peer-reviewed
OERs
• Roman architecture, Yale
• open access
• fully online (multi-media)
• free
• ‘open’ use
11

12. Model 3: Open university
Open University, UK: B.Sc. (Hons.) Science
• Open access (no formal qualifications)
• Multimedia materials, tutors and online
forums
• Cost: 17,000 euros = 5,700 per year;
(international students pay more)
• Top 20% in research and teaching in UK
12

13. Model 4: Online credit program
UBC online credit program: Master in
Educational Technology (fully online)
• Single courses/certificate: open access
• Masters: graduate entry qualifications
(bachelor’s degree)
• From anywhere in world
• Cost: 11,000 euros =5,500 per year or 1,100
per course
• Tier 1 research university
13

14. Applying the criteria:
Access (developed world)
14
Technology
(max 10 = good)
Admission
(max 10 = open)
MOOC 8 10
OER 9 10
Open university 9 10
Credit online 9 7

15. Applying the criteria:
Cost
15
Institutional
(max 10 =low)
Student
(max 10 = low)
MOOC 4 10
OER 7 10
Open university 6 4
Credit online 6 5

16. Applying the criteria:
Teaching
16
Content
(max 10 = good)
Pedagogy/completi
on rates
(max 10 = good)
MOOC 10 2
OER 7 5
Open university 7 6
Credit online 9 8

17. Applying the criteria:
Interaction
17
With instructor
(max 10 = good)
Social (inc. tutor)
(max 10 = good)
MOOC 1 3
OER 1 5
Open university 3 7
Credit online 8 7

18. Applying the criteria:
Organization as a barrier to open
18
(max 10 = good)
MOOC 8
OER 5
Open university 8
Credit online 5

19. Applying the criteria:
Novelty/hype/PR
19
(max 10 = high)
MOOC 10
OER 8
Open university 6
Credit online 4

20. Applying the criteria:
Security/quality assessment
20
(max 10 = high)
MOOC 1
OER 3
Open university 8
Credit online 9

21. Making comparisons
21
MOOC OER Open U Credit
Access 18 19 19 16
Cost 14 17 10 11
Teaching 12 12 13 17
Interaction 4 6 10 15
Organiz. 8 5 8 5
Novelty 10 8 6 4
Security/as
sessment
1 3 8 9
Total 67 70 74 77

22. Conclusions
• ‘open’ is a multi-faceted concept
• access alone is not enough; quality
matters
• need a clear set of criteria;
alternative models possible
• more innovation needed; and
possible
• But: well-funded public education
system best guarantee of open access
22

23. Questions
Evaluating the quality of open digital resources
1. Are these the right criteria? If not, what other criteria should
be considered?
2. Do the numerical values make sense?
3. Is there a better/more ‘scientific’ way to evaluate quality?
23