[Qa of moo cs openup-ed-week-2019] - Jon Rosewell

1. Quality assurance of MOOCs:
The OpenupEd quality label
Jon Rosewell, The Open University (UK)
OpenupEd week
16 April 2019

2. Quality in e-learning

3. What do we mean by ‘quality’ in HE?
• Compliance & consumer protection
– Accreditation
– Guarantee of uniform standards
• Reputation
– Recruit good students, produce good graduates
• Quality enhancement / Process improvement
– Institutional mission
– Stakeholder engagement
– Measures of added value (‘learning gain’)

4. Approaches to QA in e-learning
• Compliance or enhancement?
• Process or product?
• Input elements?
• Pedagogical models?
• Outcome measures?
• Self-assessment or external review?
• Scorecard? Benchmarking against others?
Holistic: emphasis on process & context as well as product

5. A generic framework for QA in HE
Ebba Ossiannilsson, Keith Williams, Anthony F. Camilleri, and Mark Brown (2015)
Quality models in online and open education around the globe: State of the art and
recommendations, ICDE Report http://www.icde.org/quality

6. European Standards & Guidelines (ESG)
and e-learning
1.1 Policy for QA
1.2 Design and approval of programme
1.3 Student-centred learning, teaching & assessment
1.4 Student admission, progression, recognition & certification
1.5 Teaching staff
1.6 Learning resources and student support
1.6 Information management
1.8 Public information
1.9 Ongoing monitoring and periodic review
1.10 Cyclical external quality assurance

7. ENQA: Considerations for QA of e-learning
• Published 2018
• Supplement to ‘European Standards and Guidelines’ 2015
• Additional guidance and indicators
Huertas et al (2018) ENQA Occasional Papers, No. 26
https://enqa.eu/index.php/publications/papers-reports/occasional-papers/

8. Poll – do you use a QA process/framework?
• No
• Yes, internally defined
• Yes, defined by QA / Govt agency

9. Poll – why would you do more QA?
❑I want to improve my teaching
❑My boss tells me to
❑It is my job!
❑The QA agency / Ministry make me
❑I want a promotion
❑Other reasons
Check as many as apply

10. E-xcellence

11. E-xcellence label
http://e-xcellencelabel.eadtu.eu/
Manual

12. Organisation of resources
Strategic Management a high level view of how the institution plans its e-learning
Curriculum Design how e-learning is used across a whole programme of study
Course Design how e-learning is used in the design of individual courses
Course Delivery the technical and practical aspects of e-learning delivery
Staff Support the support and training provided to staff
Student Support the support, information and guidance provided to students

13. Sample benchmark
Course design
10. …
11. Learning outcomes determine the use of methods and
course contents. In a blended-learning context there is an
explicit rationale for the use of each element in the blend.
12. …

14. Sample indicators
Indicators
• Fitness for purpose drives decisions on the selection of teaching and
learning activities. The blending is such that different methods and
media are well chosen within and between courses, both in distribution
over time and extent of use.
At excellence level
• There is extensive institutional experience of delivery using blended
learning and this experience is widely shared through the organisation.
• Well informed decisions on the use of teaching and learning activities
are made routinely and reflect institutional policies regarding the
development of learner knowledge and skills.

15. Benchmarking as quality enhancement tool
• Statement of best practice
– Suggested indicators
• Collecting evidence
– Can be specific to each university
• Identification of weaknesses & strengths
• …leading to roadmap of actions for improvement

16. Poll – who should collect evidence?
• Course author
• Administrator
• Students
• External reviewer
• Team of stakeholders

17. Different ways to use E-xcellence
• Informal self-assessment using QuickScan
– Identify ‘hot’ and ‘cold’ spots
• Full internal self-assessment
– Stakeholders collect evidence
– Prepare roadmap of improvement actions
• Integrate with institutional process
– Embed selected benchmarks in internal process
• EADTU E-xcellence Associates Label
– Self-assessment, roadmap, external review
→ recognition by EADTU
NB: Resources such as
manual and benchmarks
are freely available!

18. MOOC quality

19. Why worry about MOOC quality?
Students – know what they are committing to
Employers – recognition of content and skills
Authors – personal reputation, 'glow' of success
Universities / providers – brand reputation
Funders – philanthropists, government, investors
Quality agencies – on behalf of all above

20. Are MOOCs different from e-learning?
• MOOC vs Higher Education e-learning
– Short, free, no entry requirements
– Not accredited
– Reputational risk
• MOOC participants
– Motivations differ from degree students
– Completion may not be not their goal
But a MOOC is a Course so maybe it should be judged like any other
HE course?

21. OpenupEd Quality Label
• Derived from E-xcellence
– Lightweight process
• Self-assessment
• Formal label
– External review
www.openuped.eu/quality-label

22. OpenupEd MOOC features
• Openness to learners
• Digital openness
• Learner-centred approach
• Independent learning
• Media-supported interaction
• Recognition options
• Quality focus
• Spectrum of diversity

23. OpenupEd MOOC benchmarks
• Derived from E-xcellence benchmarks
• For the institution:
– To be checked every 3-5 years
– 21 benchmark statements, in six groups:
Strategic management, Curriculum design, Course design, Course delivery, Staff support,
Student support
• For the course:
– To be checked for each MOOC
– 11 benchmark statements

24. Benchmarks – course level
22. A clear statement of learning outcomes for both knowledge and skills is provided.
23. There is reasoned coherence between learning outcomes, course content,
teaching and learning strategy (including use of media), and assessment
methods.
24. Course activities aid participants to construct their own learning and to
communicate it to others.
25. The course content is relevant, accurate, and current.
26. Staff who write and deliver the course have the skills and experience to do so
successfully.
27. Course components have an open licence and are correctly attributed. Reuse of
material is supported by the appropriate choice of formats and standards.
28. The course conforms to guidelines for layout, presentation and accessibility.

25. Benchmarks – course level
29. The course contains sufficient interactivity (student-to-content or student-to-
student) to encourage active engagement. The course provides learners with
regular feedback through self-assessment activities, tests or peer feedback.
30. Learning outcomes are assessed using a balance of formative and summative
assessment appropriate to the level of certification.
31. Assessment is explicit, fair, valid and reliable. Measures appropriate to the
level of certification are in place to counter impersonation and plagiarism.
32. Course materials are reviewed, updated and improved using feedback from
stakeholders.

26. Additional notes – example
31. Assessment is explicit, fair, valid and reliable. Measures appropriate to the
level of certification are in place to counter impersonation and plagiarism.
See comments to OpenupEd benchmark 29 above.
The advent of digital badges (for example Mozilla open badges) provides a method of
rewarding achievement that may be appropriate for MOOCs. The award of digital badges
can be linked to automated or peer assessment. Digital badges have an infrastructure that
verifies the identity of the holder and provides a link back to the issuer and the criteria and
evidence for which it was awarded. Badges thus may provide a validated award that can be
kept distinct from the HEI’s normal qualifications.
See also:
E-xcellence benchmark #17
Chapter 3 Course design
§ 2.3.1 Transferable skills
§ 2.4 Assessment procedures
§ 3.4 Assessment
§ 4.2.5 Online assessment

27. Templates – evidence

28. Quick scan

29. NA: Not achieved
PA: Partially achieved
LA: Largely achieved
FA: Fully achieved
Quick scan

30. OL: Openness to learners
DO: Digital openness
LC: Learner-centred approach
IL: Independent learning
MI: Media-supported interaction
RO: Recognition options
QF: Quality focus
SD: Spectrum of diversity
Quick scan

31. Whiteboard – your MOOC experience
Not achieved Fully achieved
22. Clear learning outcomes
23. Aligned LOs, content, assessment
24. Activities construct learning
25. Relevant, accurate, current
29. Interactivity, active learning, self-ass.
30. Formative & summative assessment

32. Whiteboard – MOOC features
• Openness to learners
• Digital openness
• Learner-centred approach
• Independent learning
• Media-supported interaction
• Recognition options
• Quality focus
• Spectrum of diversity

33. New checklists (OpenupEd, SCORE2020)
Checklist 1: Is it a MOOC or not?
– 14 items
Checklist 2: Quality of the design of MOOC
– 26 items
Checklist 3: Accessibility
– 6 items
Checklist 4: Technical platform and support for staff and
participants
– 12 items

34. Checklist 2: Quality of design
Dimension Criteria Level
Target group MOOCs are accessible to all people and as such various target
groups are identified
For each target group the needs, challenges and prior
knowledge are described
The description of each target group is supported by references
different studies
Overall goal The overall objective of the course is described in a few
sentences
Learning ob. The course describes a limited number of learning objectives
There is a reasoned coherence between learning outcomes,
course content, teaching and learning strategy (including use of
Levels: Not achieved, Partially achieved, Largely achieved, Fully achieved

35. Checklist 2: Quality of design
Dimension Criteria Level
Learning
activities
Activities aid participants to construct their own learning and to
communicate it to others
The ‘pathways’ (activities, tasks and routes) are designed in
such a way that they can be performed at different levels of
difficulty or complexity, to account for the broad spectrum of
participants’ knowledge and skills that are expected
Various activities are proposed with different formats. For
example: quizzes, peer evaluation, video conferences, activities
in forums or external social networks).
The course contains sufficient interactivity (learner-to-content,
learner-to-learner, or learner-to-teacher) to encourage active
engagement.
Levels: Not achieved, Partially achieved, Largely achieved, Fully achieved

36. Checklist 2: Quality of design
Dimension Criteria Level
Feedback
mechanism
Feedback by an academic tutor is limited and scalable
(characteristic of a MOOC).
The course provides learners with regular feedback through self-
assessment activities, tests or peer feedback.
The frequency of monitoring has been planned (forum, group,
post)
A weekly announcement or mass mailing with orientations for the
following week is planned.
In each weekly session, the pedagogical team makes a
synthesis of artefacts from the previous week’s session.
Some live events (Hangout, Tweetchat) are scheduled
Levels: Not achieved, Partially achieved, Largely achieved, Fully achieved

37. QRF from MOOC-Quality.eu
Phases / perspectives / roles

41. QRF phases, processes, criteria
• Analysis 7 processes 25 criteria
• Design 11 processes 54 criteria
• Implementation 6 processes 16 criteria
• Realization 3 processes 35 criteria
• Evaluation 4 processes 11 criteria

42. Poll – your preference
• Few generic benchmarks supplemented by evidence
• Many detailed criteria

43. Learning design

44. What is learning design?
• A way of documenting the design of a course
• A way of thinking about & discussing design
– before it is too late!
• A way to think about appropriate use of technologies
• A framework for evaluating courses
• A framework for evaluating designs of courses

45. The 7Cs of Learning Design (Gráinne Conole)
Many activities to help teams
consider each of these stages
– Course tweet (elevator pitch)
– Personas
– Resource audit
– Interactive > Constructive >
Active > Passive
– Constructive alignment (of
learning outcomes, activities,
assessment)
http://www2.le.ac.uk/projects/oer/oers/beyond-distance-research-alliance/7Cs-toolkit

46. UCL Learning Designer (Diana Laurillard)
• Acquisition
• Inquiry
• Practice
• Production
• Discussion
• Collaboration
https://www.ucl.ac.uk/learning-designer/
Laurillard, D. (2002). Rethinking university teaching: A conversational
framework for the effective use of learning technologies. Routledge.

47. UCL Learning Designer (Diana Laurillard)
https://www.ucl.ac.uk/learning-designer/

48. 8 Learning Events (Leclercq & Poumay)
• Imitation / modelling
• Reception / transmission
• Exercising / feedback
• Exploration / resources
• Experiment / scenarios
• Creation / facilitate
• Reflection / advise
• Debate / prompting
www.labset.net/media/prod/8LEM.pdf

49. First Principles of Instruction (Merrill)
– with a few more! (Margaryan)
1. Problem-centred
2. Activation
3. Demonstration
4. Application
5. Integration
6. Collective knowledge
7. Collaboration
8. Differentiation
9. Authentic resources
10. Feedback
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs).
Computers & Education, 80, 77-83.

50. OU Learning design – module map

51. OU Learning design – activity classes
Assimilative read, watch, listen, think about, observe
Finding & handling info list, analyse, collate, find, select, manipulate
Communicative communicate, debate, discuss, collaborate, present
Productive build, write, make, design, construct, produce, draw
Experiential practice, apply, experience, investigate, perform
Interactive/adaptive explore, experiment, improve, model, simulate
Assessment write, demonstrate, critique, peer review, self-assess, receive feedback

52. OU Learning design – activity planner

53. OU Learning design – in practice
Toetenel, Lisette and Rienties, Bart (2016). Learning Design – creative design to visualise learning activities. Open Learning, 31(3) pp. 233–244.

54. OU Learning design – in practice
Toetenel, Lisette and Rienties, Bart (2016). Learning Design – creative design to visualise learning activities. Open Learning, 31(3) pp. 233–244.

55. In summary…
• A quality framework should underpin e-learning provision
– to help create a quality culture
– that is more likely to produce quality e-learning
– and quality enhancement
• And that also applies to MOOCs
• There is no simple recipe, but…
– Work in a course team
– Think about learning design at an early stage
– Don’t let QA procedures get in the way of the day job!

56. THANK YOU
Jon.Rosewell@open.ac.uk

57. Extra slides!

58. What students want – and what they need
“Student satisfaction is “unrelated” to learning behaviour and
academic performance, a study has found.
[…] while students dislike collaborative learning, they are
more likely to pass if they take part in it”
(Times Higher Education, Feb 12th 2018)
From an analysis of 100,000 students
on 151 modules
More at Bart Rientes, OU Inaugural Lecture

59. How does student satisfaction relate to module performance?Satisfaction
Students who successfully completed module
Slide from Bart Rienties Inaugural lecture

60. MOOC case study: OU + FutureLearn
A representative Open University MOOC … published on FutureLearn
• Evidence for OpenupEd features and benchmarks
• Quality emerges from joint efforts of OU (university) &
FutureLearn (platform provider)
• Holistic approach:
• Institutional and course level
• Process as well as product
• Structures and processes embed a concern for quality
throughout development, delivery and evaluation
Jansen, D., Rosewell, J., & Kear, K. (2017). ‘Quality Frameworks for MOOCs.’ In: M. Jemni, Kinshuk, & M. K. Khribi (Eds.), Open
Education: from OERs to MOOCs, 261–281. Springer http://oro.open.ac.uk/47595/

61. Summary

62. In summary…
• A quality framework should underpin e-learning provision
– to help create a quality culture
– that is more likely to produce quality e-learning
– and quality enhancement
• There is no simple recipe, but…
– Work in a module team
– Think about learning design
– Think about student support

63. Checklist 1: Is it a MOOC or not?
Dimension Criteria Level
Massive Pedagogical model means effort doesn’t increase
significantly as number of participants increase
Open Course accessible to (almost) all people without limitation
Full course experience available without cost
Online All aspects are delivered online
Course At least 1 ECTS (25-30 hours of study)
Participants receive some feedback (e.g. automated
quizzes, peers, general feedback from staff)
At least some recognition like badge or certificate of
completion.
Levels: Not achieved, Partially achieved, Largely achieved, Fully achieved