2017-08-30 EARLI 2017 - MOOQ Interactive Workshop Results moderated by Christian M. Stracke from OUNL

1. MOOQ for the quality of MOOCs:
“We will make MOOCs better”
Quality Reference Framework with
indicators for design & comparison
www.MOOC-quality.eu
Frameworks: MOOQ

2. Quality of Open
Education and MOOCs:
Interactive Workshop
@ EARLI 2017 in Tampere, Finland
by Christian M. Stracke
Open University of the Netherlands

3. Open CC License for
sharing & re-using slides
This work is free to share under the creative commons licence:
"Attribution – Noncommercial – Share Alike 3.0"
You can copy, distribute and transmit the work under the following conditions:
1. Attribution –
2. Noncommercial –
3. Share Alike
Licence: Attribution-Noncommercial-Share Alike
Some rights reserved, see: http://creativecommons.org/licenses/by-nc-sa/3.0/

4. Hands-on-Session
Explore the quality indicators of
MOOCs

5. MOOQ for the quality of MOOCs:
“We will make MOOCs better”
Quality Reference Framework with
indicators for design & comparison
www.MOOC-quality.eu
Frameworks: MOOQ

6. Our proposal
for discussion:
The Quality Reference
Framework (QRF)

7. Quality Reference Framework with
indicators for design & comparison
Our main goal is the collaboration with all
to improve future MOOC design for
better MOOC learning experiences

8. 16 questions in 4 clusters (n=45):
1. Experience with MOOCs (5 questions)
2. Perception and Expectation (5 questions)
3. Personal Goals (4 questions)
4. Recognition and Certification (2 questions)
n=45: reveals that many MOOC learners
do not share the intentions of the
MOOC designer & have their own goals
MOOQ Pre-Survey

10. Three main surveys for learners,
designers and facilitators of MOOCs
Please support us to improve next MOOCs:
www.survey.MOOC-quality.eu
Your feedback is most welcome!

11. Global MOOC Survey

12. Combines 3 surveys on 13 constructs:
1. MOOC learners (69 questions)
2. MOOC designers (89 questions)
3. MOOC facilitators (58 questions)
3 months with huge support (n=625):
reveals that most MOOC learners
reported positive learning experiences
with MOOCs
MOOQ Survey

13. MOOQ Survey: 1st results

14. What is Open Education (OE)?
Quality dimensions of OE in practice?
Quality indicators for OE in practice?
Quality Open Education

15. 1. Pedagogical
2. Technological
3. Business Model
MOOQ Matrix

16. 1. MOOC Learners
2. MOOC Designers
3. MOOC Facilitators
4. MOOC Providers
MOOQ Target Groups

17. OE Quality Dimensions
Evaluation
Analysis
Implemen-
tation
Learning
process
Design
Learner

18. QRF Target Groups
Meso
Micro
Macro
MOOC providers
MOOC designers
MOOC facilitators
MOOC learners
National ministries
Local authorities

19. 2 Parts
of the interactive
MOOQ Workshop:
Survey and
Quality indicators

20. Part 1
of the interactive
MOOQ Workshop:
Survey

21. How much do you agree with the
five phases?
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q1: 5 Phases
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 1 3 11 1 0

22. The Quality Reference
Framework (QRF)
Target groups

23. How much do you agree with the
planned target groups?
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q1: Target Groups
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 1 12 1 0

24. The Quality Reference
Framework (QRF)
Instruments

25. How much do you agree that the
planned instruments are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q1: 9 Instruments
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 1 9 2 1

26. Part 2
of the interactive
MOOQ Workshop:
Quality indicators

27. OE Quality Dimensions
Evaluation
Analysis
Implemen-
tation
Learning
process
Design
Learner

28. Quality dimensions of OE in practice
Dimension 1: Analysis
Dimension 2: Design
Dimension 3: Implementation
Dimension 4: Learning Process
Dimension 5: Evaluation & Optimization
OE Quality Dimensions

29. Quality indicators for OE in practice
For Dimension 1:
For Dimension 2:
For Dimension n:
OE Quality Indicators

30. Quality indicators for analysis in OE:
• Content structure
• Learning tasks
• Characteristics of target groups
• Budgets and resources
• Learning context
• Why provided as MOOC? And not other type?
• Convince stakeholders
OE D1: Analysis

31. Quality indicators for design in OE:
• Usability
• Accessibility
• Constructive alignment of teaching (content
and objectives are aligned)
• Interactivity design
• Action design
• Multimedia design
• Motivation design
OE D2: Design

32. Quality indicators for design in OE:
• Quality of learning tasks
• Teaching strategy (deductive vs inductive)
• Integration of content into narration
• Flexibility
• Heterogenuous learning paths
• Focus on precise definition of learning content
+ what you can learn (goals, structure, intention)
• What is the added value of the MOOC?
OE D2: Design

33. Quality indicators for design in OE:
• Level of the MOOC (basics vs expertise)
• Strategic plan for realization + marketing
• Target audience definition + characteristics
• Basic and special features (also for marketing)
• Definition of indicators for learning success
(maybe clustering individual intentions into
groups of learners)
OE D2: Design

34. Quality indicators for design in OE:
• Question: How are the indicators related to
each other? (dependencies/intertwinned)
OE D2: Design

35. Q. indicators for implementation in OE:
• Platform selection based on criteria for
learning goals and design
• Communicate with the stakeholders
• Meet the technical conditions + requirements
• Create the course area with all assignments
according to the design
• Define implementation period
• Marketing
OE D3: Implementation

36. Q. indicators for implementation in OE:
• Changing from implementation to preparation?
OE D3: Implementation

37. Quality indicators for learning in OE:
• Collect enrollments before the start of the
Learning process
• Scaffolding or help
• Enough interactions among peers
• Feedback from instructors
• Measuring of indicators for learning success
(maybe clustering individual intentions into
groups of learners)
OE D4: Learning Process

38. Quality indicators for learning in OE:
• Monitoring system of learning process (that
learners know where they are)
• Self-directed feedback through assignments
and self-assessment (in MOOCs without grades)
• Assessment of students learning within the
learning process
OE D4: Learning Process

39. Quality indicators for evaluation in OE:
• Kirkpatrick‘s four levels
• All quality indicators should be part of the
evaluation (quality criteria for the design have
to be evaluated)
OE D5: Evaluation