2017-09-13 EC-TEL MOOQ Interactive Workshop Results moderated by Christian M. Stracke and Esther Tan from OUNL

1. MOOQ for the quality of MOOCs:
“We will make MOOCs better”
Quality Reference Framework with
indicators for design & comparison
www.MOOC-quality.eu
Frameworks: MOOQ

2. Quality of Open
Education and MOOCs:
Interactive Workshop
@ EC-TEL 2017 in Tallinn, Estonia
by Christian M. Stracke & Esther Tan
Open University of the Netherlands

3. Open CC License for
sharing & re-using slides
This work is free to share under the creative commons licence:
"Attribution – Noncommercial – Share Alike 3.0"
You can copy, distribute and transmit the work under the following conditions:
1. Attribution –
2. Noncommercial –
3. Share Alike
Licence: Attribution-Noncommercial-Share Alike
Some rights reserved, see: http://creativecommons.org/licenses/by-nc-sa/3.0/

4. Hands-on-Session
Explore the quality indicators of
MOOCs

5. MOOQ for the quality of MOOCs:
“We will make MOOCs better”
Quality Reference Framework with
indicators for design & comparison
www.MOOC-quality.eu
Frameworks: MOOQ

6. Our proposal
for discussion:
The Quality Reference
Framework (QRF)

7. Quality Reference Framework with
indicators for design & comparison
Our main goal is the collaboration with all
to improve future MOOC design for
better MOOC learning experiences

8. 16 questions in 4 clusters (n=45):
1. Experience with MOOCs (5 questions)
2. Perception and Expectation (5 questions)
3. Personal Goals (4 questions)
4. Recognition and Certification (2 questions)
n=45: reveals that many MOOC learners
do not share the intentions of the
MOOC designer & have their own goals
MOOQ Pre-Survey

10. Three main surveys for learners,
designers and facilitators of MOOCs
Please support us to improve next MOOCs:
www.survey.MOOC-quality.eu
Your feedback is most welcome!

11. Global MOOC Survey

12. Combines 3 surveys on 13 constructs:
1. MOOC learners (69 questions)
2. MOOC designers (89 questions)
3. MOOC facilitators (58 questions)
3 months with huge support (n=625):
reveals that most MOOC learners
reported positive learning experiences
with MOOCs
MOOQ Survey

13. MOOQ Survey: 1st results

14. What is Open Education (OE)?
Quality dimensions of OE in practice?
Quality indicators for OE in practice?
Quality Open Education

15. 1. Pedagogical
2. Technological
3. Business Model
MOOQ Matrix

16. 1. MOOC Learners
2. MOOC Designers
3. MOOC Facilitators
4. MOOC Providers
MOOQ Target Groups

17. OE Quality Dimensions
Evaluation
Analysis
Implemen-
tation
Learning
process
Design
Learner

18. QRF Target Groups
Meso
Micro
Macro
MOOC providers
MOOC designers
MOOC facilitators
MOOC learners
National ministries
Local authorities

19. 2 Parts
of the interactive
MOOQ Workshop:
Survey and
Quality indicators

20. Part 1
of the interactive
MOOQ Workshop:
Survey

21. How much do you agree with the
five phases?
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q1: 5 Phases
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:

22. The analysis phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q2: Analysis Phase
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

23. The design phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q3: Design Phase
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

24. The implementation phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q4: Implementation
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

25. The learning process phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q5: Learning Process
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

26. The evaluation phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q6: Evaluation
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

27. The Quality Reference
Framework (QRF)
Target groups

28. How much do you agree with the
planned target groups?
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
School management/headmasters and producers to be added
T-Q1: Target Groups
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:

29. The national ministries are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q2: National ministries
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

30. The local authorities are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q3: Local authorities
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

31. The MOOC providers are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q4: MOOC providers
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

32. The MOOC designers are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q5: MOOC designers
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

33. The MOOC facilitators are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q6: MOOC facilitators
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

34. The MOOC learners are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q7: MOOC learners
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 1

35. The Quality Reference
Framework (QRF)
Instruments

36. How much do you agree that the
planned instruments are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q1: 9 Instruments
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:

37. How much do you agree that the
planned MOOC is helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q2: MOOC
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 1 2

38. How much do you agree that the
planned guidelines are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q3: Guidelines
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 1

39. How much do you agree that the
planned evaluation tools are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q4: Evaluation tools
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

40. How much do you agree that the
planned online communities are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q5: Online Communities
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

41. How much do you agree that the
planned design patterns are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
Maybe to change to design models (plus recommender systems?)
I-Q6: Design patterns
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 1

42. How much do you agree that the
planned checklists are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q7: Checklists
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2

43. How much do you agree that the
planned use cases are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q8: Use cases
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 1

44. How much do you agree that the
planned authoring system is helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q9: Authoring system
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 1

45. How much do you agree that the
planned OER collection is helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q10: OER collection
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 1

46. The Quality Reference
Framework (QRF)
Instrument/Target group

47. For which target groups is the
planned MOOC helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q1: MOOC
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
1 4 2

48. For which target groups are the
planned guidelines helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q2: Guidelines
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
2 4 3

49. For which target groups are the
planned evaluation tools helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q3: Evaluation tools
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
1 3 1 2

50. For which target groups are the
planned online communities helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q4: Onl. communities
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
2 2 2

51. For which target groups are the
planned design patterns helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q5: Design patterns
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
2

52. For which target groups are the
planned checklists helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q6: Checklists
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
2 2 1

53. For which target groups are the
planned use cases helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q7: Use cases
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
2 2

54. For which target groups is the
planned authoring system helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q8: Authoring system
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
1 2

55. For which target groups is the
planned OER collection helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I/T-Q9: OER collection
National
Ministries
Local
authorities
MOOC
providers
MOOC
designers
MOOC
facilitators
MOOC
learners
2

56. Part 2
of the interactive
MOOQ Workshop:
Quality indicators

57. OE Quality Dimensions
Evaluation
Analysis
Implemen-
tation
Learning
process
Design
Learner

58. Quality dimensions of OE in practice
Dimension 1: Analysis
Dimension 2: Design
Dimension 3: Implementation
Dimension 4: Learning Process
Dimension 5: Evaluation & Optimization
OE Quality Dimensions

59. Quality indicators for OE in practice
For Dimension 1:
For Dimension 2:
For Dimension n:
OE Quality Indicators

60. Quality indicators for analysis in OE:
• XXX
OE D1: Analysis

61. Quality indicators for design in OE:
• XXX
OE D2: Design

62. Q. indicators for implementation in OE:
• XXX
OE D3: Implementation

63. Quality indicators for learning in OE:
• XXX
OE D4: Learning Process

64. Quality indicators for evaluation in OE:
• Post questionnaire for four parties (analysts,
designers, implementers and learners)
• External audits
• Ratings after each activitiy within the MOOC
• Combination of the quality indicators
• Discount as incentive for next MOOC
OE D5: Evaluation