This research investigates the use and application of quality assurance rubrics and standards for online/blended course design within the context of teaching English for general academic purposes (EGAP). In an attempt to reform an existing blended course of EGAP, the Quality Matters® Higher Education Course Design Rubric (fifth edition) has been used to assure the quality of course design and development from the outset. After the course was implemented with a cohort of undergraduate students at Osaka University, it was peer-reviewed by a certified reviewer affiliated with Quality Matters (QM). The first round of peer review yielded a score of 70 out of a total of 99, resulting in failure to meet several QM standards. Based on the suggestions and recommendations of the reviewer, the course was further revised in terms of content and interface, and it was reviewed for a second time by the same reviewer, this time yielding a full score of 99. This study highlights the importance of quality assurance in online/blended course design and development, particularly in the context of teaching English to university students.

1. Quality Matters®
in Blended Course
Design & Development
Mehrasa Alizadeh
Osaka University
XXth International CALL Research Conference
The Education University of Hong Kong
July 10-12, 2019

2. In this presentation …
▰ Blended learning in Higher Education
▰ Quality Matters® Program
▰ Case Study
2

3. Blended learning
in Higher Education
A Brief Introduction
3
1

4. Blended Learning in higher education
has received considerable attention.
(Garrison & Kanuta, 2004; Garrison & Vaughan, 2008)
4
Face-
to-
Face
OnlineBlended

5. Also in Japan …
The number of blended university courses is
increasing and is expected to grow rapidly.
(Gruba & Hinkelman, 2012)
5

6. In Japanese Higher Education
Blended Learning Can …
▰ Provide a more integrated approach to teaching/learning
▰ Prevent learner alienation
▰ Enhance completion rates & lower dropout rates
▰ Improve the quality and quantity of interaction
(McCarty, Sato & Obari, 2017)
6

7. Quality Assessment
is essential in blended course design,
development and delivery.
(Gruba, Cárdenas-Claros, Suvorov, & Rick, 2016)
7

8. Checklists, Guidelines & Rubrics
▰ Rubric for e-Learning Tool Evaluation by
Western University (Anstey & Watson, 2018)
▰ Higher Education Course Design Rubric by
Quality Matters® (Adair & Shattuck, 2015)
▰ Illinois Quality Online Course Initiative Rubric
by Illinois Online Network (University of Illinois,
2018)
▰ Blackboard Exemplary Course Program Rubric
(Blackboard Community Programs, 2012) 8

9. Quality Matters® Program
What is it and why?
9
2

10. Quality Matters® is …
▰ An international organization that specializes in
quality assurance for online education in
▻ K-12
▻ Higher Education
▻ Continuing Professional Education
10

11. Quality Matters® aims to …
Promote and improve the quality of online/blended education
through:
▰ Developing course or program quality assessment rubrics
▰ Providing professional development for online/blended
course design and evaluation
▰ Offering peer review and certification of quality in online
education
11

12. Quality Matters® has been …
Adopted by numerous educational institutions
and individuals to review and assess the quality
of their online and blended courses.
(Zimmerman & Adair, 2014)
12
1195 Member Institutions

13. QM Member Institutions Worldwide
13
Japan
Fiji
Australia
Vietnam
South Africa
Saudi Arabia
Kuwait
Jordan
ColumbiaMexico
The U.S.
Canada

14. Previous Studies – Program Eval.
Harkness (2015)
Five academic years of
strategic application of QM to
online learning programs
Establishment of sustainable
online education
Passing grades increased and
failing grades and withdrawals
decreased.
Kwon, DiSilvestro, and Treff (2017)
Adult education graduate program
Online courses evaluated by students and peer
instructors
Consistency btw students’ and peer instructors’
evaluations
Courses fulfilled the key components of the QM
standards.
14

15. Previous Studies – MOOC Eval.
Lowenthal and Hodges (2015)
Six randomly selected MOOCs
Three trained QM peer reviewers
Some of the MOOCs scored very well with
minor revisions.
Two of the MOOCs could pass the QM
review.
15

16. Previous Studies – Course Eval.
Dietz-Uhler, Fisher,
and Han (2007)
Online course design and
student retention
Psychology and statistics
courses
Retention rates were
approximately 95%.
16
Al Zumor (2015)
Scrutinized the standards of the QM rubric,
2011–2013 Edition
The rubric has the potential for enhancing
online foreign language education in
general.
It can make EFL learning process more
humanized by increasing the instructors’
and learners’ sense of online presence.
EFL

17. Case Study
Blended EGAP Course at
Osaka University 17
3

18. Case Description & Setting (1)
Blended Course of EGAP: Practical English (e-learning)
Objectives:
▻ helping students improve their academic
English proficiency
▻ getting them prepared for studying in
English-speaking countries
▻ enabling them to gain a score of 490 to 520
on the TOEFL ITP® 18

19. Case Description & Setting (1)
The course included:
19

20. New Practical English (e-learning)
The new course aims at:
20
Developing students’ practical English
language skills, in particular productive
skills
Helping them gain skill and
confidence when speaking

21. Old vs. New Course
21
Course Components/ Features Old Course New Course
Placement Test ✗ ✓
Different Levels ✗ ✓
Integration of skills ✗ ✓
Speaking/writing tasks ✗ ✓
Focus on global themes ✗ ✓
Group project ✗ ✓
Free of charge ✗ ✓
Hosted on the university LMS ✗ ✓
Quizzes ✓ ✓
Audio-visual materials ✓ ✓
Face-to-face and online sessions ✓ ✓

22. Quality Matters Peer Review
▰ Fifth edition of the QM Higher Education
Course Design Rubric (Quality Matters, 2014)
▰ Eight general standards and 43 specific
standards
22

23. QM Higher Ed General Standards
23
Course Overview and Introduction
Learning Objectives (Competencies)
Assessment and Measurement
Instructional Materials
Learning Activities & Learner Interaction
Course Technology
Learner Support
Accessibility and Usability

24. Official
Course
Review
QM Peer Review Types
24
Internal
Review
Self
Review
Prepara
-tory
Review

25. QM Course Review Process
25
(Adair, 2014)

26. QM Peer Review Results
26
The first
round of
review Yielded a
score of 70 out of
99
The course did not meet all the essential QM standards.

27. Standard 1.1
Instructions were available, but they were not
readily seen by the students.
A welcome page was created and set as the
course entry page, in which information about
navigating the course menu and content was
provided through written instructions and
screenshots.
27
Instructions make clear how to get started and
where to find various course components.

28. Standard 2.4
Previously, the course activities were not
clearly linked to the course objectives and
learning outcomes mentioned in the syllabus.
The connection was clarified by assigning
each type of activity to the corresponding
learning outcome in the syllabus.
28
The relationship between learning objectives or
competencies and course activities is clearly stated.

29. Standard 3.3
This standard was met by sharing the
evaluation rubrics for speaking and writing
tasks with the students.
These rubrics were not immediately visible to
guest viewers, and therefore the course was
evaluated as lacking this essential
component in the first round of review.
29
Specific and descriptive criteria are provided for the
evaluation of learners’ work and are tied to the course
grading policy.

30. Standard 5.3
A new section was added to the syllabus
titled “Response Time and Feedback
Schedule” in which a rough schedule was
provided for responding to inquiry emails and
grading assignments.
30
The instructor’s plan for classroom response time
and feedback on assignments is clearly stated.

31. Standard 7.2
A new link was added to the course menu
through which students could access a page
containing information on Osaka University
institutional policies for each faculty.
31
Course instructions articulate or link to the
institution’s accessibility policies and services.

32. Standard 8.2
A new page was created which contained
information on the technologies required in
the course, for instance a computer with a
standard browser.
Links were provided to the accessibility pages
of the websites introduced to the students.
32
Information is provided about the accessibility of
all technologies required in the course.

33. Review Process at a Glance
33
First Round
of Review
Score: 70/99
Amendments
were made
Second
Round of
Review
Score: 99/99

34. Comparison to Previous Studies
▰ Harkness (2015)
▰ Hollowell et al. (2017)
34
Effectiveness of the QM rubrics and peer review in assuring
excellence in online/blended courses or programs

35. Suggestions for Course Designers
▰ Embracing change
▰ Using open educational resources (OERs)
▰ Considering time demands
▰ Prioritizing course objectives
▰ Ensuring learner-friendly course structure and
navigation
35

36. Suggestions for Course Designers
▰ Explicitly stating the requirements
▰ Setting clear evaluation criteria
▰ Paying attention to accessibility and usability
▰ Fostering social presence
▰ Providing technical support
36

37. References
Adair, D. (2014). A process to improve course design: A key variable in course quality. In K.
Shattuck (Ed.), Assuring quality in online education: Practices and processes at the
teaching, resource, and program levels (pp. 81–90). Sterling, VA: Stylus.
Adair, D., & Shattuck, K. (2015). Quality MattersTM: An educational input in an ongoing
design-based research project. American Journal of Distance Education, 29(3), 159-165.
https://doi.org/10.1080/08923647.2015.1057094
Al Zumor, W. H. Q. (2015). Quality Matters rubric potential for enhancing online foreign
language education. International Education Studies, 8(4), 173–178.
http://dx.doi.org/10.5539/ies.v8n4p173
Anstey, L. & Watson, G. (2018, September). A rubric for e-learning tool evaluation.
EDUCAUSE Review. Retrieved from https://er.educause.edu/articles/2018/9/a-rubric-for-
evaluating-e-learning-tools-in-higher-education
Blackboard Community Programs. (2012). Blackboard exemplary course program rubric.
Retrieved from https://www.blackboard.com/resources/getdocs/7deaf501-4674-41b9-
b2f2-554441ba099b_bbexemplarycourserubric_nov12final.pdf 37

38. References
Dietz-Uhler, B., Fisher, A., & Han, A. (2007). Designing online courses to promote student
retention. Journal of Educational Technology Systems, 36(1), 105–112.
https://doi.org/10.2190/ET.36.1.g
Garrison, D. R., & Kanuta, H. (2004). Blended learning: Uncovering its transformative
potential in higher education. The Internet and Higher Education, 7(2), 95–105.
https://doi.org/10.1016/j.iheduc.2004.02.001
Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework,
principles, and guidelines. San Francisco, CA: Jossey-Bass.
Gruba, P., & Hinkelman, D. (2012). Blending technologies in second language classrooms.
London, England: Palgrave-Macmillan.
Gruba, P., Cárdenas-Claros, M. S., Suvorov, R., & Rick, K. (2016). Blended language program
evaluation. London, England: Palgrave-Macmillan.
Harkness, S. S. J. (2015). How a historically black college university (HBCU) established a
sustainable online learning program in partnership with quality Matters™. American Journal
of Distance Education, 29(3), 198–209. https://doi.org/10.1080/08923647.2015.1057440
38

39. References
Hollowell, G. P., Brooks, R. M., & Anderson, Y. B. (2017). Course design, quality Matters
training, and student outcomes. American Journal of Distance Education, 31(3), 207–216.
https://doi.org/10.1080/08923647.2017.1301144
Kwon, K., DiSilvestro, F. R., & Treff, M. E. (2017). Online graduate course evaluation from
both students’ and peer instructors’ perspectives utilizing Quality MattersTM. Internet
Learning, 5(1), 7-16. https://doi.org/10.18278/il.5.1.2
Lowenthal, P. R., & Hodges, C. B. (2015). In search of quality: Using quality Matters to
analyze the quality of massive, open, online courses (MOOCs). International Review of
Research in Open and Distributed Learning, 16(5), 83–101 Retrieved from
http://www.irrodl.org/index.php/irrodl/article/view/2027/3438
McCarty, S., Sato, T., & Obari, H. (2017). Implementing mobile language learning
technologies in Japan. Singapore: Springer International Publishing.
https://doi.org/10.1007/978-981-10-2451-1
39

40. References
Quality Matters. (2014). Standards from the QM higher education rubric (5th ed.). Retrieved
from
https://www.qualitymatters.org/sites/default/files/PDFs/StandardsfromtheQMHigherEduc
ationRubric.pdf
Roehrs, C., Wang, L., & Kendrick, D. (2013). Preparing faculty to use the quality Matters
model for course improvement. MERLOT Journal of Online Learning and Teaching, 9(1),
52–67 Retrieved from http://jolt.merlot.org/vol9no1/roehrs_0313.pdf
Shattuck, K., Zimmerman, W. A., & Adair, D. (2014). Continuous improvement of the QM
rubric and review processes: Scholarship of integration and application. Internet Learning,
3(1), 25-34. https://doi.org/10.18278/il.3.1.4
University of Illinois. (2018). Quality online course initiative (QOCI) rubric. Retrieved from
https://www.uis.edu/ion/resources/qoci/
40

41. 41
THANKS!
Any questions?
You can find me at
mehrasa@cmc.osaka-u.ac.jp
Presentation template by SlidesCarnival.com