This document discusses quality in e-learning environments. It begins by defining distance education and noting its accelerated expansion and increased use of online, cohort-based models. The document then discusses why quality is important, given that growth may be slowing and students care about outcomes. It reviews arguments that e-learning is of the same quality as traditional learning and that no significant differences exist. However, it notes that quality can be subjective. The document proposes ways to measure quality, such as using best practices like clear communication and encouraging interaction. It provides resources for ensuring quality, such as benchmarks from the National Education Association. It concludes by posing emerging questions about how to define and ensure quality in e-learning.

1. Addressing the Question
of Quality in E-Learning
Environments
Gonzalo Bacigalupe, EdD, MPH
Associate Professor & Chair
Department of Counseling & School Psychology
University of Massachusetts Boston
gonzalo.bacigalupe@umb.edu
Distance Education Definitions
•Instructor and students need not be in the same
physical place. (NCES, U.S. DOE)
•A formal educational process … the major portion
of the instruction occurs when the learner and the
instructor are not in the same place at the same
time through virtually any media. A Distance
Learning Program … over half of the required
courses in the program occur when the learner and
the instructor are not in the same place at the
same time. (NCATE GUIDELINES, 2005)
1

2. ODE Features
• Accelerated expansion
• Vast majority of new distance programs are
offered online with a cohort-based model (as
opposed to self-paced) (DOE-OPE, 2006)
• Access (flexibility, working students, travel,
technical application) is core in mission of
institutions offering ODE
ODE in the U.S.
(Sloan Consortium, 2005)
2

3. Why worry •The days of astonishing
growth in ODE may be
about over
quality? •Brands based on student
outcomes may prove
more successful
•Faculty, curriculum, and
institutional reputation
are attributes online
learners care about.
(Stokes, 2006)
Faculty have been proceeding for
years teaching courses, hoping
that they are effectively
presenting what they themselves
have learned, and that students
then are demonstrating that they
have learned what the faculty
think they should be learning.
The advent of online learning has
brought to the fore pressure on
us in higher education to
demonstrate outcomes in a way
that we have never felt before…
(Hamlin, et.al., 2004, p.3)
Note for those not teaching online:
This is not just in the case of e-
learning
3

4. ODE Defenders Responses
• ODE program and courses are of the same
caliber as the one on campus
• ODE increases access to a more diverse
constituency and enhances inclusiveness
• ODE offer educational opportunities for
all students that may not be possible in
campus
•…?
No significant difference hypothesis
•Instructors adopting learning management systems must consider
how students' evaluations of learning reflect the overall quality of
instruction. Traditional course evaluations are used for faculty
retention, tenure and promotion decisions, but also provide instructors
with valuable information on the quality of their instruction.
•Topper (2007) studied the response rates and compared instructional
quality, using student course evaluations along with additional data
from online and face-to-face graduate education courses, to evaluate
the effectiveness of instruction. The statistical analysis of course
evaluations showed no significant difference in instructional quality
based on the format used.
•Together with comparisons of student work, these results provide
additional evidence in support of the finding of no significant difference
between formats in the area of instructional quality.
•Topper, (2007). Are they the same? Comparing the instructional quality of online and face-to-face graduate education courses. . Assessment & Evaluation
in Higher Education 32 (6), 681-691.
4

5. Caveat
Quality is subject to interpretation
5

6. Measuring
• Everyone involved in online learning would agree that quality is
paramount; however, objectively measuring quality is difficult and
infrequently undertaken.
• Quality is an unstated expectation, yet we rarely use a formal
process for assessing quality of training products.
• Poor design, project under funding, overly optimistic schedules, and
technical barriers are all enemies of quality.
• A methodical approach to evaluation can help remove subjective
biases and achieve a more authoritative analysis.
• (WBTIC, 1994-2007)
Learning styles
“Educators
must do more
than provide
access to
information”
(Campbell,
1998, p.vii).
6

7. Sloan (Moore, 2005)
Sloan
7

8. Scorecard
Questions by Accreditation Body:
Example
Unit Accreditation Board = UAB
8

9. Red Flags
 ODE Curriculum approval differs from F2F
 Large numbers of students in DE courses
 Target population needs are not addressed
 Single method of assessment
 Students’ dissatisfaction
Red Flags
 Unclear course objectives
 Cookie cutter approach to course development
 Very little board/threaded/lively discussion
 Lack of substance and reflexivity of students’
postings
 Rapid turnover in DOE adjunct faculty teaching
 No regular update of courses
9

10. Best Practices Principles
• Communicate clearly your expectations
• Timely submissions
• Prompt feedback
• brief assignments, and/or
• accumulative scaffolding
• High expectations for students
• Encourage faculty-student contact
Best Practices Principles
• Stimulate student collaboration and cooperation
• Uncover students’ knowledge and skills
• Different learning styles
• Resources for teaching and peer tutoring
• Make the course your “brand”
• Share the outcome (OCW) be accountable
10

11. NATIONAL EDUCATION ASSOCIATION (NEA)
Benchmarks for Success in Internet-Based Distance Education
• Institutional Support Benchmarks
• Technology plan to ensure information integrity
• Reliable technology system and centralized support system
• Course Development Benchmarks
• Guidelines for minimum standards are used for course
development, design, and delivery, while learning outcomes
determine the technology being used to deliver course content.
• Instructional materials reviewed periodically to meet program
standards
• Courses design requires students to engage themselves in
analysis, synthesis, and evaluation as part of their course and
program requirements
NEA Benchmarks for Success in Online Distance Education
• Technology/Learning
• Student-faculty and student interaction is essential
• Feedback is timely and constructive
• Students learn about proper methods of effective research
• Course Structure
• Students are advised before program start to determine self-motivation
and commitment to distance learning and to assess technology access
• Course information for students is exhaustive (i.e., syllabus)
• Students have access to sufficient library resources
• Clear expectations about assignment completion
11

12. NEA Benchmarks for Success in Online Distance Education
•Student Support
•Students receive information about requirements, tuition and fees, student support, etc. Hands on
training is available to students to secure digital information
•Faculty Support
•Course development technical assistance
•Support in transition from classroom to online
•Permanent training and peer mentoring
•Evaluation and Assessment
•Multi-method of assessment and specific standards to assess effectiveness and teaching/learning
process
•Data on enrollment, costs, and successful/innovative use of technology are used to evaluate
program effectiveness
•Intended learning outcomes are reviewed regularly
Some best practices at UMB
• Instructor and designer have been the same
• Curriculum is approved by faculty governance
• Multiple technologies are employed in delivery of
online and hybrid courses
• Some programs have acquired a rich experience
in the use of synchronous modalities
• Wide variety of disciplines and programs have
developed ODE offerings
12

13. What needs improvement?
• System wide decision-making may have
hindered the ability of campus to innovate
and adopt appropriate resources
• Technology resources/support for hybrid
and on campus courses are not comparable
with ODL -> Pace of technology adoption may
slow down and hurt ability to mainstreaming
use across classrooms
• Resources for assessment, evaluation, and
research have not accompanied allocation
of resources for course development
ODE Quality: Emerging Questions
• Do synchronous versus asynchronous venues
impact access?
• What criteria should we employ to measure
adequate instructor’s availability?
• How can faculty and students successfully
manage the potential of social networking tools?
• Does ODE increases/enhances access of
marginalized groups or foster the status quo?
• Is ODE generative of knowledge creation, quality
scholarship, collaboration, community service,
etc.)?
13

14. Useful Resources
• The Sloan Consortium Effective Practices Quality Framework www.sloan-c.org/effective/framework.asp
• ADEC Guiding Principles for Distance Teaching and Learning www.adec.edu/admin/papers/distance-
teaching_principles.html
• EFA Global Monitoring Report portal.unesco.org/education/en/ev.php-
URL_ID=35939&URL_DO=DO_TOPIC&URL_SECTION=201.html
• How effective is distance education? An AskEric Response www.eduref.org/cgi-
bin/printresponses.cgi/Virtual/Qa/archives/Educational_Technology/Distance_Education/disted.html
• UMASS Information Technology Council www.massachusetts.edu/itc/mission.html
• Quality Principles University of Wisconsin-Madison www.wisc.edu/depd/html/quality3.htm
• ESAC Faculty Development, Quality Matters: Inter-Institutional Quality Assurance in Online Learning
www.esac.org/fdi/rubric/finalsurvey/demorubric.asp
• National Standards of Quality for Online Courses www.nacol.org/nationalstandards/
• What Constitutes Quality in Web-Based Training? www.webbasedtraining.com/primer_quality.aspx
• My e-learning links at delicious http://del.icio.us/bacigalupe/e-learning
• REFERENCES: In development, please
email me for a copy in a couple of weeks.
• gonzalo.bacigalupe@umb.edu
14