Evaluating the Quality of Online Teaching

Evaluating the Quality of Online Teaching
Barry Dahl, Sr. Community Manager, D2L

What is a Good Definition for Quality
Generally speaking…
•High Grade
•Superiority
•Excellence

Define Quality in Education
•High standards
•Consistently applied
•With an eye on efficiency (or value for money)

Define Quality in Online/Digital Education
Passing the Quality Matters (QM) rubric review
But…
It shouldn’t be

3 Major Components of e-Quality
Learning
Teaching Design

Learning Level
Is High
Teaching Level
Is High
Course Design
Meets Standards
Learning Assessment
Measuring the Major Components

Comprehensive Faculty Evaluations
• Purpose
• Accreditation
• History of Online Faculty Evaluation
• Components
• A few examples
• What about classroom observations for online courses?

Purposes of Faculty Evaluation
1. Encourage excellence in teaching and learning.
2. Facilitate long-term quality improvement by continually
monitoring instructional performance.
3. Provide constructive feedback to faculty by identifying
areas of strength and areas for improvement in classroom
instruction.
4. Inspire professional growth and development.

What about Accreditation?
• Here’s an example from the Higher Learning
Commission
• Guidelines for the Evaluation of Distance Education
(On-line Learning)
• NOTE: most other regional accreditors use the
same or similar language.

Guidelines for the Evaluation of DE
Faculty responsible for delivering the on-line learning
curricula and evaluating the students’ success in
achieving the on-line learning goals are appropriately
qualified and effectively supported.
• Examples of evidence: a. On-line learning faculties
are carefully selected, appropriately trained,
frequently evaluated, and are marked by an
acceptable level of turnover.

A Little History…Maybe
• The first known (to me, anyway) online faculty
evaluation system was started in 2001 at
Park University.
• Initially based on the evaluation system used
for face-to-face classroom instructors.
• As such, they attempted to replicate the
traditional classroom evaluation in the online
classroom.

But Online Was (Is) Different
The unique features and functions in an online classroom
were not initially considered.
For example, the original evaluation system did not include
an examination
• of learning outcomes
• of overall classroom management
• facilitation and guidance
• the concept of faculty presence
• communication response rates
• accessibility of materials
• course-related administrative tasks, etc.

Adapting the Instrument
• To address the unique evaluation concerns for
online teaching, Park University College for
Distance Learning developed a formalized Online
Instructor Evaluation System (OIES).
• OIES launched as a pilot in Fall 2004, and was
used through 2008.
• OIES morphed into the Faculty Online Observation
(FOO) model and is still in use today.

Evaluation System Components
• Student Evaluation of Instruction
• Faculty Self-Evaluation
• Professional Development Plan
• Faculty Peer Review
• Faculty Portfolio
• Supervisor’s Evaluation of Faculty
Usually some combination of the above components

Student Evaluation of Online Instruction
• Not our main focus here.
• However, some questions or issues to focus on:
• How to get decent response rates
• Use incentives, or not?
• What effect does the length of survey have on responses?
• Does it need to be comparable to F2F evaluation surveys?
• How valuable are these overall?

Faculty Self-Evaluation
• You might love these, you might hate these. You also might
mostly ignore these.
• Let’s look at two examples:
• Open-ended questions
• Specific questions tied to student eval

Open-ended questions
Sample questions from faculty self-evals:
1. How has your understanding of your role as a faculty
member changed/developed since your last evaluation?
2. As you prepare for future semesters, what more could you
do to provide students with a successful learning
experience?
3. What more can the university do to support you in your
professional goals and development?

Questions tied to student evaluations
Wharton County Junior College
• Online Course Evaluation (completed by students)
4. Instructor requires me to be an active participant in class.
9. Instructor is present regularly in the course
(announcements, emails, discussion board, feedback, etc.).
• Online Faculty Self-Evaluation
4. I require students to be an active participant in class.
9. I am regularly present in my course (announcements,
emails, discussion board, feedback, etc.).

Faculty Peer Review of Online Teaching
Great example comes from Penn State University
• College of Earth and Mineral Sciences
• Based on the “Seven Principles for Good Practice in
Undergraduate Education”
• Two Parts:
• An Instructor Input Form to be completed for the reviewer by the
reviewee in advance of the peer review, and
• The actual Peer Review Guide for Online Teaching at Penn State,
which is to be completed by the reviewer during the peer review.

Screenshot of Guide
Last revision June 23, 2017 - Ann H. Taylor, Dutton e-Education Institute, College of Earth and Mineral Sciences, The Pennsylvania State University

Examples of Evidence to Look For

Where to Look, and Resources Links

Portfolio for Faculty (Austin CC)
The portfolio will include the following:
a. Syllabus for each course (not section) taught, up to four courses
b. Samples of major assignments, tests, projects
c. Statement of Teaching Philosophy
d. Course Commentary
e. Professional Development Plan

Supervisor’s Evaluation of Teaching
• Most schools have some sort of process where a Dean,
Department Head, Director, or similarly positioned person
evaluates faculty performance.
• One part of this process is typically the classroom
observation.
• In-class observations for online courses/instructors are
inherently different from F2F class observations.

Issues with Teaching Observations
Timeline:
• F2F – typically a single class period
• Online – as much or as little as you choose/allow
Familiarity:
• Most deans have taught F2F at one time or another
• Most deans have NOT taught online, although this is improving
Criteria:
• F2F – Most schools have established criteria for observations
• Online – Many do not have established criteria

Remaining Agenda
Review the Online Faculty Evaluation Rubric
• Sharable & Editable (Creative Commons BY-NC-SA 3.0)
Components included:
1. Pre-evaluation worksheet
2. Professional Development documentation
3. Expectations to be Evaluated
• Examples of Meeting Expectation
• Evaluator Comments

Creative Commons License
• Creative Commons BY-NC-SA 3.0
• You are free to:
• Share — copy and redistribute the material in any medium
or format
• Adapt — remix, transform, and build upon the material
• The licensor cannot revoke these freedoms as long as you
follow the license terms.

Under the following terms:
Attribution — You must give appropriate credit, provide a link to the
license, and indicate if changes were made. You may do so in any
reasonable manner, but not in any way that suggests the licensor endorses
you or your use.
NonCommercial — You may not use the material for commercial
purposes.
ShareAlike — If you remix, transform, or build upon the material, you
must distribute your contributions under the same license as the
original.
No additional restrictions — You may not apply legal terms or technological
measures that legally restrict others from doing anything the license permits.

Quality of
Learning
Quality of
Instruction
Quality of
Course Design
Learning Assessment
Measuring the Major Components

Evaluation System Components
• Student Evaluation of Instruction
• Faculty Self-Evaluation
• Professional Development Plan
• Faculty Peer Review
• Faculty Portfolio
• Supervisor’s Evaluation of Faculty
Usually some combination of the above components
Shared rubric could
be used for any of
these pieces.

Poll Questions – show of hands
Choose each statement that is true:
1. My school has established a process for online classroom observation to
evaluate the quality of online teaching.
2. Our deans and/or other academic leaders have the online education
experience and knowledge to effectively evaluate the quality of online
teaching.
3. My school uses Quality Matters or a similar process/rubric for reviewing
online course design.
4. My school has a formal process of faculty peer review of online course
design.
5. My school has a formal process of faculty peer review for online teaching.

Pre-Evaluation Worksheet for Online Faculty Evaluation
Faculty name
Evaluator name
Course to be
evaluated
Term/Semester
Yes No Item:
Have you previously taught this course in the online format?
Have you previously had an administrative evaluation completed for this course, at this college?
Are you the primary course designer for this course? If not, indicate designer's name below.
Have you made significant design changes to this course during the past year?
Additional comments:

Sources of Instructional Content
“X” if Yes Item: (check all that apply)
Traditional printed textbook – insert ISBN here:
Textbook publisher's e-text or “e-pack” (add link if possible)
Textbook publisher's website (add link to site)
Independent e-book (add link if possible)
Open Educational Resources (add link if possible)
Multimedia components available on the Internet
Instructor-owned instructional assets (describe below)
College-owned instructional assets (describe below)
Commercially-available instructional assets (CD-ROM, etc.- add link if possible)
Instructor's or departmental website, blog, wiki, etc. (add links to sites)
Other – indicate nature of other resources in comments section below.

Instructional Communication Tools
Inside
LMS
Outside
LMS
No Item: (NOTE: Add URLS for any tools used outside course)
Discussion forums
Chat room
Web conferencing tool
Instant messaging or pager
Assignments folder feedback comments
Quiz feedback comments
Email
Blog or wiki
Social networks or other websites
Other – indicate nature of other tools in comments section below.

End of Pre-Eval
Use the following space to communicate anything else that would be beneficial to
you or the reviewer prior to the teaching evaluation.
Adapted from: Lake Superior College Online Faculty Pre-evaluation Worksheet

Online Faculty Evaluation Rubric

Basic Info
Professional Development:
Is there documentation on file indicating the following?
Faculty name
Evaluator name
Course evaluated
Term/Semester
Start/End dates
Yes No Faculty participation in the following:
Training sessions for faculty who are new to online teaching
Online faculty peer review process
Web accessibility training
Online faculty workshops, webinars, conferences

Evaluation Grading Scale:
A = Exceeds expectations
B = Meets expectations
C = Will meet expectations with minor changes
D = Falls short of expectations

Category 1: Basic Expectations for Online Courses
Grade Expectation Examples of Meeting Expectation
Evaluator
Comments
1A: Course syllabus
information matches
approved college
curriculum.
1A: Syllabus will include, but not be limited to, these items:
1) Learning outcomes listed in course match official outcomes.
2) Course description matches official description.
3) Disabilities statement and other standard curriculum language are included.
1B: All required and optional
classroom resources are
clearly identified with
information about access.
1B: Syllabus or introductory information should include:
1) textbook information, including ISBN
2) lab kits, or similar resources
3) equipment needed by students
4) software or other resources
1C: Course time line is
clearly identified and easy to
find; with due dates, and
appropriate windows of
opportunity.
1C: Students often choose online learning because they seek time flexibility in
their schedule management.
1) The "window of opportunity" refers to the number of days that a student may
complete an assignment/quiz/etc. A quiz period that starts on Monday and
closes on Thursday is a four day window of opportunity.
2) Generally speaking, windows of opportunity from 3-7 days are reasonable.
3) Periodic due dates (for example, weekly), are important as many online
students have been found to not keep up with coursework when time line is left
wide open.

Category 1: Continued
Evaluator
Comments
1D: Expectations for
faculty response times
are clearly articulated for
all major course
components.
1D: Response time, grading turnaround time, or feedback frequencies are clearly
articulated to students for the following, if used in course:
1) email responses
2) quiz grading/feedback (continued next page)
3) assignment grading/feedback
4) discussion forum postings, etc.
1E: Instructor-prepared
course materials are
accurate, professional,
and consistent.
1E: Expectations include:
1) Good grammar
2) No obvious errors, including typos
3) Language level is appropriate for the audience
4) Consistent look and feel on content pages and syllabus
5) “Instructor voice,” in text or multimedia, is appropriate.
1F: Instructor provides
and maintains a
welcoming, safe, and
positive learning
environment.
1F: Examples include:
1) Clearly state the types of behavior that will not be tolerated in the online class.
2) Post and enforce rules for netiquette.
3) Maintain a safe learning environment by:
a) focusing on academic achievement, b) maintaining high standards,
c) fostering positive relationships between students and faculty,
d) reducing negativity, and e) resolving conflict

Category 2: Learning Experiences/Opportunities
Evaluator
Comments
2A: Has designed learning
opportunities that
encourage higher order
thinking skills such as
analysis, evaluation,
synthesis, problem solving,
and critical thinking.
2A: Non-exclusive list of possibilities:
1) scenario-based problems
2) Case studies and/or simulations/games
3) Peer tutoring, 4 Journaling
5) Exercises in diagramming or visualizations
6) Cooperative learning in groups
2B: Instructor uses
discussion forums
effectively.
2B: Student discussions should serve to:
1) stimulate thinking, 2) challenge attitudes and beliefs
3) develop interpersonal skills
4) investigate questions without simple answers
2C: Learning opportunities
are sufficient for students to
achieve all of the expected
course outcomes.
2C: Significant learning opportunities/requirements:
1) are correlated to specific course learning outcomes.
2) are not tangential to course learning outcomes.

Evaluator
Comments
2D: Learning experiences
are current, relevant, and
presented in a logical
sequence.
2D: Smaller “chunks” of content or modules tend to enhance student learning.
Content should be kept up-to-date (including links). When possible, course content
in latter part of course should build on content from earlier parts. Possibilities for
chunking include: by module, by week, by chapter, by topic
2E: Learning
opportunities enable
students to engage in a
variety of activities that
promote practice and a
transfer of skills.
2E: For example:
1) Learners can sometimes choose among available options for activities that are
relevant to them.
2) Variety of learning opportunities may allow students to engage with course
content in more than one modality.
Additional comments on Category 2:

Category 3: Activity and Interactivity
Evaluator
Comments
3A: Class introductions and
start-up procedures are
appropriate for instructor
and students.
3A: Getting off to a good start is crucial in eLearning:
1) Start building a sense of community among faculty and students
during the 1st week.
2) Consider alternate strategies for course introductions (including
audio/video) to build strong sense of instructor presence.
3) Create a quiz on key items from the syllabus and/or key services
available to online students.
3B: Encourages students to
be active participants in
class.
3B: Examples include:
1) Require frequent contributions to the class by all students.
2) Prompt students to share their experiences and knowledge with
other students.
3) Vigilantly track non-attendance or non-participation by lagging
students. Contact students by email or send note to advisor for
students not participating.
4) Add links to relevant and timely news articles and have students do
the same.

Evaluator
Comments
3C: Instructor
communicates with
students in a clear and
effective manner.
3C: Consider the following:
1) Be professional in your all communications.
2) Double check spelling and factual accuracy of news postings,
discussion postings, emails, assignments, quizzes, etc.
3) Provide written feedback on quizzes/assignments beyond a simple
“good job” or “needs work.”
4) Avoid jargon as much as possible.
3D: Instructor
communicates with
students in a timely
manner.
3D: Possibilities
1) Challenge yourself to meet or exceed the feedback timing
expectations that you have set for the class.
2) Take sanity breaks, but always let students know when they can
expect to hear from you.
3) Intelligent agents (in D2L) can be effective ways of providing timely
feedback on quizzes, dropbox assignments, etc.

Category 3: Continued again
Evaluator
Comments
3E: Instructor has created
multiple opportunities for
student-to-student
interactions.
3E: Through various online communication tools, require or encourage
students to interact with others on a regular basis. Possible tools:
1) discussion boards
2) web conferencing
3) phone conferencing (or PC)
4) personal profile pages
5) email
6) social networking
3F: Students are required to
engage in active learning
exercises.
3F: Some examples include:
1) Create course activities that take the students away from the
computer screen to accomplish certain tasks or achieve goals toward
course outcomes.
2) Where appropriate, have students engage with community or
industry members for course-related work.
3) Create assignments that encourage use of the college library,
tutoring center, or other support services.

Category 4: Evaluations, Assessments, and Grading
Evaluator
Comments
4A: Grading policy is easy
to locate, easy to
understand, and
complete.
4A: Instructor’s grading policy:
1) Should always be available in the course shell and in an easy-to-find
location
2) Language used to describe grade determination should be easy to
understand.
3) All components of the policy for determining student grades should be
located in one place.
4B: Utilizes multiple
assessment methods as
required of all college
courses.
4B: Consider a mixture of various options for graded work, including, but
not limited to:
1) quizzes or exams, 2) writing assignments, 3) portfolios
4) lab reports, 5) discussion postings,
6) event participation and analysis, 7) problem sets,
8) oral presentations, 9) debates,
10) simulations, 11) case studies

Evaluator
Comments
4C: Utilizes the LMS grade
book for students to track
their progress through the
course.
4C: The LMS grade book is essential since it provides a password-
protected, one-stop shop for students to view all assigned grades.
4D: Take appropriate steps
to mitigate security,
honesty, and/or
authentication concerns
regarding student
assessments.
4D: Examples might include:
1) Proctored exams
2) Timed quizzes/exams
3) Lock-down browser
4) Document submission via assignments folder
5) Plagiarism detection
6) Original writing assignments each term

Category 5: Other Faculty Expectations
5A: Research Completed or in Progress
Comments:
5B: Committee work or campus involvement
Comments:
5C: Community service
Comments:
5D: Service to students (non-teaching)
Comments:

Finding the Rubric
T&L Community Post
Evaluating The Quality Of Online Teaching - Webinar
Archive
bit.ly/dahlrubric

Evaluating the Quality of Online Teaching

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Evaluating the Quality of Online Teaching

Ähnlich wie Evaluating the Quality of Online Teaching (20)

Mehr von D2L Barry

Mehr von D2L Barry (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Evaluating the Quality of Online Teaching