Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Examining the Validity of the
International Association for K–12
Online Learning (iNACOL) Standards:
Improving K–12 Online...
Background
• Only two of the most commonly used standards have published any
research testing validity
o Virtual High Scho...
Background
• iNACOL National Standards for Quality Online Courses were
created in 2007 and updated in 2011
• Based on SREB...
Purpose
The ultimate purpose of the research was
to create a revised instrument for designers
based off the iNACOL standar...
Methodology
• Research followed a commonly used multi-step
approach of literature review, expert review, and real
world fi...
Phase One: Literature Review
• Compared the 52 areas of measurement of the iNACOL standards to
current K-12 and other rele...
Phase One Results
• Results showed that the elements were either fully or partially supported by
current research and lite...
Phase Two: Expert Review
• Designed a revised rubric through three rounds of expert review
• Content validity of the revis...
Phase Two Results
• Results created a revised 40 element rubric with a focus solely
on K-12 online course design
• Element...
Phase Three: Field Test
• Four pairs of K-12 online educators reviewed the revised rubric
against current K-12 online cour...
Phase Three Results
• Results of the overall rubric were below the acceptable percentage for
reliability (i.e. at least 80...
Conclusions
• Research was conducted in three phases, taking almost two years
to fully complete
• Phase one showed the iNA...
Implications
• Literature review provides minor support for the iNACOL
standards
• A revised and focused design rubric all...
Limitations
• There was a lack of K-12 online learning literature and
research, leading to the use of higher education lit...
Post-Study
Adelstein, D. & Barbour, M. (2018). Redesigning The iNACOL
Standards For K-12 Online Course Design. Journal of ...
Your
Questions
and
Comments
Associate Professor of Instructional Design
Touro University, California
mkbarbour@gmail.com
http://www.michaelbarbour.com
Nächste SlideShare
Wird geladen in …5
×

AERA 2019 - Examining the Validity of the International Association for K–12 Online Learning (iNACOL) Standards: Improving K–12 Online Course Design

59 Aufrufe

Veröffentlicht am

Adelstein, D., & Barbour, M. K. (2019, April). Examining the validity of the International Association for K–12 Online Learning (iNACOL) Standards: Improving K–12 online course design. A paper presentation at the annual meeting of the American Education Research Association, Toronto, ON.

Veröffentlicht in: Bildung
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

AERA 2019 - Examining the Validity of the International Association for K–12 Online Learning (iNACOL) Standards: Improving K–12 Online Course Design

  1. 1. Examining the Validity of the International Association for K–12 Online Learning (iNACOL) Standards: Improving K–12 Online Course Design David Adelstein Manager, Global Leadership Curriculum Development Tiffany & Co. Michael K. Barbour Associate Professor of Instructional Design Touro University California
  2. 2. Background • Only two of the most commonly used standards have published any research testing validity o Virtual High School (VHS) o Quality Matters (QM) • VHS standards were developed in the early days of K-12 online learning (late 1990s) • QM has significant research, however, it is primarily focused on higher education and also proprietary • Most schools, institutions, and programs look for free, non- proprietary standards o International Association for K-12 Online Learning (iNACOL) standards are starting to be used by several states
  3. 3. Background • iNACOL National Standards for Quality Online Courses were created in 2007 and updated in 2011 • Based on SREB and iNACOL’s participation in the Partnership for Twenty-First Century Skills Initiative • The standards are set up as a design instrument on a 0-4 point scale • Five areas of content with 52 unique areas of measurement
  4. 4. Purpose The ultimate purpose of the research was to create a revised instrument for designers based off the iNACOL standards, with a focus strictly on design. The desire was for the revised instrument to improve the design process and the structure of K-12 online courses.
  5. 5. Methodology • Research followed a commonly used multi-step approach of literature review, expert review, and real world field test o (Alad-wani & Palvia, 2002; Dray et al., 2011; Fitzpatrick, 1983; Gandek & Ware, 1998; Haynes et al., 1995; Legon & Runyon, 2007; Roblyer & Wiencke 2003; Stellmack et al., 2009; Taggart et al., 2001; Thaler et al., 2009; Walker & Fraser, 2005; Yang et al., 2013) 1. Literature Review: Content validity of iNACOL standards 2. Expert Review: Content validity of the revised rubric 3. Field Test: Inter-rater reliability of the revised rubric
  6. 6. Phase One: Literature Review • Compared the 52 areas of measurement of the iNACOL standards to current K-12 and other relevant online learning research • Examined the content validity of the standards (Haynes, Richard, & Kubany, 1995; Fitzpatrick, 1983) o Similar to the process that Ferdig, Cavanaugh, DiPietro, Black, and Dawson (2009) undertook with the iNACOL National Standards for Quality Online Teaching. • Over one year was spent compiling data through Wayne State University’s library and subscribed databases • K-12 literature was primarily used, with higher education and other relevant literature supplemented in where appropriate
  7. 7. Phase One Results • Results showed that the elements were either fully or partially supported by current research and literature • It was noted that student motivation were not directly addressed in the standards o Motivation is integral to education (McCombs & Vakili, 2005; Chen & Jang, 2010) • Traditional journal length constrains limited the depth allowed to each element • The manuscript was published in the online Journal of Online Learning Research Adelstein, D., & Barbour, M. K. (2016). Building better courses: Examining the content validity of the iNACOL national standards for quality online courses. Journal of Online Learning Research, 2(1), 41-73. Retrieved from http://www.editlib.org/p/171515)
  8. 8. Phase Two: Expert Review • Designed a revised rubric through three rounds of expert review • Content validity of the revised rubric (Roblyer & Wiencke 2003; Taggart, Phifer, Nixon, & Wood, 2001) • Eight experts, two from each of the following roles: o Researcher/evaluator, online educator, online administrator, online course designer • Multiple years of experience in area of expertise required, as well as a recommendation • Round one – Each element and suggestion was reviewed using a 1-3 Likert scale, plus expert suggestions • Round two – Experts were asked to mark each poorly rated element as (K)eep, (D)elete, (C)ombine, or (R)evise • Round three – Google Hangouts to discuss all ratings and reach consensus
  9. 9. Phase Two Results • Results created a revised 40 element rubric with a focus solely on K-12 online course design • Elements were eliminated if they did not pertain to online course design • It is recommended to increase the face-to-face discussion time, which is where significant refinement took place Adelstein, D., & Barbour, M. K. (2017). Improving the K-12 online course design review process: Experts weigh in on iNACOL National Standards for Quality Online Courses. International Review of Research in Open and Distance Learning, 18(3). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2800
  10. 10. Phase Three: Field Test • Four pairs of K-12 online educators reviewed the revised rubric against current K-12 online courses • Inter-rater reliability of the revised rubric o Taggart, Phifer, Nixon, & Wood, 2001; Legon & Runyon, 2007) • Recruitment through: o Edgenuity, Michigan Virtual University, Association for Educational Communication & Technology, Huron Valley Schools, Dissertation Advisor • K-12 online teaching experience required for reviewers • A sample course was sent to each reviewer and discussed via Google Hangouts • Each pair was assigned five online courses from two different content providers, which were reviewed individually using a 1-3 Likert scale
  11. 11. Phase Three Results • Results of the overall rubric were below the acceptable percentage for reliability (i.e. at least 80% exact agreement, (Neuendorf, 2002)) • However, numerous individual elements considered reliable • The small number of reviewers and courses reviewed was limiting to the research • Unable to use kappa coefficient, data shared was shared through percentage agreement Adelstein, D., & Barbour, M. K. (2016). Redesigning design: Field testing a revised design rubric based of iNACOL quality course standards. International Journal of E-Learning & Distance Education, 31(2). Retrieved from http://www.ijede.ca/index.php/jde/article/view/976
  12. 12. Conclusions • Research was conducted in three phases, taking almost two years to fully complete • Phase one showed the iNACOL standards aligned with current literature, but not necessarily K-12 research • Phase two created a revised rubric with a focus solely on design by combining, deleting and revising the iNACOL standards • Phase three tested the revised rubric against current K-12 online courses, with overall results not meeting the reliability threshold. However, 10 elements meet the 80% threshold and 15 meet the 70% threshold for exact match. If combined with one point difference, the number greatly increases
  13. 13. Implications • Literature review provides minor support for the iNACOL standards • A revised and focused design rubric allows for a streamlined creation process, which in turn can save time and money • Researchers and designers have a stronger platform from which to build from • Allows educators to judge their current courses with an instrument specifically created to look solely at design • Ultimately, better design standards impact student engagement and comprehension
  14. 14. Limitations • There was a lack of K-12 online learning literature and research, leading to the use of higher education literature to supplement • The expert panel was limited by volunteers and time, with the video conference limited by schedules • The field test was impacted by the number of reviewers and number of courses offered up for review o This ultimately hampered the ability to use kappa and other statistical procedures
  15. 15. Post-Study Adelstein, D. & Barbour, M. (2018). Redesigning The iNACOL Standards For K-12 Online Course Design. Journal of Online Learning Research, 4(3), 233-261. Retrieved from https://www.learntechlib.org/primary/p/178229/ • QM and the Virtual Learning Leadership Alliance are collaborating to refresh the iNACOL standards under the banner of “THE National Standards for Quality Online Learning” o https://www.nsqol.org/
  16. 16. Your Questions and Comments
  17. 17. Associate Professor of Instructional Design Touro University, California mkbarbour@gmail.com http://www.michaelbarbour.com

×