Predictors of Success: Linking Student Achievement to School and Educator Successes through Professional Learning This study show how some schools have seen a dramatic increase in student achievement after developing a strong, online professional learning program.

1. Predictors of Success: Linking Student Achievement to School and Educator
Successes through On-Demand, Computer-Based Professional Learning
Steven H. Shaha, PhD, DBA Professor, Center for Public Policy & Administration
Independent Program Evaluator
Abstract
Early research has begun to investigate the potential
Year-over-year changes in student achievement were benefits of computer-based and online professional
analyzed for 734 schools selected due to utilization learning (cf. Farnsworth et al. 2002; Lewis et al. 2003;
history for online (i.e. on-demand) and computer-based Magidin et al. 2012; Rienties et al. 2013). However, at
professional learning applications. Results showed that least one recent study cited a “dearth of scientific
schools with higher engagement in on-demand profes- research … on whether changes in teachers’ knowl-
sional learning by educators significantly outperformed edge and instructional practices resulting from online
their lower engagement counterparts in measures of professional learning are linked to changes in students’
quantity and quality of utilization, participation, and knowledge and practices” (Masters et al. 2012). We
engagement. Higher engagement schools also had sig- also found that very few studies have investigated the
nificantly greater gains in student achievement as mea- importance of leadership’s engagement in ensuring
sured by percentages of students performing at profi- the efficacy of professional learning, regardless of the
cient or advanced levels. Higher engagement schools mode of delivery (cf. Sebastian and Allensworth, 2012).
also outperformed their lower engagement counter-
parts for gains in four key school- and educator-related Finally, there are education-related metrics that have
measures: teacher retention, dropout rates, student dis- societal implications—metrics reflecting factors that are
cipline issues, and rates of students with college-related critical in assessing the success of educators, schools,
goals. Conclusions were that higher levels of utilization, and education as a societal institution. Despite the
engagement, and active use are correlated with higher importance of these assessments, we found virtually
student achievement and successes for both educators no connection established in research between teacher
and the schools in which they operate. participation in professional learning and improvements
in student-related measures of non-test performance
I. Introduction and Overview beyond at-risk preschool children, such as dropout
rates or disciplinary rates (Wasik and Hindman 2011).
Educators need high-impact help to keep their skills Even such seemingly simple correlations as improve-
well honed and to maintain their educational effec- ments in teacher retention (cf. Lathan and Vogt 2007)
tiveness. Yet the body of literature linking professional or teacher attitudes and perceptions (cf. Guskey 2002)
learning and development to gains in student perfor- resulting from professional learning are unaccountably
mance and teacher-related outcomes arguably remains minimal in the research literature.
inadequate (cf. Shaha et al. 2004). Some studies have
shown that professional learning can lead to improved Taken as a whole, research indicates that providing edu-
student performance (cf. Garet et al. 2001; Desimone et cators with readily accessible learning opportunities has
al. 2002; Shaha et al. 2004; Meiers and Ingvarson 2005; a substantive and favorable impact. We relabeled this
Buczynski and Hansen 2010; Avalos 2011). Yet, it seems approach as “on-demand learning” to accentuate why
clear from research that the more active an educator’s it is effective instead of how it is delivered. One reason
participation is beyond traditional, passive profession- for the effectiveness of the on-demand approach is that
al learning—such as sitting in a workshop or passively educators learn about what they are most interested
watching a video alone—the greater the impact of par- in, or most in need of, at the time of interest or need,
ticipation (cf. Garet et al. 2002; Desimone et al. 2002; rather than when it fits sequentially into any prescrip-
King 2002; Darling-Hammond 2004; Santagata, 2009). tive curriculum.
In addition, roadblocks to teacher participation in pro-
fessional learning and implementation of skills learned Thus timeliness of learning, synchronized with inter-
have been cited in recent research and remain import- est and need, mean that educators benefit from what
ant barriers to impact (cf. Buczynski and Hansen 2010). could be labeled “just-in-time learning.” This concept
1
Steven H. Shaha, PhD, DBA, Professor and/or Lecturer at University of Utah, Zayed University (UAE) and Harvard University

2. mirrors the business successes achieved in other in- exclusion criteria were implemented reflecting state or
dustries through just-in-time approaches—or JIT—as a district, rural or urban areas, school size or any other
science for maximizing efficiency and profitability while variable associated with school or student demograph-
minimizing costs associated with doing business (cf. ics. Year-over-year improvement was computed as the
Bongiorni 2004; Hirano and Makota 2006; Ohno 1988; percentage change (i.e. gain or loss) for each metric
Ruffa 2008). Education benefits from sciences proven (i.e. [2011-2010]/2010]).
in industry further refine educator efficacy and its im-
pact on students. In this case, the near immediate and Educator Engagement: Levels of educator utilization,
personally customized benefits of online accessibility participation, and engagement were loaded directly
provide for JIT educator learning: on-demand profes- from the on-demand applications as captured automat-
sional learning. ically and transparently to users, thus ensuring objec-
tivity and accuracy, representing 27 metrics (further
We undertook the designing and execution of an evalu- explained in Results).
ation study of on-demand professional learning in order
to answer a crucial set of inquiries regarding its impact Student Performance: Performance data were gath-
on students, educators, and schools. The driving re- ered from publically available sources. In order to
search questions therefore were whether schools with enable analyses across states with varying testing and
higher utilization or engagement experienced greater scoring approaches, data analyzed reflected the per-
impact than those of lower utilization or engagement centage of students classified as either proficient or
for the following: advanced on whatever approach applied within any
state or appropriate governing body. Data were limited
• ducator engagement in other metrics or areas of
E to reading and math only (2 metrics), as these were the
utilization, participation, and engagement only two areas of measurement consistent across all
• Student performance states. Sixteen schools were excluded from analysis for
• Other measures of school- and educator-related success inadequate data regarding student performance.
Additional questions to be addressed: School-and Educator-Related Measures: A set of four
• s viewing professional learning alone as strong a
I metrics were gathered by structured phone interviews
predictor of success and impact as other metrics of with each school, including rates for teacher retention,
educator utilization, participation, and engagement? dropouts, student discipline, and the number of stu-
• s there a model or framework for predicting maxi-
I dents reported as being college bound. Year-over-year
mum impact from educator utilization, participation, improvement was computed as the percentage change
and engagement in on-demand or computer-based (i.e. gain or loss) in the rate or percentage for each metric.
teacher development applications?
The final study included 734 schools in 211 districts
within 39 states. Schools were next classified into
II. Methods quartiles reflecting their average minutes of use by
educator as a proxy for relative utilization or engage-
A retrospective study was undertaken leveraging a ment rates. To make analyses and conclusions more
sample of 750 schools reflecting high engagement in straightforward for execution and interpretation,
on-demand professional learning (i.e. PD 360® and analyses contrasted only the top and bottom quar-
Observation 360®, School Improvement Network, tiles: the highest quartile of schools (higher engage-
Salt Lake City, UT). Data included the 2009-2010 and ment schools) versus the lowest quartile schools
2010-2011 school years, categorized during analyses (lower engagement schools).
as pre versus post. Schools were selected for inclusion
from the universe of on-demand users based upon All analyses were conducted by an independent,
their active use as measured by minutes of viewing doctoral prepared, internationally recognized stat-
professional learning videos, and minimum criterion for istician and program evaluator, using SPSS version
inclusion was set at a minimum average of 90 minutes 17.0 or higher, and SAS for confirmatory purposes as
per educator within any school, and all schools meeting needed or appropriate.
those minimum criteria were included. No inclusion or
2

3. higher engagement schools (p.001, see Figure 2),
III. Results Initial Interpretations illustrating that comparative gains were great against
lower engagement schools for measures reflecting
Viewed collectively, results showed that higher en- more active participation and engagement by educa-
gagement schools outperformed their lower engage- tors. Similarly, the magnitude of difference in teacher
ment counterparts in every area of measurement: observations performed by leadership was 63.8%
higher for higher engagement schools (p.001, see
Educator Engagement: Higher engagement schools Figure 3), illustrating that active engagement by
outperformed their lower engagement counterparts leadership was greater versus in lower engagement
in 15 of the 27 metrics of utilization, participation, and schools, as well.
engagement, and performed equally well or better in
the remaining 12 metrics, although none significantly
(p0.05). Higher engagement schools were significant-
ly higher in measures of implementation, accountability,
and oversight, or those metrics most appropriately as-
cribed to leaders and their role in successful execution
of the on-demand or computer-based, educator-learn-
ing program.
Metrics reflecting greater gains for higher engagement
schools included, for example, number of focus objec-
tives set up, observations performed, percent of regis-
tered users, and percent of users in communities. High- Figure 1. Comparative difference average minutes viewed per educator
er engagement schools performed significantly higher
in utilization metrics and measures of more passive
participation, including minutes viewed, forums viewed, Forums Posted
23.1
programs viewed, segments viewed, and links viewed. 25.0
In metrics classified as measures of engagement, higher 20.0
15.0 13.7
engagement schools outperformed the lower engage-
10.0
ment counterparts in metrics reflecting more active
5.0
engagement, including follow-up questions answered,
0.0
reflection questions answered, focus objectives set up,
Lower Engagement Higherer Engagement
forums posted, downloaded files, uploaded files, and Schools Schools
participation in communities.
Figure 2. Comparative difference in forums posted per educator
Regarding the degree of comparative impact of
minutes viewed versus the other engagement met-
rics, higher engagement schools had 4.3% greater
Teacher Observations Performed
minutes viewed (p.01), a significant and important 40.0
33.7
utilization-related advantage (see Figure 1). This was 35.0
30.0
expected, since the assignment of schools to higher
25.0 20.6
and lower engagement categories was based upon 20.0
their comparative measures of viewing. 15.0
10.0
5.0
However, more revealing were the comparative 0.0
gains for higher versus lower engagement schools, Lower Engagement Higher Engagement
Schools Schools
which were substantially and significantly greater for
utilization-related metrics, reflecting great levels of
active participation and engagement beyond simple Figure 3. Comparative difference in observations performed per educator
viewing. For example, we noted the magnitude of
difference in forums posted was 68.6% higher for
3

4. A complete view of the 15 measures for which high- While the lower engagement schools improved by an
er engagement schools outperformed their lower impressive 4.9% year over year (p.001), the higher en-
engagement counterparts is found in Table 1. For gagement schools improved by 18.0% (p.001), nearly
convenience in interpretation of the results, the 15 four times the rate of improvement comparatively.
metrics were categorized into logical groupings re-
flecting the apparent nature of the underlying con- In math, higher engagement schools not only closed
structs being measured. The grouping labeled Lead- the pre-existing performance gap, but significantly
ership, Implementation, and Accountability included surpassed the lower engagement schools year over
metrics reflecting program setup and active leader year (p.001, see Figure 5). Lower engagement schools
engagement. The Educator Utilization grouping in- did experience improvement from on-demand profes-
cluded metrics reflecting the more passive measures sional learning at 0.5% year over year (p.05). However,
of participation as contrasted with educator engage- the higher engagement schools improved by 18.9%
ment, for which the metrics reflected more active (p.001), over 30 times the rate of improvement com-
and productive participation, for example, beyond paratively. Interestingly, this rate of improvement very
viewing alone. nearly equaled the rate achieved in reading.
Table 1. omparative performance in measures of educator
C
participation as categorized Percent of Students Proficient or Advanced:
Reading
Higher Lower 68.0 66.6
Engagement Engagement Percent 66.0 63.5
Schools Schools Difference Difference 64.0 67.2
Leadership, Implementation, Accountability 62.0 Lower Engagement
60.0
Focus Objectives Set Up 130.1 29.2 100.9 345.5% 58.0
Schools
Observations Performed 3120.7 2149.0 971.7 45.2% 56.0 Higher Engagement
Percent Registered Users 87.8% 83.2% 0.0 5.5% 54.0 56.9
52.0
Schools
Percent of Users in Communities 43.0% 36.5% 6.5% 17.8%
50.0
Educator Utilization Pre Post
Minutes Viewed 359.9 80.6 279.3 346.5%
Forums Viewed 138.7 87.0 51.7 59.4%
Programs Viewed 588.7 223.5 365.2 163.4% Figure 4. Comparative gains in reading performance
Segments Viewed 2298.0 528.9 1769.1 334.5%
Links Viewed 12.2 10.6 1.6 15.1%
Educator Engagement Percent of Students Proficient or Advanced:
Follow-up Questions Answered 359.6 167.0 192.6 115.3%
Reflection Questions Answered 588.3 340.4 247.9 72.8%
Math
Focus Objectives Set Up 3120.7 2149.0 971.7 45.2% 72.0
69.5
Forums Posted 28.5 23.4 5.1 21.8% 70.0
68.0
Downloaded Files 45.2 35.4 9.8 27.7%
66.0 Lower Engagement
Uploaded Files 46.3 29.3 17.0 58.0%
64.0 62.7 63.0 Schools
Participation in Communities 43.0% 36.5% 6.5% 17.8%
62.0
60.0 58.4 Higher Engagement
58.0 Schools
56.0
Pre Post
The implication was that video viewing alone, or other
more passive metrics (e.g. % users registered), were
not as great of predictors or discriminators of educator Figure 5. Comparative gains in math performance
engagement as were measures of utilization reflecting
more active engagement. School-Related Engagements: Results revealed statis-
tically significant relationships between key metrics of
Student Performance: Higher engagement schools educator/school-related success and higher and more
collectively began at a significant performance disad- active utilization of the on-demand professional learn-
vantage in both reading (p.001) and math (p.001) in ing. Both higher and lower engagement school cohorts
terms of the percentage of students classified as either saw statistically significant gains in school-related
proficient or advanced. However, in reading, higher en- metrics. However, higher engagement schools, which
gagement schools successfully closed the performance were consistently those with higher utilization rates for
gap with the lower engagement schools (see Figure 4). on-demand professional learning, also achieved better
improvement year over year versus lower engagement
4

5. schools in every measure of educator- and school-relat- success, including teacher retention, student discipline,
ed success available, including the following: dropout rates, and the number of students reported as
college bound.
• 0.0% lower dropout rates (p.001) versus 4.9%
2
lower dropout rates for the lower engagement Additionally, video viewing alone was not as great an
schools (p.01), representing 4-times greater in gains indicator of student and educator- and school-relat-
(see Figure 6) ed gains as the other host of utilization, participation,
• .6% gain in rate of students with goals to attend col-
9 and engagement metrics. Performance on the other
lege (p.001) versus flat gains for lower engagement metrics of engagement far exceeded those found for
schools (p=ns), or 12-times the gains video viewing alone, generally by magnitudes of 10 to
• 3.2% lower rate for student discipline occurrences
3 20 times. Several interpretations might fit to explain
(p.001) versus 7.4% lower for the lower engagement the finding, perhaps best expressed as questions: Does
schools (p.01), greater than 4-times the gains video viewing alone possibly include multi-tasking by
• .8% higher teacher retention rates (p.001) versus
2 the participants who may be reading, updating grade
1.7% lower for the lower engagement schools (p.01), books, or emailing while videos are streaming? Do
nearly 2/3 greater gains metrics of more active participation reflect higher levels
of personal engagement, and therefore more active
learning and focus and higher likelihood of application
Dropout Rate of things learned? While other explanations may apply,
5.5 5.3
5.0 these data support conclusions that the gains achieved
5.0 for students, educators, and school-related impacts
5.1
4.5 Lower Engagement support leveraging professional learning programs that
Schools
4.0 go far beyond video watching alone.
4.1 Higher Engagement
3.5 Schools
Taken as a whole, results reveal significant predictive
3.0
Pre Post correlations between the quantity of educator utiliza-
tion, participation, and engagement with better student
Figure 6. Comparative gains/reductions in dropout rates
results and school-related outcomes. While correlation
cannot prove causation, the systematic and consistent
findings within these data clearly support a conclu-
IV. Discussion Conclusions sion that participation and engagement in this form of
teacher development resulted in the advantages and
Results substantiated significant and substantive gains found. It is intuitive that highly active and more
advantages to the use of on-demand and comput- frequent participation in professional learning should
er-based professional learning. Further, results clearly lead to educators more focused on critical behaviors
indicated that the more engaged the user is beyond and techniques that would help them teach better, and
video participation alone, the greater the impact of the help their students achieve more proficiency.
professional learning. All five research objectives were
achieved. Higher, more frequent on-demand activity, combined
with the perception of improved student success
Higher engagement schools—those with higher uti- before quantified, also resulted in the higher teacher
lization and participation—outperformed their lower retention rates observed, another indicator of educator
engagement counterparts in the majority of the metrics satisfaction and greater enthusiasm for teaching. While
analyzed, and never underperformed on any other met- it could be argued that retention is a prescient indicator
rics. Higher engagement schools experienced signifi- of improved success—better teachers stay—the oppo-
cantly greater gains for students in math and reading, site causal perspective is at least equally supported in
equaling or exceeding 18% gains over prior-year per- this study. In each measurement area, higher engage-
formance levels. For school-related measures, higher ment schools outperformed the corresponding lower
engagement schools experienced significantly great- engagement cohort, even when student performance
er gains in critical measures of educator and school began at comparatively lower levels. Thus, the data
support the conclusion that higher engagement in
5

6. on-demand learning results in higher teacher percep- Gus ey, Thomas R. “Professional Development and Teacher
k
Change.” Teachers and Teaching 8, no. 3 (08/01; 2012/11,
tions of their impact on students, contributing to higher
2002): 381-91.
satisfaction, and ultimately higher retention.
Hir no, Hiroyuki, and Furuya Makota, eds. JIT is flow: Practice and
a
Principles of Lean Manufacturing. ISBN 0-9712436-1-1 ed.: PCS
The next phase of research should include a repeat of Press, Inc., 2006.
these analyses for data reflecting the 2011-2012 school Kin , Kathleen P. “Identifying Success in Online Teacher Education
g
year. That may include a contrast of schools for which and Professional Development.” The Internet and Higher Edu-
cation 5, no. 3 (0, 2002): 231-46.
the 2011-2012 was the first year of participation versus
Lat am, N., and W. P. Vogt. “Do Professional Development Schools
h
the gains found in these year-to-year analyses for 2010
Reduce Teacher Attrition? Evidence from a Longitudinal Study
versus 2011. Additionally, new analyses might contrast of 1,000 Graduates.” Journal of Teacher Education 58, no. 2
the newly implemented schools for 2012 with a two- (2003): 153-67.
year history of gains for those included in these anal- Lews, V. K., S. H. Shaha, B. Farnsworth, L. Benson, and D. Bahr.
i
yses, thus adding the insight of multiple year-on-year “The Use of Assessment in Improving Technology-Based
Instruction Programs.” Journal of Instructional Psychology 30,
participation from 2010 through 2012 versus a first year
no. 2 (2003).
alone.
Magdin de Kramer, Raquel, Jessica Masters, Laura M. O’Dwyer,
i
Sheralyn Dash, and Michael Russell. “Relationship of Online
Future research might also include contrasts of gains Teacher Professional Development to Seventh-Grade Teachers’
with changes experienced in equivalent non-participat- and Students’ Knowledge and Practices in English Language
ing schools to ensure that the accomplishments found Arts.” The Teacher Educator 47, no. 3 (06/27; 2012/11, 2012):
236-59.
here reflect achievements above and beyond chance
Mas ers, Jessica, Raquel Magidin de Kramer, Laura O’Dwyer, Sher-
t
alone. This would be ideally executed with a study that
alyn Dash, and Michael Russell. “The Effects of Online Teacher
includes analyses of changes contrasting schools with Professional Development on Fourth Grade Students’ Knowl-
and without the on-demand or computer-based teach- edge and Practices in English Language Arts.” Journal of Tech-
er learning, thus controlling for important underlying nology and Teacher Education 20, no. 1 (January, 2012): 21-46.
differences such as socioeconomic mixes. Additional Mei rs, M. and L. Ingvarson. “Investigating the Links Between
e
Teacher Professional Development and Student Learning Out-
research might also include the contrasting metrics
comes.” Australian Council for Educational Research 1, (2005):
from higher utilization schools with non-participating
1-93.
schools from their respective districts, thus controlling Ohn , Taiichi, ed. Just-in-time for today and tomorrow. ISBN
o
for differences in demographics and socioeconomics. 0-915299-20-8 ed.: Productivity Press, 1988.
Rie ties, Bart, Natasa Brouwer, and Simon Lygo-Baker. “The Ef-
n
fects of Online Professional Development on Higher Education
Teachers’ Beliefs and Intentions Towards Learning Facilitation
References
and Technology.” Teaching and Teacher Education 29, no. 0 (1,
2013): 122-31.
Avaos, Beatrice. “Teacher Professional Development in Teaching
l
Ruf a, Stephen A. “Going Lean: How the Best Companies Apply
f
and Teacher Education Over Ten Years.” Teaching and Teacher
Lean Manufacturing Principles to Shatter Uncertainty, Drive In-
Education 27, no. 1 (1, 2011): 10-20.
novation, and Maximize Profits.” AMACOM (American Manage-
Bon iorni, Sara. “All in the Timing.” The Greater Baton Rouge Busi-
g
ment Association) (2008).
ness Report (July 19, 2004).
San agata, R. “Designing Video-Based Professional Development
t
Buc ynski, Sandy, and C. Bobbi Hansen. “Impact of Professional
z
for Mathematics Teachers in Low-Performing Schools.” Journal
Development on Teacher Practice: Uncovering Connections.”
of Teacher Education 60, no. 1 (2009): 38-51.
Teaching and Teacher Education 26, no. 3 (2010): 599-607.
Seb stian, J. and E. Allensworth. “The Influence of Principal Lead-
a
Daring-Hammond, L. “Standards, Accountability, and School Re-
l
ership on Classroom Instruction and Student Learning: A Study
form.” Teachers College Record 106, no. 6 (2004): 1047-85.
of Mediated Pathways to Learning.” University Council for
Desmone, L. M., A. C. Porter, M. S. Garet, K. S. Yoon, and B. F.
i
Educational Administration 10, (2012): 626-63.
Birman. “Effects of Professional Development on Teacher’s
Sha a, S. H., V. K. Lewis, T. J. O’Donnell, and D. H. Brown. “Evaluat-
h
Instruction: Results from a Three-Year Longitudinal Study.” Ed-
ing Professional Development: An Approach in Verifying Pro-
ucational Evaluation and Policy Analysis 24, no. 2(2002): 81-112.
gram Impact on Teachers and Students.” Journal of Research
Far sworth, B. and et al. “Preparing Tomorrow’s Teachers to use
n
in Professional Learning 1, no. 1 (2004): 1.
Technology: Learning and Attitudinal Impacts on Elementary
Students.” Journal of Instructional Psychology 29, no. 3(2002).
Gar t, Michael S., Andrew C. Porter, Laura Desimone, Beatrice
e
F. Birman, and Kwang Suk Yoon. “What Makes Professional
Development Effective? Results from a National Sample of
Teachers.” American Educational Research Journal 38, no. 4
(Winter 2001, 2001): 915.
6