Master Class slides from Systems Centre at the University of Bristol 21st May 2014.

1. Anthony S. Bryk
Master Class, University of Bristol

2. 2
Triple Aims of Educational Improvement
EFFICIENCY
EFFECTIVENESS
ENGAGEMENT
Be0er
Use
of
Resources
Ambi<ous
Learning
For
All
Students
More
Relevance

3. How We Work Now: Tower of Babel Problem
3

4. The Educational R&D Problem
• Accelerate Improvement Efforts
• Aim for Quality, Reliably at Scale
4

5. How We Are Working on This
• Analogical Scavengers—The Gawande
Inspiration
• Learning by Doing—Can we actually
make the ideas work?
• Engaging a Larger Community

6. An Inspiration: Improvement Science in Healthcare
Protecting 5 Million from Harm,
Saving 100,000 Lives

7. 7
We can
accomplish more
together, than
even the best of us
can do alone.
Complex systems
problems that we
now seek to solve
Power of
Networks

8. Networked Improvement Communities:
What are they?
Integrating Two Big Ideas:
• The discipline of Improvement Science
joined to
• The Power of Networks
Accelera'ng
Learning
in
and
through
Prac'ce
to
Improve

9. Six Principles Guide the Work
(plus useful tools to scaffold the activity)
9
Taken Together:
• Disciplined Inquiry
• Rudiments a scientific community
• Aim: systematic practice improvement

10. I.
Problem-­‐
&
User-­‐Centered
• What
is
the
specific
problem
we’re
trying
to
solve?
• What
we
tend
to
do
now:
a
general
issue
comes
into
view
and
we
jump
on
solu<ons

11. 60-­‐70%
Students
assigned
to
developmental
math
course.
80%
Percent
of
these
students
that
never
get
past
this
gate.
500,000
students
in
every
cohort
will
never
complete
college
math
requirement.
11
The
Problem

12. A
Solu<on
Framework:
Integrated
Pathways
12
Through
college-­‐level
sta5s5cs
“To-and-through” college-level
quantitative reasoning
Two 1-year pathways “to and through college math”
1
2

13. II.
Varia5on
in
Performance
is
the
problem
to
solve
• “What
Works”
is
typically
the
wrong
ques<on
• Real
Issue:
Quality
Improvement
Ques<on
“How
to
advance
effec<veness
among
diverse
faculty
engaging
varied
popula<ons
of
students
and
working
in
different
organiza<onal
contexts?”
• Goal:
Achieve
efficacy
with
reliability
at
scale

14.
TraditionalSequenceStatway
Effects: Time to Complete a College Level Math Course
1
Year
2
Years
Triple the
success
rate in half
the time.
6%
51%
15%

15. What is Next?
• Normal Course of Events: “It Works”
– Tout success
– Publish results
– Hope others pick this up
– Go onto our next project

16. Varia<on
in
Pathways
Success
Rates
by
College
(n=19)
16
1
23
4
5
6
7
8
9
11
1213
14
15
17
18
19
0%
50%
100%
0% 50% 100%
StatwayStudents
Non-Statway Matched Comparisons
No improvement
line
We also have a failure, why?
What can we learn?
Triple success rate
line

17. III.
See
the
System
to
Improve
it
• Put
simply:
It
is
hard
to
improve
what
we
do
not
fully
understand.

18. How Do We Heal Medicine? Atul Gawande April, 2012

20. Gawande’s Closing Observation
Making systems work is the great task of my
generation of physicians and scientists.
But I would go further and say that making systems
work — whether in healthcare, education, climate
change, making a pathway out of poverty — is the
great task of our generation as a whole.

21. The
Invisible
Complexity
Schooling
21
The
Invisible
Complexity
of
Schooling

22. 60-­‐70%
Students
assigned
to
developmental
math
course.
80%
Percent
of
these
students
that
never
get
past
this
gate.
500,000
students
in
every
cohort
will
never
complete
college
math
requirement.
22
Returning
to
The
Presen<ng
Problem

23. The
Orien<ng
Problem
Extraordinarily
high
failure
rates
among
students
assigned
to
developmental
math
instruc<on
Consolidate
the
courses
into
a
1-­‐year
pathway
Real
world
problems
from
sta<s<cs
as
the
organizer
Psycho-­‐social
interven<ons
aimed
at
“produc<ve
persistence”
Rapid
analy<cs
capacity
Faculty
development
Causal
Systems
Analysis:
Why
do
we
con<nue
to
get
the
outcomes
observed?
Primary
Causes
for
High
Failure
Rates
Organizing
Improvement
Hypotheses
event
???

24. Ins$tu$onal*structures*don’t*
support*student*success*
Students*are*not*engaged*
or*mo$vated*
The*course*material/content*is*
problema$c*
Instructors*lack*skills*and*beliefs*
that*students*can*succeed*
State*policy*does*not*
support*student*success*
Low*success*rates*
in*development*
math*func$on*as*a*
gatekeeper*to*
opportunity*
Students*lack*the*skills*to*
succeed*
Ineffec$ve*
advising*system*
Ineffec$ve*learning*
support*services*
Lack*of*social*$es*
to*each*other*and*
to*their*instructors*
Inaccurate*
placement*
High*dropAout*
rates*between*
courses*
Don’t*believe*
they*can*learn*
Math*and*tes$ng*
anxiety*
Poorly*prepared*
mathema$cally*
They*don’t*know*
how*to*study*math*
They*don’t*know*how*to*
“navigate”*the*college*world*
Not*interes$ng*or*
relevant**
Not*seen*as*
useful*
Text*is*inaccessible*
Does*not*leverage*what*
we*know*about*how*
students*learn*
Too*many*courses*in*
the*developmental*
sequence*
Few*opportuni$es*
to*learn*from*others*
Lack*knowledge*
of*learning*theory*
Weak*pedagogy*
Don’t*believe*
suppor$ng**student*
success*is*their*job*
Lower*reimbursement*for*
developmental*math*
Tradi$onal*transfer*
requirements*impede*
innova$on*
Arcane*curricular*
topics*create*
needless*hurdles*
Funding*based*on*
enrollment*rather*
than*outcomes*
OOen*taught*by*adjunct*
faculty**with*liPle*
professional*support*

25. The
Orien<ng
Problem
Embedded
literacy
and
language
barriers
Extraordinarily
high
failure
rates
among
students
assigned
to
developmental
math
instruc<on
Lose
large
#
of
students
at
the
transi<ons
Consolidate
the
courses
into
a
1-­‐year
pathway
Students
mindsets
undermine
success
Real
world
problems
from
sta<s<cs
as
the
organizer
Students
“gone”
before
we
know
it
Psycho-­‐social
interven<ons
aimed
at
“produc<ve
persistence”
Rapid
analy<cs
capacity
Course
material
and
instruc<on
are
not
engaging
Faculty
development
Analy<c
Summary
of
Causal
Systems
Analysis
Primary
Causes
for
High
Failure
Rates
Organizing
Improvement
Hypotheses
Eventually
leads
to
a
“Pathways
Strategy”

26. Pathways
Driver
Diagram:
Organizing a
Networked
Improvement
Community
Aim: increase
from 5% to
50%, students
achieving
college math
credit within
one year of
continuous
enrollment
Instructional
System: Organized
around productive
struggle, explicit
connections, and
deliberate practice.
Productive
Persistence:
Students develop
skills and maintain
positive mindsets
Language and
Literacy: Students
use language in
understanding
problems, reason
mathematically, and
communicate results
Advancing
Teaching: Effective
teaching within 2
years of
implementation
Reduce transitions +
assure enrollment
across semesters
Deliberate focus on
“Starting Strong”
Promote students’ ties
to peers, faculty,
pathway
Math that matters:
students see material
interesting, relevant
Enhance faculty’s
beliefs and relational
practices
Opening lessons engage
interest, assure early
success
Direct interventions to
influence student mindsets
Real-time data tracking on
student engagement
Detail supportive
classroom norms and social
connections
Professional development
on “Starting Strong”
A
Community
Explicates
its
Causal
Thinking:
A
Community
Explicates
its
Causal
Thinking:
A
Driver
Diagram
to
Organize
Its
Major
Improvement
Hypotheses

27. Pathways
Driver
Diagram:
Organizing a
Networked
Improvement
Community
Aim: increase
from 5% to
50%, students
achieving
college math
credit within
one year of
continuous
enrollment
Instructional
System: Organized
around productive
struggle, explicit
connections, and
deliberate practice.
Productive
Persistence:
Students develop
and maintain
positive mindsets
Language and
Literacy: Students
use language in
understanding
problems, reason
mathematically, and
communicate results
Advancing
Teaching: Effective
teaching within 2
years of
implementation
Reduce transitions +
assure enrollment
across semesters
Deliberate focus on
“Starting Strong”
Promote students’ ties
to peers, faculty,
pathway
Math that matters:
students see material
interesting, relevant
Enhance faculty’s
beliefs and relational
practices
Opening lessons engage
interest, assure early
success
Direct interventions to
influence student mindsets
Real-time data tracking on
student engagement
Detail supportive
classroom norms and social
connections
Professional development
on “Starting Strong”
Elabora<ng
Out
The
Driver
Diagram
Produc<ve
Persistence

28. IV.
You
cannot
improve
at
scale
what
you
cannot
measure
• Measureable
targets:
“Some
is
not
a
number;
soon
is
not
a
<me”-­‐-­‐Valued
outcome
measures
– But,
you
just
can
not
stand
at
the
end
of
the
line.
• We
need
process
measures
<ed
to
intermediate
targets.

29. Produc<ve
Persistence
Suppor<ve
social
rela<onships
Target:
How
do
we
measure
it?
Mindsets
about
the
value
of
math
Mindsets
about
poten<al
to
learn
math
Anxiety
Regula<on
Study
Skills
Conceptual
Task:
reduce
to
5
core
ideas
focus
on
underlying
malleable
causes
+
change
evidence
Prac5cal
Measurement:
reduce
900
items
to
26
“you
have
3
minutes”

30. V.
Accelerate
Improvement:
Embrace
Disciplined
Inquiry
• Policy
Romance
of
the
Silver
Bullet
– Move
quickly
to
large
scale
implementa<on,
but…
• We
typically
don’t
know
whether:
–
We
can
make
these
ideas
work
at
all;
–
We
have
capacity
and
will
to
execute
with
efficacy
at
scale.
• Instead,
a
DEED
orienta<on
– Quick,
minimally
intrusive,
an
empirical
warrant
–
Mantra:
Learn
Fast,
Fail
Fast,
Improve
Fast!

31. A System of Social Learning to Improve
Transla5onal
Research
Interven5ons
(Alpha
Labs)
Will
they
work
for
community
college
students,
and
if
so,
how?
Expert
Prac55oner
Knowledge
(Subnet)
Building
robust
clinical
knowledge
about
effec<ve
materials
and
instruc<onal
prac<ces.
Learning
from
Network
Data
(Hub
Analy5cs)
Learning
from
observed
variability.
Discerning
the
unseen.

32. 32
Transla5onal
Research
Interven5ons
(Alpha
Labs)
• Will
they
work
for
community
college
students,
and
if
so,
how?

33. Initial Alpha Lab: Mindset Intervention
• A carefully designed experimental intervention has
changed student mindsets.
• But just because an intervention can work in one
setting does not mean it will work in another.
• Need to engineer it to “fit” in instructional contexts.
– Conduct rapid R&D using DEED methodology.
– “Smell testing”
– 4 months from small-scale test to larger scale use.

34. Rapid Iterative DEED cycles
• Research-Practitioner Team
• Testing
– Small double-blind randomized
trial in Algebra course (n = 26)
– Larger double-blind
experiment (n = 288)
• Introduce to faculty network,
carefully study emerging results,
continue to revise, refine, and extend.
34
Roberta Carew,
Statway faculty
Valencia College

35. 35
Learning
from
Network
Data
(Hub
Analy5cs)
• Learning
from
observed
variability.
Discerning
the
“unseen.”

36. 36
!
1.Assessing Change: Initial Evidence of Efficacy
of Starting Strong Package

37. 2. Predictive Analytics—targeting support
(a simple at-risk indicator scoring 5 key items/item clusters-day 1)
37
%
of
who
failed
the
end-­‐of-­‐term
common
assessment

38. Connections to Stereotype Threat
12%
13%
14%
28%
40%
7%
11%
14%
50%
71%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Never
Hardly
Ever
Some<mes
Frequently
Always
Pathways
Dropout
All
students
Black
students
“How
oqen,
if
ever,
do
you
wonder:
‘Maybe
I
don't
belong
here?’”
N
=
714
math
students

39. 39
Expert
Prac55oner
Knowledge
(Subnet)
Building
robust
clinical
knowledge
about
effec<ve
instruc<onal
materials
and
prac<ces.

40.
PDSA Cycle: Rapid, Small Experimental Trials
PLAN
DO
ACT
STUDY
The
Three
Ques5ons:
• What
specifically
are
we
trying
to
accomplish?
• What
change
might
we
introduce?
• How
will
we
know
that
the
changes
are
an
improvement?

41. Improving Instructional Routines in Support
of Productive Persistence: PDSA Cycles
• Faculty routines and email scripts re: absent students
• Student group noticing routine
• Effective scaffolding for group roles (rich problems)
41

42. Sample Run Chart for a PDSA Cycle
(Student Group Noticing Routine)
0%#
20%#
40%#
60%#
80%#
100%#
120%#
1/14/13#
1/21/13#
1/28/13#
2/4/13#
2/11/13#
2/18/13#
2/25/13#
3/4/13#
3/11/13#
3/18/13#
3/25/13#
4/1/13#
4/8/13#
4/15/13#
4/22/13#
4/29/13#
Percent'of'Student'in'A/endance'
A/endance'(By'Day)'
Typical#a4endance#
Observed#a4endance#
n=44#
Median:#
0.85#
Median#

43. Developing a Quality Process Reliably at Scale
Develop
A
Change
Test
under
mul<ple
condi<ons
Test
under
increasingly
varied
condi<ons
Make
the
change
permanent
Ini5al
Hunches
System
Changes
1
school
1
administrator
5
schools
Many
administrators
En<re
ver<cal
team
A
more
diverse
group
of
administrators
District
Wide
All
administrators
Seeing
Task
Complexity
Seeing
Organiza<onal
Complexity
Learning
to
improve
feedback
conversa<ons
between
principals
and
new
teachers
PLAN
DO
ACT
STUDY

44. A
Developmental
Dynamic
Hunches
Theories
Ideas
Ini<a<ng
Resources
P D
SA
P D
SA
P D
SA
P D
SA
Moving
out
toward
More
diverse
condi<ons:
“factor
of
5
rule
of
thumb”
Aiming
for
Efficacy
with
Reliability
at
Scale

45. VI.
Accelerate
Improvement:
Tap
the
Power
of
Networks
• A
source
of
innova<on
– Dig
into
the
details:
what
worked,
how,
for
whom?
– Can
we
adap<vely
integrate
this
into
other
contexts?
• Mul<ple
fast
replica<on
– Can
we
make
this
happen
with
efficacy,
reliably
at
scale?
• Innova<on
diffusion—it
is
largely
about
who
is
connected
to
whom
and
what
they
think
and
do
A
Learning
Educa'onal
System

46. A
A
Improvement
Networks:
Accelerate
Learning
in
Prac<ce
for
Improvement
A
B
A
A
A
B
A
A
A
B
A
A
A
B
C
(Englebart,1994)

47. It is all about accelerating how we learn
in and through practice to improve.