1.
IMPROVING
DECISION
MAKING
IN
SMALL
SCHOOL
SYSTEMS:
AN
EXAMINATION
OF
DATA
LITERACY
AND
DATA
DASHBOARD
DESIGN
Client:
Dorothy
I.
Height
Community
Academy
Public
Charter
Schools
Project
Liaison:
Colin
Welch,
Data
Specialist,
Dorothy
I.
Height
Community
Academy
Public
Charter
Schools
Prepared
By:
Jennifer
Briones,
Alison
Friedman,
Isabel
Huston,
Emily
MacNeil,
and
Michael
Gaskins
May
5,
2014
2.
2
Table
of
Contents
Acknowledgements.........................................................................................................................................3
List
of
Acronyms..............................................................................................................................................4
Executive
Summary.........................................................................................................................................5
Project
Rationale.............................................................................................................................................6
Introduction ...............................................................................................................................................6
Data-‐Driven
Decision
Making.....................................................................................................................6
Dashboard
Creation ...................................................................................................................................6
Current
Data
Systems.................................................................................................................................7
Research
Questions....................................................................................................................................8
Background .....................................................................................................................................................8
Community
Academy
Public
Charter
Schools ............................................................................................8
Table
1:
CAPCS
Student
Population
by
Campus………………………..………………….…………………………...………...9
Accountability
and
CAPCS .........................................................................................................................9
Accountability
and
the
Need
for
Accessible
Data:
The
No
Child
Left
Behind
Act
of
2001 .....................10
Applied
Data-‐Driven
Decision
Making:
Turning
Data
into
Actionable
Knowledge..................................10
Figure
1:
Framework
for
Describing
Data-‐Driven
Decision
Making
in
Education………………………………....11
Factors
Affecting
Data-‐Driven
Decision
Making......................................................................................12
Overview
of
the
Study ..................................................................................................................................12
Phase
1:
Research-‐Informed
Prototype
Creation ....................................................................................13
Phase
2:
Data
Collection
with
Semi-‐Structured
Interviews......................................................................14
Figure
2:
Data
Should
Be
Used
to
Improve
Outcomes………………………………………….………………………...……15
Table
2:
CAPCS
Stakeholders
Optimistic
About
Data
Literacy……………………………………………………….……..16
Table
3:
Context
is
Crucial
to
a
Dashboard…………………………………..………………………………….………………....17
Table
4:
CAPCS
Stakeholders
Seek
Trend
Indicators
on
Dashboards……….…….……………………………………..18
Figure
3:
CAPCS
Stakeholders
Reveal
Most
Important
Data
Points………………………………………………………18
Phase
3:
Final
Dashboard
Prototype
Creation .........................................................................................19
Figure
4:
Sample
Final
Dashboard
Prototype……………………………………………………………………………………….20
Dashboard
Recommendation ..................................................................................................................20
Further
Recommendations
for
Dashboard
Use ............................................................................................20
Conclusion.....................................................................................................................................................22
Appendix
A:
Current
CAPCS
Dashboard........................................................................................................23
Appendix
B:
Board
Summary
Document ......................................................................................................29
Appendix
C:
Initial
Dashboard
Prototype......................................................................................................31
Appendix
D:
Final
Dashboard
Prototype.......................................................................................................33
Appendix
E:
Interview
Protocol
and
Script ...................................................................................................35
References ....................................................................................................................................................44
3.
3
Acknowledgements
We
would
like
to
extend
our
sincere
gratitude
to
the
following
individuals,
without
whom
we
would
not
have
been
able
to
complete
this
report:
Colin
Welch,
our
Dorothy
I.
Height
Community
Academy
Public
Charter
Schools
liaison,
for
guiding
us
through
the
dashboard
creation
process
and
connecting
us
with
multiple
stakeholders;
The
administration
and
management
staff
at
the
Dorothy
I.
Height
Community
Academy
Public
Charter
Schools,
for
providing
their
time
and
honest
feedback
during
interviews;
Professor
Yas
Nakib,
for
offering
advice
and
providing
us
with
resources
and
literature
to
write
this
report;
Megan
Hatch,
our
Research
Advisor,
for
guiding
us
throughout
the
research
and
report
writing
processes;
And
Professor
Elizabeth
Rigby,
for
providing
us
with
the
necessary
feedback,
information,
and
tools
to
work
with
CAPCS
and
write
this
report.
4.
4
List
of
Acronyms
ANet
-‐
The
Achievement
Network
CAPCS
-‐
Community
Academy
Public
Charter
Schools
DDDM
-‐
Data-‐driven
decision
making
ELL
-‐
English
Language
Learners
LEA
-‐
Local
education
agency
NCLB
-‐
The
No
Child
Left
Behind
Act
of
2001
OSSE
-‐
Office
of
the
State
Superintendent
for
Education
PCSB
-‐
Public
Charter
School
Board
PMF
-‐
Performance
Management
Framework
SPED
-‐
Special
Education
5.
5
Executive
Summary
The
Dorothy
I.
Height
Community
Academy
Public
Charter
Schools
(CAPCS)
form
a
charter
school
network
in
Washington,
DC
that
serves
grades
pre-‐kindergarten
through
six.
Like
many
schools,
CAPCS
uses
data-‐driven
decision
making
(DDDM)
to
track
progress
toward
goals,
determine
effective
instructional
strategies,
and
meet
accountability
requirements
set
by
local,
state,
and
federal
education
agencies.
CAPCS
desires
a
data
dashboard
that
can
be
utilized
universally
by
school
administrators,
central
office
staff,
and
the
Board
of
Trustees
to
aid
in
these
processes.
In
collaboration
with
CAPCS
and
under
the
advisement
of
Professor
Elizabeth
Rigby
and
Research
Advisor
Megan
Hatch,
we
developed
the
following
research
questions
to
guide
the
redesign
of
CAPCS’
current
dashboard:
1. What
are
the
current
best
practices
for
creating
dashboards?
2. How
should
CAPCS
visualize
data
for
use
in
making
decisions?
3. What
are
essential
contextual
factors
to
foster
implementation
of
data
dashboards?
To
address
these
questions,
we
conducted
research
to
inform
creation
of
an
initial
dashboard
prototype,
collected
feedback
from
relevant
CAPCS
stakeholders,
and
created
a
finalized
prototype
based
on
that
feedback.
We
also
crafted
this
report,
which
includes
analysis
of
stakeholder
feedback
and
recommendations
for
the
use
of
the
revised
dashboard.
Initial
research
for
this
project
examined
the
concepts
of
DDDM
and
data
literacy
in
an
educational
context
to
gain
an
understanding
of
how
schools
successfully
implement
these
processes
and
integrate
them
into
staff
workflow.
We
found
that
developing
a
common
culture
of
data
literacy
and
buy-‐
in
for
DDDM
is
perhaps
as
important
as
providing
stakeholders
with
high-‐quality
data
analysis
tools.
Through
semi-‐structured
interviews
with
a
variety
of
stakeholders
at
CAPCS,
we
gained
an
understanding
of
what
features
people
most
wanted
in
a
dashboard,
and
what
the
context
of
data
use
and
data
literacy
is
at
CAPCS.
We
found
that
CAPCS
stakeholders
are
comfortable
using
data
in
their
work,
but
they
do
not
always
feel
that
there
is
a
strong
culture
of
data
literacy
throughout
the
organization.
For
the
dashboard,
stakeholders
were
interested
in
a
document
that
allowed
them
to
find
personally
significant
data
quickly,
and
to
see
performance
trends
over
time.
In
addition
to
creating
the
dashboard
prototypes,
we
have
included
a
detailed
analysis
of
the
feedback
we
received
on
the
culture
of
data
literacy
and
the
use
of
data
at
CAPCS.
In
the
final
section
of
this
report,
we
explain
the
features
of
the
new
dashboards
and
provide
a
set
of
further
recommendations
for
implementing
this
revised
dashboard.
The
recommendations
for
successful
implementation
are
as
follows:
1. Focus
resources
on
building
a
strong
and
supportive
culture
of
data
literacy
and
use.
2. Individualize
dashboards
to
meet
stakeholders’
diverse
needs.
3. Standardize
protocol
for
dashboard
dissemination
and
create
regular
space
for
data
analysis
and
collaboration.
4. Continue
to
improve
dashboard
and
data
systems
as
needs
and
culture
at
CAPCS
evolve.
6.
6
Project
Rationale
Introduction
The
Dorothy
I.
Height
Community
Academy
Public
Charter
Schools
(CAPCS)
were
founded
in
1998
as
a
response
to
the
pressing
need
for
a
high-‐quality
educational
option
for
urban
students
in
Washington,
DC.
CAPCS
has
five
campuses
and
serves
mostly
low-‐income
and
minority
students
from
grades
pre-‐kindergarten
through
six.
Like
many
schools,
CAPCS
uses
data-‐driven
decision
making
to
track
progress
toward
goals,
determine
effective
instructional
strategies,
and
meet
accountability
requirements
set
by
local
and
state
education
agencies.
One
of
the
tools
that
CAPCS
uses
for
data-‐driven
decision
making
is
a
data
dashboard,
which
uses
graphs
and
charts
to
present
and
summarize
critical
school
and
student-‐level
data
such
as
attendance,
enrollment,
and
academic
performance.
For
our
Master
of
Public
Policy
Capstone
project,
Colin
Welch,
our
CAPCS
liaison,
asked
us
to
create
updated
prototypes
for
a
new
dashboard
that
could
be
used
beginning
in
the
2014-‐2015
school
year.
We
conducted
research
to
inform
creation
of
an
initial
dashboard
prototype,
collected
feedback
from
relevant
CAPCS
stakeholders,
and
created
a
finalized
prototype
based
on
that
feedback.
In
addition,
we
prepared
an
analysis
of
stakeholder
feedback
and
recommendations
for
the
use
and
implementation
of
the
revised
dashboard,
which
can
be
found
later
in
this
report.
Data-‐Driven
Decision
Making
In
2001,
the
passage
of
the
No
Child
Left
Behind
Act
(NCLB)
became
the
impetus
for
a
shift
in
focus
onto
performance-‐based
school
accountability.
The
policy
aimed
to
improve
transparency
by
mandating
that
educators
and
administrators
meet
specific
data
requirements
in
areas
such
as
academic
achievement
levels,
student
learning,
and
teacher
professional
development.
Those
districts
that
met
the
requirements
would
receive
federal
funding,
while
those
that
continually
failed
to
meet
them
risked
losing
funding
and
having
schools
closed.
The
policy
was
driven
in
part
by
the
belief
that
the
effective
use
of
data
is
necessary
to
help
leaders
at
all
levels
assess
progress,
make
informed
decisions,
and
ultimately
improve
student
achievement.
This
process,
known
as
data-‐driven
decision
making,
has
become
an
essential
part
of
school
management
practices
due
to
the
increase
in
federal
standards-‐based
accountability
requirements.
School
systems
like
CAPCS
create
strategies
that
allow
for
effective
DDDM
through
the
use
of
tools
such
as
data
dashboards.
Data
dashboards
are
documents
that
use
graphs
and
charts
to
present
and
summarize
critical
school
and
student-‐level
data
such
as
enrollment,
suspensions
and
expulsions,
teacher
attendance,
and
professional
development.
Dashboard
Creation
The
purpose
of
this
project
was
to
provide
an
improved
data
dashboard
that
would
help
better
facilitate
the
decision
making
process
of
stakeholders
at
CAPCS
beginning
in
2014-‐2015
school
year.
The
new
dashboard
was
created
with
several
aims,
including
improving
comprehension,
readability,
usability,
interactivity,
and
implementation.
The
dashboard
was
to
be
shared
internally
with
a
range
of
decision
makers
and
users
such
as
central
office
staff,
academy
leaders
(principals),
instructional
coaches,
and
the
Board
of
Trustees.
The
effective
use
of
the
data
in
the
dashboard
will
help
these
stakeholders
assess
programs
and
make
informed
decisions.
Decisions
based
on
data
are
crucial
due
to
the
high
standards
and
performance
requirements
that
must
be
achieved
annually
in
order
for
the
schools
to
retain
their
charter
and
funding.
7.
7
Current
Data
Systems
A
representative
from
CAPCS,
Colin
Welch,
provided
us
with
samples
of
dashboards
that
CAPCS
has
used
in
the
past
[Appendices
A
and
B].
Mr.
Welch
also
communicated
how
he
intends
to
use
the
dashboards
and
provided
suggestions
for
their
look
and
feel.
He
requested
that
we
review
the
samples
provided,
collect
samples
from
other
schools
(or
similar
sources),
review
the
literature
pertaining
to
the
topic,
interview
stakeholders
within
the
organization,
and
create
several
sample
dashboard
designs.
We
finalized
a
template
for
CAPCS
to
use
after
creating
an
initial
prototype
based
on
focused
research,
promising
practices,
and
feedback
from
key
stakeholders.
CAPCS
relies
on
seven
data
systems
to
manage
its
student
and
school
information.
CAPCS
manages
four
of
these
data
systems
itself,
while
the
Office
of
the
State
Superintendent
for
Education
(OSSE)
and
the
DC
Public
Charter
School
Board
(PCSB)
manage
the
other
three.
Data
from
this
collection
of
systems
flows
into
PowerSchool,
the
core
data
information
system
used
by
CAPCS.
PowerSchool
and
other
centralized
information
systems
allow
administrators
and
teachers
to
access
enrollment,
demographic,
attendance,
and
discipline
records
using
a
single
login
and
portal
rather
than
several
portals.
Mr.
Welch
uses
PowerSchool
to
create
the
existing
data
dashboard
and
a
monthly
summary
for
the
Board
of
Trustees.
By
aggregating
student
and
classroom
information,
Mr.
Welch
synthesizes
key
internal
and
accountability
metrics
into
a
single
document.
This
document
is
then
shared
electronically
and
in
print
with
school
leaders,
central
office
staff,
and
the
Board
of
Trustees.
8.
8
Research
Questions
This
project
aimed
to
answer
the
following
research
questions:
1. What
are
the
current
best
practices
for
creating
dashboards?
2. How
should
CAPCS
visualize
data
for
use
in
making
decisions?
3. What
are
essential
contextual
factors
to
foster
implementation
of
data
dashboards?
Background
Community
Academy
Public
Charter
Schools
The
Dorothy
I.
Height
Community
Academy
Public
Charter
Schools
(CAPCS)
were
founded
in
1998
as
a
response
to
the
pressing
need
for
a
high-‐quality
educational
option
for
urban
students
in
Washington,
DC.
CAPCS
serves
students
in
pre-‐kindergarten
through
sixth
grade
at
four
traditional
campuses
located
in
Northwest
and
Northeast
DC
(Amos
1,
Amos
2,
Amos
3,
and
Butler)
and
an
online
campus
(CAPCS
Online).
CAPCS’
mission
is
to
create
a
caring
learning
community
where
students
acquire
the
knowledge,
skills,
and
habits
of
mind
to
think
critically;
to
read,
write,
speak,
and
listen
effectively;
to
reason
mathematically;
to
inquire
scientifically;
and
to
develop
the
social
competence
that
ensures
meeting
the
qualifications
for
acceptance
to
a
competitive
high
school
(Community
Academy
Public
Charter
Schools
2014).
The
table
below
contains
aggregated
data
from
the
District
of
Columbia
Public
Charter
School
Board
(PCSB).
As
the
table
below
demonstrates,
student
population
consists
of
primarily
minority
students
from
low-‐income
families.
9.
9
Table
1:
CAPCS
Student
Population
by
Campus
Amos
1
Amos
2
Amos
3
Butler
Total
Enrollment
510
280
479
308
African
American
65.9%
62.5%
99.0%
61.7%
Hispanic/
Latino
32.2%
35.4%
0.6%
28.2%
White
0.0%
0.7%
0.0%
3.2%
Asian/Pacific
Islander
0.2%
0.7%
0.0%
2.9%
Native
American/
Indian
1.4%
0.0%
0.2%
0.6%
Other
0.4%
0.7%
0.2%
3.2%
English
Language
Learners
40.2%
45.7%
2.9%
31.5%
Low-‐Income
87.8%
77.9%
89.4%
70.1%
Special
Education
12.0%
6.4%
12.9%
10.7%
Source:
DC
Public
Charter
School
Board.
2013
DC
Public
Charter
School
Performance
Reports.
Accountability
and
CAPCS
According
to
its
SY
2012-‐2013
annual
report,
CAPCS
is
committed
to
consistent
monitoring
of
accountability
and
increasing
its
response
to
data
results.
In
addition
to
guiding
values,
CAPCS
is
accountable
to
multiple
education
agencies.
First,
its
charter
must
be
renewed
every
five
years
by
the
PCSB.
CAPCS’
charter
was
most
recently
renewed
in
2013.
Secondly,
CAPCS
is
accountable
to
OSSE,
the
state
education
agency
that
governs
all
public
schools
in
the
District
of
Columbia.
In
addition,
CAPCS
is
accountable
to
federal
achievement
and
attendance
regulations
created
by
the
No
Child
Left
Behind
Act
(NCLB).
Finally,
the
school
system
is
also
held
accountable
by
its
own
Board
of
Trustees.
10.
10
The
combined
requirements
of
the
PCSB
and
other
localities,
including
federal
laws
like
NCLB,
oblige
CAPCS
to
amass
a
large
amount
of
data
on
their
students’
and
staff’s
achievement,
attendance,
and
other
activities.
As
a
result,
CAPCS
is
utilizing
the
required
collected
data
to
improve
decision
making
on
a
day-‐to-‐day
and
year-‐to-‐year
basis.
These
factors
combined
with
the
ability
to
access
large
swaths
of
data,
are
what
led
the
central
office
at
CAPCS
to
create
internal
data
dashboards
that
can
be
used
by
the
Board
of
Trustees,
central
office
staff,
and
academy
leaders
to
track
goals
and
inform
decision
making.
Accountability
and
the
Need
for
Accessible
Data:
The
No
Child
Left
Behind
Act
of
2001
The
2001
passage
of
NCLB
mandated
that
educators
and
administrators
meet
specific
data
requirements
in
order
to
receive
certain
federal
funding.
This
requirement
was
based
on
the
assumption
that
more
analysis
and
interpretation
of
data
would
lead
to
more
informed
decisions
for
school
reform.
The
policy
itself
is
based
on
the
premise
that
accountability
and
accessible
data
will
be
a
major
mechanism
in
improving
student
achievement
and
schools
as
a
whole
(Linn
2002).
School
districts
and
charter
management
organizations
are
now
required
to
report
on
a
variety
of
performance
measures
such
as
achievement
levels,
student
learning,
and
professional
development
(Park
2009).
Performance-‐
based
accountability
has
improved
transparency
in
education.
Specifically,
NCLB
required
that
performance
data
be
disaggregated
by
sub-‐group
such
as
low-‐income
and
minority,
students
with
disabilities,
and
English
Language
Learners
(ELL).
This
provided
data
analysts
with
a
clearer
understanding
of
the
situation
at
the
school
and
district
levels
(Wong
2003).
The
increase
in
available
data
allows
teachers
and
administrators
to
evaluate
existing
capacities
and
identify
weaknesses,
monitor
progress
and
efficacy
of
programs,
and
inform
future
development
plans
and
decisions
(Park
2009).
These
factors
together
will
hopefully
lead
to
improved
student
performance.
However,
the
benefits
of
data
will
not
be
realized
until
they
are
communicated
effectively
and
to
an
audience
that
is
able
to
understand
and
interpret
the
information.
A
school
needs
internal
motivation,
structure,
and
capacity
as
well
as
external
requirements
(i.e.
NCLB)
in
order
to
create
an
effective
accountability
system
and
a
culture
of
DDDM
(Sutherland
2004).
Although
NCLB
brought
accountability
and
DDDM
into
the
spotlight
of
education
reform,
it
is
not
a
novel
idea.
DDDM
in
education
originates
from
successful
practices
in
industry
and
manufacturing,
in
which
the
assessment
of
input
data
yields
successful
and
efficient
output
(Marsh
2006).
Still,
data
were
important
in
education
reform
for
decades
prior
to
the
passage
of
NCLB.
State
requirements
for
data
use
in
school
improvement
plans
began
in
the
1970s,
and
in
the
1980s
there
were
debates
about
measurement-‐driven
instruction
(Marsh
2006).
Additionally,
data
use
for
strategic
planning
in
school
systems
dates
back
to
the
1980s
and
1990s
(Marsh
2006).
Still,
NCLB
marks
a
greater
transition
to
accountability
because
of
test-‐based
requirements
and
data
reporting
in
aggregated
and
disaggregated
forms
(Marsh
2006).
Schools
now
have
a
vast
amount
of
data
at
their
disposal
and
need
mechanisms
and
tools
that
allow
them
to
analyze
the
information
and
make
decisions.
Data
dashboards
that
clearly
and
succinctly
depict
this
information
are
an
invaluable
tool
that
educators
and
administrators
can
use
to
do
their
jobs
more
effectively.
As
Sutherland
(2004)
discussed,
both
external
and
internal
factors
are
necessary
in
order
to
create
and
maintain
a
culture
of
evaluation
and
data
use.
Assessment
and
data
are
only
useful
if
there
is
the
capacity
to
use
that
information
effectively.
A
dashboard
is
an
effective
tool
for
this
purpose.
However,
capacity
for
DDDM
goes
beyond
having
a
dashboard
for
teachers
and
administrators;
it
also
refers
to
the
capacity
of
those
teachers
and
administrators
to
interpret
and
analyze
the
information
as
it
is
presented
to
them.
Applied
Data-‐Driven
Decision
Making:
Turning
Data
into
Actionable
Knowledge
Many
schools
utilize
the
data
made
available
by
federal,
state,
and
local
requirements
to
better
inform
decision
making
and
strategy
applied
by
various
stakeholders.
In
the
case
of
CAPCS,
the
Board
of
Trustees
uses
data
to
ensure
that
year-‐end
goals
are
met.
Other
stakeholders
such
as
central
office
staff,
11.
11
academy
leaders,
and
instructional
coaches
use
data
to
track
their
students’
achievement
and
attendance,
teacher
professional
development,
and
other
important
factors.
A
base
of
literature,
both
theoretical
and
applied,
examines
effective
and
ineffective
ways
for
a
school
system
or
school
to
practically
apply
DDDM
to
its
day-‐to-‐day
practices
(see
Figure
1).
Figure
1
shows
an
applied
framework
that
we
created
based
on
the
literature
and
research
that
was
conducted.
It
illustrates
a
path
that
might
be
taken
when
an
actor
employs
DDDM.
The
dashed
feedback
line
indicates
that
an
actor
might
move
between
stages
instead
of
following
the
arrows
from
step
to
step.
The
remainder
of
this
section
details
the
steps
that
might
be
taken
by
an
actor
to
fully
implement
DDDM.
Figure
1:
Framework
for
Describing
Data-‐Driven
Decision
Making
in
Education
In
coordination
with
Figure
1,
the
following
steps
are
based
on
the
literature
and
research
and
might
be
taken
by
a
set
of
actors
engaged
in
DDDM.
Step
1
-‐
Gather
and
Organize
Raw
Data
First,
actors
gather
and
organize
raw
data
to
use
in
what
is
ideally
the
most
effective
manner
that
matches
their
needs.
There
can
be
many
types
of
data:
input
(school
expenditures
or
demographics),
process
(information
on
financial
operations
or
quality
of
instruction),
outcome
(dropout
rate
or
student
assessment),
and
satisfaction
(opinions
from
teachers,
students,
parents,
or
members
of
the
community)
(Marsh
2004).
These
data
can
be
described
in
a
quantitative,
qualitative,
simple,
or
complex
manner
(Ikemoto
2007)
and
can
be
organized
and
stored
in
numerous
ways.
Some
schools
use
student
information
systems
like
PowerSchool
or
data
management
systems
that
are
created
specifically
for
their
needs.
Others
export
data
from
a
management
system
and
place
it
into
a
spreadsheet
that
then
configures
the
data
into
a
tool
that
can
be
used
to
inform
selected
stakeholders.
12.
12
Step
2
-‐
Information
and
Data
Literacy
Once
the
data
are
gathered,
they
are
presented
to
the
relevant
stakeholders
and
become
information.
Information
might
be
presented
in
the
form
of
a
PDF,
an
Excel
spreadsheet,
or
via
a
program
such
as
PowerSchool
that
is
accessed
via
the
Internet.
The
form
data
takes
when
presented
as
information
is
extremely
important.
Bambrick-‐Santoyo
(2010)
notes
that
it
is
easy
to
gather
data
but
hard
to
analyze
and
utilize
its
conclusions
effectively.
He
also
asserts
that
the
ultimate
end
users
must
be
kept
in
mind
when
creating
a
template
that
will
be
used
for
decision
making.
In
this
step,
a
separate
but
important
consideration
is
data
literacy.
Data
literacy
is
a
fundamental
aspect
of
effective
data
use.
The
modern
era
of
DDDM
causes
a
transition
such
that
now
not
only
an
exceptional
principal,
expert
teacher,
or
central
office
member
manages
a
school’s
vital
information,
but
all
teachers
and
administrators
are
expected
to
be
capable
to
conduct
their
own
data
analysis
within
their
professional
role
(Park
2009).
If
stakeholders
do
not
feel
comfortable
and
regard
data
as
overwhelming
rather
than
as
a
useful
tool,
a
dashboard
will
be
unable
to
serve
its
intended
purpose
or
be
utilized
to
its
maximum
potential
(Almy
2014).
Additionally,
in
their
study
of
district-‐wide
data
systems,
Hayman
and
Cho
found
that
it
is
important
for
district
leadership
to
set
a
vision
for
how
data
will
be
used
by
all
stakeholders
across
positions.
Districts
that
actively
cultivated
a
common
culture
of
data
literacy
and
data
use
were
most
successful
at
fully
implementing
DDDM
(Hayman
and
Cho
2014).
Step
3
-‐
Decisions
from
Data
In
the
third
step,
decisions
are
made
when
information
is
turned
into
actionable
knowledge
(Park
2009).
Depending
on
what
is
being
tracked,
these
decisions
might
inform
a
decision,
compare
metrics,
or
lead
the
actor
to
take
a
new
course
of
action.
According
to
Bambrick-‐Santoyo
(2010),
the
decisions
must
be
made
and
implemented
in
a
timely
manner.
Additionally,
the
context
of
why
and
how
the
decisions
are
made
and
executed
should
be
considered
(Park
2009).
Step
4
-‐
Implement
Decisions
for
Impact
During
the
final
step,
the
relevant
actors
implement
decisions
that
were
made
based
on
the
earlier
steps.
Like
many
actions
in
a
school
setting,
proper
implementation
is
vital
not
only
for
DDDM
to
be
effective
but
to
ensure
that
the
goal
or
metric
is
met
or
improved
upon
(Marsh
2006).
Factors
Affecting
Data-‐Driven
Decision
Making
Often,
the
reality
of
data-‐driven
decision
making
is
not
as
linear
as
is
outlined
in
the
steps
above
or
in
the
literature
(Ikemoto
2007).
Like
any
system,
there
is
a
possibility
that
an
actor
might
not
follow
the
prescribed
framework
and
instead
make
a
decision
based
on
intuition,
context,
or
a
separate
factor.
This
reality
makes
it
necessary
for
the
following
factors
and
implications
to
be
considered
by
any
group
that
is
engaging
in
DDDM:
accessibility
and
timeliness
of
data;
perceived
validity
of
data;
staff
capacity
and
support;
time;
partnerships
with
external
organizations;
tools
used;
organizational
culture
and
leadership;
and
policy
context
(Ikemoto
2007).
Finally,
the
leaders
of
the
school
system
or
school
should
anticipate
that
an
actor
might
make
a
decision
outside
the
framework
and
in
turn
be
impacted
by
the
factors
listed.
Overview
of
the
Study
The
study
used
a
three-‐phase
methodology
to
achieve
the
ultimate
goal
of
creating
a
more
effective
and
easily
understood
data
dashboard
for
CAPCS.
The
first
phase
used
data
visualization
research
and
CAPCS’
stated
needs
to
create
a
framework
for
the
new
dashboard
prototype.
The
second
13.
13
phase
utilized
semi-‐structured
interviews
with
key
stakeholders
to
optimize
the
school
performance
dashboard.
Stakeholders
included
different
members
of
the
CAPCS
community
with
a
vested
interest
in
data
and
accountability
such
as:
academy
leaders,
central
office
leaders,
instructional
coaches,
an
English
Language
Learners
(ELL)
representative,
a
data
associate,
and
a
human
resources
representative.
The
final
stage
created
the
new
dashboard
prototype
for
CAPCS
to
use
to
report
school
progress
more
effectively
to
stakeholders.
Phase
1
Research-‐Informed
Prototype
Creation
A
dashboard
is
a
visual
display
of
the
most
important
information
needed
to
achieve
one
or
more
objectives.
Typically,
the
information
presented
on
a
dashboard
is
consolidated
and
arranged
on
a
single
screen
so
the
information
can
be
monitored
at
a
glance.
Dashboards,
which
began
to
appear
in
the
1980s
as
a
way
for
corporate
executives
to
monitor
key
performance
indicators
for
their
entire
organization,
have
recently
become
standard
tools
for
decision
makers
at
all
levels
and
in
all
types
of
organizations.
The
widespread
use
of
dashboards
by
technology
companies
led
to
the
perception
that
the
efficacy
of
a
dashboard
results
from
the
sophistication
of
the
software
used
in
its
creation.
While
technology
plays
an
important
role
in
the
speed
and
efficiency
of
information
transfer,
many
dashboards
fail
to
communicate
with
and
add
value
to
organizations
due
to
poor
design
and
implementation
(Few
2006,
4).
Most
recently,
CAPCS
relied
on
two
data
dashboards:
one
for
CAPCS
board
members
[Appendix
B]
and
another
designed
for
school
leaders
[Appendix
A].
The
board
member
dashboard
was
a
two-‐page
document
that
listed
CAPCS’
charter
agreement
targets,
the
status
of
each
target,
and
notes
on
each
target
in
tabular
format.
The
school
leader
dashboard
was
a
ten-‐page
document
that
featured
a
detailed
account
of
metrics
related
to
literacy,
math,
and
behavior
with
over
twenty
graphs,
seven
tables,
and
a
notes
section.
Findings:
Research-‐Informed
Prototype
Creation
While
the
dashboards
provided
a
detailed
account
of
the
academic
and
behavioral
performance
of
CAPCS
students,
several
aspects
of
well-‐designed
dashboards
were
absent.
First,
the
multi-‐page
design
of
the
school
leader
dashboard
made
it
impossible
to
view,
understand,
and
interpret
information
with
a
simple
glance.
The
human
brain
has
a
limited
amount
of
information
that
can
be
stored
in
working
memory,
often
referred
to
as
short-‐term
memory.
Research
has
shown
that
the
human
brain
can
hold
between
five
to
nine
items
in
working
memory
at
any
given
time
before
they
are
forgotten
(Miller
1956).
In
short,
it
is
nearly
impossible
for
the
average
person
to
make
sense
of
large
amounts
of
data
spanning
several
pages.
Second,
the
graphs
lacked
visual
indicators
such
as
trend
arrows
or
icons,
which
would
alert
users
of
improving
or
declining
performance
over
time.
Given
the
large
number
of
metrics
that
schools
must
monitor
and
the
limited
amount
of
time
that
staff
are
able
to
spend
analyzing
data,
it
is
imperative
to
design
dashboards
that
quickly
highlight
progress
and
areas
of
concern.
Based
on
the
research
by
Few
(2006)
and
Miller
(1956),
we
created
a
dashboard
prototype
to
address
the
shortcomings
listed
above
[Appendix
C].
Our
dashboard
prototype
shortened
the
dashboard
from
eleven
pages
to
two
by
limiting
the
scope
of
data
presented
to
include
only
primary
indicators
of
academic
and
behavioral
performance.
Secondly,
color-‐coded
trend
arrows
were
placed
to
the
left
of
all
graphs
to
indicate
an
improvement
or
decline
in
performance
from
the
previous
month.
Thirdly,
all
graphs
featured
data
spanning
the
previous
three
months
in
order
to
show
longer-‐term
trends
for
each
metric.
Fourthly,
all
graphs
featured
visual
indicators
marking
CAPCS’
current
performance
in
relation
to
its
end
of
year
goals.
The
twofold
aim
of
the
prototype
was:
to
create
graphics
to
help
users
quickly
identify
areas
of
progress
and
concern,
and
to
present
key
aspects
of
each
metric
without
taxing
the
user’s
capacity
of
working
memory,
thereby
allowing
the
overall
picture
of
student
performance
to
be
more
easily
understood
in
a
short
period
of
time.
14.
14
Phase
2
Data
Collection
with
Semi-‐Structured
Interviews
In
Phase
2,
we
conducted
in-‐person
semi-‐structured
interviews
to
collect
feedback
from
a
representative
set
of
stakeholders
on
the
two
current
dashboards
and
our
prototype.
A
total
of
21
stakeholders
from
CAPCS
were
contacted
along
with
one
stakeholder
from
another
Washington,
DC-‐
based
public
charter
school
system.
Twelve
of
the
21
stakeholders,
all
of
whom
were
from
CAPCS,
were
interviewed
for
a
response
rate
of
57
percent.
All
twelve
interviews
took
place
in
Washington,
DC
at
CAPCS’
central
office
and
its
four
physical
campuses.
Of
the
twelve
stakeholders
interviewed,
seven
were
central
office
employees,
two
were
academy
leaders,
and
three
were
either
instructional
coaches
or
curriculum
specialists.
The
interviews
took
place
on
various
dates
throughout
the
weeks
of
March
24,
March
31,
and
April
7,
2014.
All
interviews
were
conducted
in
person
because
displaying
and
explaining
the
multiple
dashboards
over
the
phone
would
have
likely
caused
confusion
and,
therefore,
less
useful
responses.
Research
shows
that
face-‐to-‐face
is
the
best
method
for
interviews
that
require
visual
aids
or
contain
many
open-‐ended
questions
(Wholey
et
al.
2010).
We
elected
to
conduct
interviews
with
stakeholders
in
a
variety
of
roles
because
stakeholders
tend
to
make
sense
of
data
systems
based
on
their
personal
perceptions
and
the
dominant
data-‐orientation
of
their
respective
workplaces
(Cho
2014).
That
is
why
we
anticipated
that
each
CAPCS
stakeholder
group
would
use
the
data
dashboard
in
different
ways.
We
created
an
interview
script,
which
also
contained
the
interview
protocol
[Appendix
E].
The
purpose
of
this
document
was
to
maintain
a
standard
interview
process
for
all
four
interviewers.
Three
dashboards
were
used
to
assist
the
interview
process
and
inform
the
creation
of
the
final
dashboard
prototype.
These
dashboards
were
referred
to
as
“Current
Tool”
[Appendix
A],
“Dashboard
A”
[Appendix
B],
and
“Dashboard
B”
[Appendix
C].
They
were
chosen
for
use
during
interviews
due
to
the
differences
in
layout
and
content,
which
allowed
the
stakeholders
to
compare
and
contrast
them
to
one
another.
The
“Current
Tool”
is
a
dashboard
created
using
Microsoft
Excel
that
Mr.
Welch
and
the
CAPCS
data
team
use
to
display
campus-‐specific
information
such
as
in-‐seat
attendance,
enrollment
changes,
and
academic
interventions.
“Dashboard
A”
is
a
summary
document
that
Mr.
Welch
prepares
monthly
on
Microsoft
Word
and
contains
campus-‐specific
information
such
as
charter
agreement
targets,
attendance,
re-‐
enrollment,
and
community
engagement.
“Dashboard
B”
is
the
initial
prototype
we
created
using
Microsoft
Word.
It
was
developed
based
on
existing
research
on
data
visualization
and
conversations
with
Mr.
Welch.
“Dashboard
B”
contained
fabricated
campus-‐specific
data
such
as
reading
and
math
proficiency,
student
absences,
and
parent
event
attendance.
We
encountered
some
limitations
while
working
on
the
interview
portion
of
the
project.
First,
we
did
not
initiate
contact
with
any
CAPCS
stakeholders
because
we
agreed
that
Mr.
Welch
would
connect
us
via
email
with
all
of
the
stakeholders.
Many
of
the
stakeholders
may
not
have
responded
due
to
the
fact
that
the
interviews
were
being
conducted
during
the
DC
CAS
testing
period.
Additionally,
central
office
managers
determined
that
it
would
not
be
feasible
for
us
to
discuss
the
data
dashboards
with
members
of
CAPCS’
Board
of
Trustees.
While
these
factors
all
led
to
a
small
sample
size,
our
results
are
representative
of
different
levels
of
DDDM
and
data
use
at
CAPCS.
Additionally,
out
of
respect
for
each
interviewee’s
time,
interviews
were
limited
to
30
minutes
and
therefore
certain
questions
that
we
deemed
unessential
were
omitted
in
some
interviews.
In
a
few
cases,
follow-‐up
questions
that
were
not
on
the
interview
script
needed
to
be
asked
for
clarification
purposes.
Interviews
with
higher-‐level
staff
members
or
those
who
were
more
familiar
with
the
dashboards
tended
to
be
much
more
open-‐ended
because
their
increased
levels
of
data
literacy
led
to
more
opinions
and
input
on
the
prototypes
and
data
in
general.
This
gave
us
additional
information,
which
we
were
able
to
apply
during
creation
of
the
final
dashboard
prototype.
15.
15
Phase
2
Findings
Data
Literacy
Levels
During
the
semi-‐structured
interviews,
CAPCS
staff
members
self-‐reported
their
personal
levels
of
comfort
using
data
to
inform
workplace
decisions.
They
were
asked:
“On
a
scale
of
1
to
5,
with
one
being
not
at
all
comfortable
and
five
being
very
comfortable,
how
comfortable
would
you
say
you
are
with
using
data
to
inform
your
work?”
Of
the
twelve
respondents,
75
percent
scored
their
comfort
levels
at
4
or
5.
In
addition,
the
majority
of
surveyed
CAPCS
staff
use
data
regularly
in
their
decision
making
process.
They
were
asked:
“In
your
position,
how
often
do
you
use
data
to
make
decisions?”
Of
the
twelve
respondents,
67
percent
said
they
use
data
to
make
decisions
at
least
once
a
week.
From
these
data,
we
can
see
that
CAPCS
has
a
basic
culture
of
DDDM.
For
the
most
part,
CAPCS
staff
fall
somewhere
between
the
second
and
third
steps
of
Ikemoto’s
DDDM
framework
(2007).
None
of
the
stakeholders
reported
that
they
never
use
data
in
decision
making,
so
we
can
conclude
that
data
is
viewed
as
a
tool
at
CAPCS
and
it
may
not
be
necessary
to
focus
resources
on
developing
very
basic
data
literacy
skills
in
staff
members.
CAPCS
stakeholders
are
also
on
the
same
page
when
it
comes
to
how
data
is
used
at
CAPCS.
As
Figure
2
shows,
central
office
employees,
academy
leaders,
and
instructional
and
curriculum
staff
all
agree
that
CAPCS
uses
data
in
multiple
ways.
Figure
2
counts
the
number
of
stakeholders
who
identified
one
of
three
main
buckets
of
data
use:
improving
outcomes,
tracking
progress
toward
goals,
and
accountability.
Each
letter
in
the
circles
represents
one
respondent
who
has
identified
that
CAPCS
uses
data
in
a
specific
way.
Letters
are
not
unique
across
circles,
so
one
respondent
may
be
represented
in
multiple
circles.
This
shows
that
many
CAPCS
employees
have
a
complex
understanding
of
how
data
is
used
within
the
organization.
So,
looking
only
at
the
“Improving
Outcomes”
bucket,
four
central
office
employees,
two
academy
leaders,
and
two
instructional
coaches
agree
that
CAPCS
uses
data
to
improve
outcomes.
Additionally,
we
can
see
that
there
are
three
central
office
employees
who
identified
all
three
buckets
as
ways
in
which
CAPCS
uses
data.
No
single
stakeholder
thought
that
there
was
only
one
proper
way
to
use
data
at
CAPCS,
and
a
majority
of
those
responding
to
the
question
agreed
that
CAPCS
used
data
to
improve
student
outcomes,
track
progress
toward
goals,
and
for
accountability
(internal
and
external).
Figure
2:
Data
Should
Be
Used
to
Improve
Outcomes
16.
16
The
quote
at
the
bottom
of
Figure
2
gets
at
the
heart
of
the
culture
that
is
being
cultivated
among
these
stakeholder
groups.
Across
all
groups,
data
use
is
purposeful—these
numbers
are
not
used
punitively
to
“catch”
stakeholders
doing
wrong
or
underperforming;
they
are
useful
tools
to
be
employed
in
the
effort
of
creating
the
best
schools
possible
for
the
students
CAPCS
serves.
In
their
study
of
data
use
and
sense
making
in
school
districts,
Cho
and
Wayman
(2014)
found
that
school
districts
where
multiple
groups
of
stakeholders
in
disparate
positions
had
a
common
understanding
of
the
“why”
of
data
use
were
more
successful
at
creating
a
positive
and
productive
data
culture.
The
attitudes
expressed
in
the
interview
process
show
that
CAPCS
has
done
a
good
job
of
setting
a
comprehensive
and
multifaceted
vision
of
data
use
for
its
staff.
When
asked
if
CAPCS
actively
cultivates
a
culture
of
literacy,
responses
were
more
mixed.
Only
one
third
of
respondents
agree
outright,
but
those
who
were
neutral
or
disagreed
gave
optimistic
or
aspirational
feedback
about
how
CAPCS
could
reach
a
point
where
there
was
a
true
culture
of
data
literacy
(see
Table
2
below).
Table
2:
CAPCS
Stakeholders
Optimistic
About
Data
Literacy
Do
you
agree
or
disagree
that
CAPCS
cultivates
a
culture
of
data
literacy?
Agreement
"Absolutely.
Definitely.
Well,
everybody
is
data-‐driven,
from
the
top—from
the
central
office—down
to
the
campus…We
understand
the
importance
of
data,
I
think
more
than
we
have
before...and
not
just
data
as
far
as
numbers.
I
mean
data
even
as
far
as
how
many
parents
did
you
have
show
up
at
parent/teacher
conferences?
What
do
you
think
is
attributed
to
them
not
coming?
Just
being
able
to
talk
teachers
through
certain
things
like
that
is
one
way
to
track
the
data”—Academy
Leader
“There
are
many
data
meetings
where
we
provide
data
to
teachers
and
explain
as
well
as
show
them
where
to
find
the
information
themselves.
There
is
a
focus
on
making
sure
everyone
knows
what
the
data
means
and
how
to
use
it.”—Central
Office
Employee
Aspiration/Optimism
“I
agree,
we
are
moving
in
that
direction.
We
have
someone
specifically
assigned
to
work
on
data
and
push
that
down
into
schools.”—Central
Office
Employee
"I
have
worked
in
other
cultures
that
are
very
big
with
numbers.
We
look
at
numbers
but
we
don’t
let
them
drive
us
crazy."—Instructional
Coach
“I
think
that
certain
individuals
at
CAPCS
cultivate
a
culture
of
data
literacy.
I
think
they
are
really
good
about
sharing
their
knowledge
about
data
and
helping
other
people
understand
data.”
—Central
Office
Employee
Those
individuals
who
did
agree
that
CAPCS
has
a
culture
of
data
literacy
pointed
out
ways
that
the
organization
has
provided
more
opportunity
for
employees
to
engage
in
analysis
and
discussion
around
data.
Data
conferences
involving
multiple
stakeholder
groups
were
a
popular
example,
and
are
exactly
the
sort
of
occasion
that
will
eventually
lead
to
data
sharing
and
collaboration
across
stakeholder
groups.
As
one
central
office
employee
stated,
“They
[CAPCS
stakeholders]
are
now
seeing
how
data
is
helpful
to
guide
instruction.
Now
it
gives
a
reason
to
teachers...why
we
need
them
to
do
the
things
that
17.
17
they
do.”
Continuing
these
practices
will
be
fundamental
to
strengthening
CAPCS’
common
vision
of
how
and
why
data
is
important.
For
those
who
did
not
agree
that
CAPCS
cultivates
a
culture
of
data
literacy,
a
recurring
theme
was
a
certain
“skills
silo”
in
which
the
data
person
has
the
knowledge
and
access
to
help
others,
but
without
whom
analysis
would
not
occur
at
all.
Such
perception
can
be
dangerous
to
an
organization,
as
it
causes
groups
without
access
to
disengage
from
DDDM
and
to
reject
data
as
part
of
their
own
vision
of
CAPCS’
essential
properties
and
values
(Cho
and
Wayman
2014).
As
a
curriculum
specialist
stated,
“I
think
that
certain
individuals
at
CAPCS
cultivate
a
culture
of
data
literacy...I
don’t
think
it’s
been
infused
in
everybody.”
It
will
be
important
for
CAPCS
to
continue
to
offer
individuals
opportunities
to
engage
with
data
and
to
understand
its
role
in
their
own
responsibilities
in
order
to
continue
to
cultivate
a
productive
culture
of
DDDM.
Prototype
Feedback
During
the
semi-‐structured
interviews,
CAPCS
staff
members
were
presented
with
three
dashboards:
the
current
tool
being
used
by
CAPCS
[Appendix
A],
a
board
summary
document
referred
to
as
“Dashboard
A”
[Appendix
B],
and
our
initial
dashboard
prototype,
referred
to
as
“Dashboard
B”
[Appendix
C].
73
percent
of
respondents
reported
that
the
layout
of
the
current
tool
was
easy
to
read
and
understand.
In
addition,
73
percent
of
respondents
reported
that
based
on
the
information
included
on
the
current
tool
and
the
way
in
which
it
is
presented,
the
tool
would
help
them
make
decisions
more
quickly.
However,
many
of
the
stakeholders
were
unwilling
to
look
at
a
dashboard
for
a
long
period
of
time
in
order
to
find
the
information
they
needed.
This
unwillingness
became
evident
as
they
flipped
through
the
current
tool,
which
is
over
ten
pages
in
length.
While
looking
through
the
current
tool,
one
central
office
employee
said,
“There
is
way
too
much
information
on
here.”
Stakeholders
of
all
positions
did
like
the
first
page
of
the
current
tool
which
is
a
summary
page
containing
information
such
as
enrollment
changes,
attendance,
academic
interventions,
and
professional
development.
However,
all
pages
following
the
summary
page
contain
various
charts
and
graphs
for
specific
metrics.
During
the
interviews,
stakeholders
were
asked
what
was
missing
from
both
Dashboard
A
and
Dashboard
B.
Of
the
twelve
respondents,
only
42
percent
stated
that
there
were
elements
missing.
This
low
response
rate
indicates
what
we
had
anticipated,
which
is
that
stakeholders’
ideal
dashboard
would
combine
the
textual
summaries
and
descriptions
featured
on
Dashboard
A
with
the
visual
charts
and
graphs
featured
on
Dashboard
B.
Those
who
were
able
to
identify
what
was
missing
had
suggestions
that
can
be
seen
in
Table
3.
It
became
evident
that
context
is
an
extremely
important
aspect
of
a
data
dashboard.
Stakeholders
suggested
that
perhaps
there
should
be
a
dashboard
containing
subject-‐specific
metrics.
This
emphasized
the
fact
that
people
in
different
positions
are
looking
for
different
metrics
–
an
instructional
coach
who
focuses
on
math
will
want
to
see
the
students’
progress
in
math,
while
a
central
office
staffer
may
be
more
interested
in
attendance
and
enrollment.
Table
3:
Context
is
Crucial
to
a
Dashboard
What
features
are
missing
from
both
of
these
prototypes
that
you
want
to
see?
Why?
“The
summative
information
is
good,
but
I
would
need
a
break
out
per
campus
to
really
help
inform
decisions.
It
would
also
be
helpful
to
see
the
comparison
of
performance
to
other
ANet
schools.”
—Central
Office
Employee
“It’s
just
that
when
you
say
reading
proficiency,
I
think,
‘based
on
what?’
I
think
that
the
sub-‐skill
information
would
be
most
useful.
In
literacy,
for
example.”—Curriculum
Specialist
“...it
would
be
helpful
to
see
what's
happening.
Maybe
a
one
or
two-‐word
description
of
what
that
intervention
is,
what
that
activity
is."—Central
Office
Employee
18.
18
Reactions
to
Dashboard
A
and
Dashboard
B
were
positive.
When
asked
to
choose
which
Dashboard
(A
or
B)
they
preferred
at
first
glance,
90
percent
of
respondents
chose
Dashboard
B.
This
was
largely
due
to
the
colors,
graphics,
and
simple
layout
of
the
prototype.
After
being
given
the
chance
to
carefully
review
all
three
prototypes,
73
percent
of
respondents
stated
they
preferred
Dashboard
B.
One
of
the
main
reasons
for
their
preference
was
the
colored
trend
arrows
feature,
which
specified
whether
metrics
had
increased
or
decreased
from
the
previous
time
period.
As
can
be
seen
in
Table
4,
stakeholders
had
mixed
reactions
when
asked
what
features
of
the
current
dashboard
they
prefer
over
the
prototypes.
Their
responses
once
again
depended
upon
their
position.
For
instance,
one
curriculum
specialist
indicated
that
she
preferred
the
current
tool
because
the
information
that
is
relevant
to
her
work
was
not
displayed
on
either
Dashboard
A
or
Dashboard
B.
One
academy
leader
found
the
large
amount
of
information
displayed
on
the
current
tool
useful:
“It’s
all
useful,
it’s
all
right
here
together…”
Other
responses
were
based
on
whether
or
not
each
individual
was
a
visual
learner
and
preferred
graphs
and
charts
over
paragraph
descriptions.
The
most
common
element
that
stakeholders
identified
as
important
was
trend
indicators.
Table
4:
CAPCS
Stakeholders
Seek
Trend
Indicators
on
Dashboards
What
features
of
the
current
dashboard
do
you
prefer
over
the
prototypes?
"There's
more
data
here,
for
sure.
It
looks
like...it's
more
complete
here.
Whether
or
not
that's
a
plus
or
minus
depends
on
the
audience
and
what
they
want
to
see."—Central
Office
Employee
"I
like
the
actual
numbers
versus
percentages.
Although,
when
you
have
the
percentages
on
Dashboard
B
where
it
says
if
you
increased
from
last
month,
those
are
very
helpful.
But
for
the
actual
count
within
each
domain,
I
would
prefer
the
number
versus
the
percentage."
—Central
Office
Employee
“The
last
year
column
for
comparison
is
useful.
I
would
like
a
full
year
summary,
not
just
three
months.”
—Instructional
Coach
Figure
3:
CAPCS
Stakeholders
Reveal
Most
Important
Data
Points
19.
19
CAPCS
stakeholders
were
also
asked
to
identify
the
top
three
most
important
pieces
of
data
on
each
dashboard.
Figure
3
counts
the
number
of
stakeholders
who
identified
attendance,
enrollment,
or
academic
interventions
and
strategies
as
important
on
the
current
tool.
Each
letter
in
the
circles
represents
one
respondent
who
has
identified
that
item
as
important.
Letters
are
not
unique
across
circles,
so
one
respondent
may
be
represented
in
multiple
circles.
The
figure
shows
that
stakeholders
at
all
positions
identified
attendance,
enrollment,
and
academic
interventions
and
strategies
as
important.
All
three
of
the
items
were
displayed
on
the
first
page
of
the
current
tool,
which
is
a
summary
page.
Only
a
small
minority
took
the
time
to
flip
through
the
document
before
answering
the
question,
which
shows
the
importance
of
having
both
a
summary
page
and
different
metrics
for
different
stakeholders.
For
both
Dashboards
A
and
B,
a
majority
of
stakeholders
claimed
that
the
literacy
and
math
targets
were
the
most
important
aspects
on
display.
Phase
3
Final
Dashboard
Prototype
Creation
After
we
conducted
interviews
with
the
CAPCS
stakeholders,
interview
responses
were
transcribed
and
analyzed.
Through
an
analysis
of
stakeholder
responses
to
questions
comparing
CAPCS’
current
dashboards
to
our
prototype,
several
themes
emerged.
First,
stakeholders
were
reluctant
to
spend
more
than
fifteen
seconds
reviewing
a
dashboard.
Second,
stakeholders
favored
visual
indicators
that
specified
when
metrics
had
increased
or
decreased
from
the
previous
period.
Third,
in
addition
to
accountability
metrics,
which
relate
to
students’
overall
proficiency
in
a
subject
area,
stakeholders
suggested
that
subject-‐specific
skills
metrics
would
provide
more
actionable
insight.
These
results
are
expanded
upon
in
the
“Prototype
Feedback”
section
of
this
report.
To
address
the
major
concerns
listed
above,
we
created
two
additional
prototypes:
a
document-‐
based
dashboard
using
Microsoft
Excel
and
a
web-‐based
interactive
dashboard
using
Google
Spreadsheets.
Stakeholders
wanted
to
identify
problem
areas
in
as
little
as
fifteen
seconds,
yet
they
also
desired
a
greater
level
of
detail
for
each
subject
area.
We
provided
Mr.
Welch
with
two
strategies
to
reconcile
both
needs:
(1)
a
document-‐based
dashboard
that
featured
conditionally
formatted
tables
instead
of
charts,
and
(2)
a
web-‐based
dashboard
that
allowed
users
to
interactively
explore
accountability
and
behavioral
metrics.
For
the
final
document-‐based
dashboard,
metrics
were
summarized
using
tables
instead
of
charts.
Despite
the
positive
feedback
we
received
regarding
the
use
of
graphs
in
our
prototype,
it
was
impossible
to
summarize
all
of
CAPCS’
required
metrics
while
maintaining
a
one-‐page
limit.
In
order
to
compensate
for
the
lack
of
charts,
we
utilized
Microsoft
Excel’s
conditional
formatting
features
to
quickly
highlight
areas
of
progress
and
concern.
We
also
used
arrow
icons
to
indicate
the
increase
or
decrease
of
each
metric.
Conditional
formatting
was
configured
so
that
metrics
where
CAPCS
was
failing
to
meet
its
yearly
goals
were
automatically
highlighted
in
red,
while
metrics
where
CAPCS
was
successfully
achieving
its
annual
goals
were
highlighted
in
green.
Data
related
to
the
primary
metric
was
listed
below
the
key
metric.
Green
and
red
arrow
icons
were
used
to
show
the
increase
or
decrease
of
each
related
sub-‐metric
[Appendix
D].
Using
tables
allowed
us
to
increase
the
number
of
metrics
listed
from
a
maximum
of
six
metrics
per
page
to
a
maximum
of
44
metrics
per
page.
This
approach
resulted
in
a
dashboard
on
which
all
accountability
metrics
and
subject-‐specific
skills
fit
comfortably
on
a
single
page.
The
final
web-‐based
dashboard
featured
interactive
graphs
that
were
created
using
Google
Spreadsheets.
The
web-‐based
dashboard
separated
reading,
math,
and
non-‐academic
metrics
into
three
separate
tabs.
The
tabs
featured
an
interface
that
allowed
users
to
select
metrics
on
an
x-‐y
axis
and
see
how
metrics
changed
in
relation
to
one
another
over
time.
Users
also
had
the
option
to
choose
between
two
additional
interactive
viewing
modes,
an
interactive
bar
chart
and
interactive
line
chart
[Appendix
D].
Both
charts
gave
users
the
ability
to
view
animations
of
metrics
as
they
changed
over
time
20.
20
Dashboard
Recommendation
Based
on
our
review
of
the
literature
and
interaction
with
stakeholders,
the
following
is
a
sample
of
our
final
dashboard
prototype
recommendation.
The
final
dashboard
can
be
seen
in
its
entirety
in
Appendix
D.
Figure
4:
Sample
Final
Dashboard
Prototype
Further
Recommendations
for
Dashboard
Use
The
new
dashboard
is
an
improved
tool
to
assist
with
DDDM,
but
successful
practice
is
dependent
upon
successful
implementation.
This
will
take
capacity
building,
professional
development,
and
buy-‐in
from
all
stakeholders.
The
following
are
a
set
of
further
recommendations
for
implementation
of
the
dashboard
that
we
feel
will
allow
CAPCS
to
maximize
the
utility
of
this
tool.
1.
Focus
resources
on
building
a
strong
and
supportive
culture
of
data
literacy
and
use.
Creating
a
whole
school
culture
of
data
use
is
important
because
educators
interpret
data
using
existing
beliefs,
values,
assumptions,
and
practices
(Sutherland
2004,
280).
Research
has
found
that
in
order
for
this
to
be
achieved,
a
teacher
should
lead
the
process
and
administrators
should
provide
support
by
promoting
data
use.
Central
office
staff
are
instrumental
in
making
the
concept
of
data
use
well
known,
but
it
seeing
one’s
peer
using
data
regularly
will
encourage
others
to
use
it
in
everyday
practice
(Cho
2014).
Implementation
research
finds
that
teachers
often
respond
to
peers
rather
than
superiors.
In
order
to
ensure
greater
data
literacy
among
teachers
and
administrators,
CAPCS
may
wish
to
increase
access
to
data
and
promote
data
skills
through
quality
professional
development
and
school
policies
(Almy
2014).
This
process
should
be
done
through
tiered
supports
for
varying
levels
of
data
literacy.
There
should
be
an
emphasis
on
developing
the
skills
of
those
who
are
less
literate,
but
the
focus
of
most
resources
should
be
on
integrating
data
into
the
daily
practices
of
all
stakeholders.
This
focus
will
21.
21
help
all
staff
see
how
they
can
use
dashboards
to
go
deep
into
interpretation
to
support
better
student
outcomes
and
reach
charter
goals.
There
was
an
indication
from
the
interviews
that
because
previous
dashboard
implementation
was
not
smooth,
buy-‐in
from
implementers
will
need
to
be
obtained
to
ensure
this
roll
out
has
a
more
positive
outcome.
Most
people
interviewed
were
not
willing
to
spend
more
than
fifteen
seconds
looking
for
the
information
they
need;
therefore,
a
pre-‐existing
familiarity
with
the
dashboard
will
promote
use.
2.
Individualize
dashboards
to
meet
stakeholders’
diverse
needs.
Individuals
consistently
gave
feedback
that
they
would
like
to
see
dashboards
more
specifically
tailored
to
their
needs
in
their
specific
position.
Such
a
structure
would
be
beneficial
and
useful
to
staff
members
in
different
positions
who
make
disparate
types
of
decisions.
Therefore,
a
recommendation
for
the
new
dashboard
is
to
create
a
universal
dashboard
in
addition
to
dashboards
that
contain
subject-‐
specific
data
such
as
ELL,
SPED,
math,
and
reading.
These
specialized
dashboards
would
contain
less
data
that
are
irrelevant
to
certain
stakeholders’
needs
and
therefore
those
stakeholders
would
be
more
likely
to
use
them
for
decision
making.
This
can
be
facilitated
by
the
use
of
the
Google
dashboard
prototype,
which
is
the
easiest
and
least
time
consuming
way
to
customize
data
and
give
all
stakeholders
independent
access
to
the
specific
information
they
need.
3.
Standardize
protocol
for
dashboard
dissemination
and
create
regular
space
for
data
analysis
and
collaboration.
Standard
protocol
for
dashboard
distribution
is
key
to
effective
implementation.
Stakeholder
feedback
indicates
that
dashboard
delivery
should
occur
at
a
consistent
time
every
week.
This
would
allow
individuals
to
plan
and
budget
time
to
review
the
data
weekly
and
be
prepared
for
professional
development
sessions
and
data
discussion
meetings.
Creating
consistency
for
distribution
will
reinforce
data
use
as
a
regular
part
of
stakeholders’
routines
and
help
foster
a
culture
of
data
use.
One
of
the
most
important
factors
considered
during
the
creation
of
the
data
dashboard
was
the
ease
of
access
to
clear
and
actionable
data.
CAPCS
has
an
extended
school
day,
meaning
there
is
limited
time
for
teacher
professional
development
during
the
day.
This
makes
it
even
more
essential
to
ensure
that
the
time
spent
working
with
data
dashboards
is
productive.
Based
on
feedback
from
stakeholders,
it
would
be
beneficial
to
use
professional
development
to
give
a
basic
overview
of
the
dashboard
and
how
to
use
it
quickly
and
effectively.
For
instance,
the
data
meetings
and
conferences
that
CAPCS
holds
could
be
scheduled
regularly
to
coincide
with
the
release
of
the
dashboard.
4.
Continue
to
improve
dashboard
and
data
systems
as
needs
and
culture
at
CAPCS
evolve.
A
thoughtful
and
well-‐executed
implementation
of
the
new
dashboard
is
critical
for
success,
but
the
process
for
improved
data
use
does
not
stop
once
the
new
dashboard
is
in
place.
After
the
roll
out
of
new
dashboards,
Mr.
Welch
and
the
CAPCS
Data
Associate
should
continue
to
collect
feedback
from
stakeholder
groups.
This
feedback
can
be
used
in
an
iterative
process
of
continuous
improvement.
As
interventions,
school
performance
data,
staff,
and
internal
culture
change,
this
should
be
reflected
in
the
dashboards
and
their
delivery.
22.
22
Conclusion
The
purpose
of
this
project
was
to
facilitate
the
decision
making
process
of
stakeholders
at
Community
Academy
Public
Charter
Schools
(CAPCS)
by
creating
an
updated
data
dashboard.
To
understand
the
needs
and
data
literacy
levels
of
stakeholders
at
different
levels,
we
first
used
research
on
data-‐driven
decision
making
(DDDM)
and
conversations
with
the
CAPCS
liaison
to
develop
an
initial
dashboard
prototype.
We
then
conducted
twelve
semi-‐structured
in-‐person
interviews,
during
which
we
showed
each
stakeholder
three
dashboards:
the
current
tool
being
used
by
CAPCS,
a
board
summary
document,
and
our
initial
prototype.
In
terms
of
dashboard
design
and
information
displayed,
we
found
that
stakeholders
were
reluctant
to
spend
long
periods
of
time
reviewing
a
dashboard.
Stakeholders
favored
visual
indicators
that
specified
when
metrics
had
increased
or
decreased
from
the
previous
period.
In
addition
to
accountability
metrics,
which
relate
to
students’
overall
proficiency
in
a
subject
area,
stakeholders
also
suggested
that
metrics
related
to
specific
subject
area
skills
would
provide
more
actionable
insight.
We
also
found
that
the
majority
of
stakeholders
use
data
to
inform
their
work
multiple
times
a
week,
which
shows
that
CAPCS
has
a
basic
culture
of
DDDM
and
data
literacy.
This
report
provides
additional
recommendations
and
promising
practices
to
assist
CAPCS
in
improving
decision
making.
35.
35
Appendix
E:
Interview
Protocol
and
Script
5/4/2014 Interview Questions - Google Forms
https://docs.google.com/forms/d/1MYhT_zVC9fIn4emK7_znSxXsELZoPZ8DA0Idj0q3Anw/edit 1/9
Interview Questions
Intro: Good morning/afternoon. Thank you for taking the time to meet with me today. As you might
know, I am part of a group of GW students working with CAPCS as part of our capstone project for our
Master’s Degree. We are helping to redesign a data dashboard that can be used to help a variety of
people in the CAPCS community get a good understanding of what is going on at the schools. Your
input will help us to create the most useful tool for CAPCS. You can stop this interview or ask me to
repeat a question at any time.
This interview should take about 20 minutes to complete. We are looking for really honest feedback
about the current tools and the prototype that we’ve created. All of your answers will be completely
confidential, and it is only through collecting this feedback that we can create the best dashboard
possible. So please be as honest as you can as we go through these questions.
The prototype we created contains fabricated data and is for display purposes only. I'd like to record this
interview, unless you have any objections.
Do you have any questions for me before we begin?
Great, then let’s get started.
1. Name
2. Title
3. Department
Mark only one oval.
Amos 1
Amos 2
Amos 3
Butler
Central Office
Board
Other:
4. Date of Interview
Example: December 15, 2012
