John Cronin's presentation: Multiple Measures Done Right - The Purpose Driven Assessment System at the January, 2017 COSA conference

1. Multiple measures done right: The purpose-
driven assessment system
John Cronin, Ph.D.
VP – Education Research
Northwest Evaluation Association

2. The pursuit of compliance is
exhausting because it is always a
moving target. Governors move
on, the party in power gets
replaced, a new president is
elected, and all want to put their
own stamp on education.
It is saner and less exhausting to
define your own course and align
compliance requirements to that.
The problem of compliance-based assessment systems

3. Seven standards that define the
purpose driven assessment system

4. The purposes of all assessments are
defined and the assessments are valid and
useful for their purposes.
DEFINE YOUR PURPOSES

5. Northwest Evaluation
Association and Gallup - 2016
https://www.nwea.org/content/uploads
/2016/05/Make_Assessment_Work_for_
All_Students_2016.pdf
Make Assessment
Work for All Students:
Multiple Measures
Matter

6. TEACHERS, PRINCIPALS AND
SUPERINTENDENTS say students spend
too much time taking assessments.

7. Percent of students who say the results
from state accountability tests are useful
29
Gallup (2016, May). Make Assessment Work for All Students: Multiple Assessments Matter.
https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf

9. Teacher and Administrator perceptions on the
purposes of assessments
0%
10%
20%
30%
40%
50%
60%
70%
80%
State
Policymakers
Parents Students Teachers Administrators
% State accountability assessments % Classroom tests and quizzes
Principals: Do you believe the purpose of data from (state accountability assessments,
classroom assessments and quizzes) in your district is well understood by each of the following:

10. Students find many assessment types useful when
the assessment is tied to a specific purpose
Classroom tests and quizzes developed by teachers to
evaluate their students’ learning process
Interim assessments that are given to students two to three
times over the year so teachers can see where students are
growing and where they need to learn more
Performance tasks that require students to apply things they
have learned to a real-world problem or situation
Formative assessments
78%
76%
75%
74%

11. One action to take
First, document and prioritize your
purposes for assessment then…
inventory the assessments in use and
evaluate their alignment with your
purposes.
It’s a big job but worth the time!

12. Teachers are educated in the proper
administration and application of the
assessments used in their classrooms.
EDUCATE TEACHERS

13. What types of extra help, if any, does your teacher
or school give you if you do not do well on a…
CLASSROOM TEST STATE STANDARDIZED TEST
Before and/or after school support 43% 24%
One to one support 32% 19%
Access to added resources 29% 22%
Small group support 24% 17%
No extra support provided 20% 38%

14. Teachers need to …
Understand the purpose
Follow the procedures
Properly interpret and act

15. One action to take
Develop internal structures and policies to provide
training for all teachers (and administrators) in your
assessments and promote proper use. This training
should help educators understand what the results
can and can’t tell them, and how best to use it to
make a difference for students.

16. Assessment results are aligned to the
needs of their audiences.
ALIGN RESULTS

17. Assessment
audiences
• School Board
• Students
• Teachers
• Parents
• Principals, school
administrators, and teacher
leaders
• District Administrators
• Community members
• State and federal officials

18. Percentage of parents who say teachers
rarely or never discuss their child’s
assessment results with them
61
Gallup (2016, May). Make Assessment Work for All Students: Multiple Assessments Matter.
https://www.nwea.org/content/uploads/2016/05/Make_Assessment_Work_for_All_Students_2016.pdf

19. Questions parents
want answered
from assessment
• Core question - Do you know
and care for my child?
• What are my child’s strengths
and weaknesses?
• Is my child on track for the
next grade level?
• Is my child on track for
college?
• Is my child showing
improvement?
• Should I be concerned?

20. What kind of data do parents want?
95%
95%
93%
92%
91%
90%
88%
84%
79%
77%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Monitoring general progress
Knowing when to be concerned
Determining readiness for next grade
Knowing if I need to seek extra help
Monitoring standards
Communicating with teacher/admin
Measuring high quality teaching
Helping my child with homework
Adjust content to student needs
Providing activities for home
Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions Study.
Survey conducted by Grunwald Associates LLC

21. Questions teachers
want answered
from assessment
• What does each student
know and not know?
• What does this student need
to learn next?
• What resources will help this
student?
• How can I group these
students for instruction?

22. One action to take
Train teachers in the skills needed to have a
positive and productive parent conversation.
Many teachers find these conversations
intimidating. When educators are well prepared
they gain confidence and develop higher levels
of trust and support from most parents.

23. Redundant, mis-aligned, or unused
assessments are eliminated.
STREAMLINE MEASURES

24. Your current assessment tools
•What purposes do each assessment meet well?
• Comes from assessment design and capabilities –
not your current use
• Is there alignment between the assessment and the
information needed?
•How are you using the assessments?
• Reduce use to within the design and capability
• Use assessments more fully to meet other purposes

25. One action to take
Identify the places where you have multiple
assessments in-use validly performing the same
purpose. Determine whether you can leverage
other assessments to meet these needs.

26. Assessment results are delivered in a
timely and useful manner.
DELIVER RESULTS

27. Percent of students
who say they do
not receive their
state accountability
test results.
37%

28. For parents, assessment results
begin losing their relevance
within…
1month
after the assessments are
administered
Source – Northwest Evaluation Association (2012) – NWEA Assessment Perceptions Study. Survey conducted
by Grunwald Associates LLC
https://www.nwea.org/content/uploads/2014/07/NWEAGRUNWALD_Assessment_Perceptions_b.pdf

29. Timely results are related to use
If needed for instruction, results need to be faster than
school improvement conversations
Reports designed to fulfill purpose
• School administrators want reports that quickly
identify students who need help.
• Teachers want reports that help them make
instructional decisions.
• Parents want to know whether there student is on
track and to be warned when trouble is ahead.

30. One action to take
Review your assessment calendar, with a focus
on when data is being delivered to your
audiences. Adjust your testing calendar or
practices to ensure that data gets to educators
when they most need it. Make sure assessment
results are timed to fit your schedule for
professional days to encourage dialogue.

31. The metrics and incentives used encourage
a focus on all learners.
SUPPORT ALL LEARNERS

32. Grade Students Athletes % Fine Arts %
9 3276 1487 50% 2256 76%
10 3190 1215 41% 1656 56%
11 2967 932 31% 1233 42%
12 2795 703 24% 1028 35%
Percent of students participating in athletics
and fine arts programs

33. What gets measured and attended to really does matter
0
5
10
15
20
25
30
35
40
165
167
173
176
181
185
187
189
191
194
196
198
200
202
204
206
208
210
212
214
216
218
220
222
224
226
228
230
232
234
236
238
240
242
244
246
248
250
252
254
257
262
NumberofStudents
Fall RIT
Mathematics
No Change
Down
Up
Proficiency College Readiness
One district’s change in 5th grade mathematics performance relative to the KY proficiency cut scores

34. Changing from Proficiency to Growth means all kids matter
0
5
10
15
20
25
30
166
169
175
178
183
186
188
190
193
195
197
199
201
203
205
207
209
211
213
215
217
219
221
223
225
227
229
231
233
235
237
239
241
243
245
247
249
251
253
255
258
266
NumberofStudents
Student’s score in fall
Mathematics
Below projected growth
Met or above projected growth
Number of 5th grade students meeting projected mathematics growth in the same district

35. One action to take
Ask the question – “What 3 to 5 metrics drive
educational decision making in our school or
district? Now ask – “What behavior do those
metrics incentivize and are all students
encompassed in those metrics?
Make changes as needed to be sure your
metrics encourage focus on all students.

36. The assessment program contributes to a
climate of transparency and objectivity
with a long-term focus
INVITE COLLABORATION

37. “Everyone is entitled to their opinion, but
not to their own facts.”
Daniel Patrick Moynihan

38. What is accountability?
Accountability is a dialogue between the
stakeholders and the leaders of their schools. Part of
that dialogue is understanding and aligning the goals
and objectives of the parents, the schools, and the
larger community. Another part of that dialogue is
discussing how your schools are doing in reaching
those goals.
The most important part of the dialogue is what
you’re doing to improve performance based on
this information, which is leading.

39. The four principles that guide your
communications to your audiences.
• Clearly identify who your audience is and address
them as your partner.
• Understand what that audience wants to know
about your schools and their performance.
• Be transparent about how you are performing on
these objectives.
• Communicate your strategy to improve
performance, and corrective action when strategy
fails.

40. Florida
District
Highly
Effective
Effective Needs
Improvement
Developing Unsatisfactory VA Score Florida
Ranking
1 44.4% 55.6% 0.0% 0.0% 0.0% 0.39 109
2 25.0% 75.0% 0.0% 0.0% 0.0% 0.37 121
3 90.9% 9.1% 0.0% 0.0% 0.0% -0.14 2802
4 60.7% 39.3% 0.0% 0.0% 0.0% -0.14 2797
5 81.2% 18.8% 0.0% 0.0% 0.0% -0.16 2831
6 37.3% 54.2% 1.7% 0.0% 6.8% 0.12 880
7 81.3% 18.8% 0.0% 0.0% 0.0% 0.22 402
8 41.7% 55.6% 1.4% 1.4% 0.0% -0.34 3274
9 52.2% 47.8% 0.0% 0.0% 0.0% 0.16 664
10 27.0% 66.2% 1.4% 0.0% 5.4% 0 1764
11 7.1% 72.6% 9.5% 10.7% 0.0% -0.08 2445
Teacher Evaluation Ratings in Eleven Florida
Schools 2013

41. Source: Aviv, R (2014, July 21). Wrong Answer. The New Yorker. Retrieved on June
16, 2016 from http://www.newyorker.com/magazine/2014/07/21/wrong-answer
“After more than two thousand interviews, the investigators
concluded that forty-four schools had cheated and that a
“culture of fear, intimidation and retaliation has infested the
district, allowing cheating—at all levels—to go unchecked for
years.” They wrote that data had been “used as an abusive
and cruel weapon to embarrass and punish.” Several
teachers had been told that they had a choice: either make
targets or be placed on a Performance Development Plan,
which was often a precursor to termination. At one
elementary school, during a faculty meeting, a principal
forced a teacher whose students had tested poorly to crawl
under the table.”

42. Differences in fall-spring test durations
An illustration of gaming
15%
25%
60%
Mathematics
Spring < Fall Spring = Fall Spring > Fall
0.0
1.0
2.0
3.0
4.0
5.0
6.0
Spring < Fall Spring = Fall Spring > Fall
GrowthIndex
Mathematics
Differences in growth index score based
on fall-spring test durations

43. One action to take
Review the tone of a few of your recent
communications to stakeholders about student
learning. As you read these does it feel like your
touting, or are you truly educating your partners?
Try writing an op-ed about your educational
program that would surprise readers because of
its honesty and frankness.

44. A purposeful assessment program
assessment system ensures you
have direction, that your direction
stays true, and that you and your
students eventually reach their
intended destination

45. Thank you for joining us!
http://info.nwea.org/MMDR-Guide-LP.html?utm_medium=press-
release&utm_source=rss&utm_campaign=multiple-
measures&utm_content=text-link
Multiple Measures Done Right – can be downloaded
from