1. The document discusses new developments in educational assessments, focusing on three key points: the increasing importance of assessments, blurred lines between assessment types, and the rise of computer-based testing. 2. It provides definitions for validity, reliability, and different assessment types from standards documents and discusses attributes important for different assessments. 3. Examples of computer-based testing programs from consortia are described along with pros and cons, and a sample interim assessment report is shown, highlighting growth measurement and reporting features.

1. A New Generation of Assessments
Three Things You Need to Know
October 30, 2013
Sean McGrew, Ph.D.

2. Poll 1: What is this?
A: A tool for chopping wood
B: A murder weapon

3. Three Things to Know
1. Everything depends on use of assessments
2. Assessment distinctions have fuzzy borders
3. Computer-based testing is (finally) changing the
practice

4. Validity in the Context of Use
“Validity refers to the degree to which evidence and theory
support the interpretations of test scores entailed by
proposed uses of tests. Validity is, therefore, the most
fundamental consideration in developing and evaluating
tests. The process of validation involves accumulating
evidence to provide a sound scientic basis for the proposed
score interpretations. It is the interpretations of test scores
required by proposed uses that are evaluated, not the test
itself.” AERA/APA/NCME Standards for Educational and
Psychological Testing, 1999.

5. Reliability
Reliability “refers to the consistency of . . .
measurements when the testing procedure is
repeated on a population of individuals or
groups” (AERA, 1999)

7. Poll 2: What’s the most important attribute?
What’s the most important attribute of an educational
assessment?
 Utility
 Validity
 Reliability
 It depends

8. Fuzzy Borders
Formative vs. Summative
Norm-Referenced vs. Criterion-Referenced
Interim/Benchmark/Formative
Test/Assessment

9. Poll 3: OK: Of a formative or diagnostic?
What’s the most important attribute of an diagnostic of
formative assessment used to diagnose skill deficits and/or
inform instruction?
 Utility
 Validity
 Reliability

10. Poll 4: Of a high-stakes test?
What’s the most important attribute of a high-stakes test
used to rank or rate students?
 Utility
 Validity
 Reliability

11. Poll 5: Reporting Categories Important to You

12. Poll 6: Scores

13. Poll 7: Logistics

14. Computer-Based Coming of Age
PROS
 Easy to edit/update
 Rapid scoring and
reporting
 Possibility of
multimedia content
 Possibility of adaptive
testing
CONS
 Hardware requirements
 Often need internet
access
 Assumes comfort level
with computers
 Harder to ensure same
experience for all
students
 Less proven metrics for
reliability (if get away
from multi-choice)

15. Not all Computer-Based tests are Adaptive
Adaptive Tests
 Basic idea goes back to Binet’s 1905 intelligence tests
 First done by computers in 1970s and 1980s
 Widespread by late 1990s
PROS
 Shorter
 More accurate for wider
range of students
CONS
 Harder to know ‘what’s on
the test’
 Harder to review item and
standard results for a group
of students

16. Common Core Test Consortia
 Smarter Balanced: Computer-Based ADAPTIVE
 PARCC: Computer-Based, NOT ADAPTIVE

17. Next-Generation Reporting
 Focus on Longitudinal Analysis
 Teachers expected to incorporate results provided online

18. GROWTH

19. Catapult Learning’s Common Core
INTERIM TEST REPORTING TOOL

20. 21

21. 22

22. 23

23. Grade 3 Assessment Blueprint
Standard
2.OA.A.1: Use addition and subtraction within 100 to solve
one- and two-step word problems.
2.OA.C.4: Use addition to find the total number of objects
arranged in rectangular arrays with up to 5 rows and 5
columns.
2.OA.B.2: Fluently add and subtract within 20 using mental
strategies
3.OA.A.2: Interpret whole-number quotients of whole
numbers
3.OA.A.3: Use multiplication and division within 100 to solve
word problems in situations involving equal groups, arrays,
and measurement quantities.
3.OA.C.7: Fluently multiply and divide within 100
Section
Number
of Items
Pre-req
4
Pre-req
4
Pre-req
4
Unit 1: A,B
4
Unit 1: A,B
Unit 1: A,B
4
8
Total
28 items
24

24. To teach, or re-teach?
To teach, or else re-teach, that is the question:
Whether ’tis Nobler in the mind to suffer
The Slings and Arrows of outraged Bosses,
Or to take Arms against student weaknesses,
And by opposing end them: Re-teach, press on
No more; and by Re-teach, to say we end
The Heart-ache, and the thousand Natural shocks
Common Core enforces. ’Tis a consummation
Devoutly to be wished. Re-teach, press on,
Press on, perchance to fail; Aye, there’s the rub.
Adapted from William Shakespeare, Hamlet III.i

25. Connect with Us!
 Stay connected with our latest updates, blog posts,
upcoming webinars, and more.
 Catapult Learning on Twitter @catapultlearn
 Dr. Sean McGrew on Twitter @SeanPMGrew
 Share your successes! If you put our methods into action,
let us know about it. Use the hashtag #catapulting
26