DSPy a system for AI to Write Prompts and Do Fine Tuning
The abc's of cbm for maths, spelling and writing
1. The ABCs of CBM for
Math, Spelling, & Writing
Michelle Hosp, Ph.D.
2. Objectives
• Overview of Curriculum-Based
Measurement (CBM)
• Review CBM measures in: math, spelling,
and writing (including writing IEP goals and objectives)
• Cover criteria for progress monitoring in
math and spelling
• Provide information on where to obtain
measures for math, spelling, and writing
2
3. CBM Research
• CBM research has been conducted over
the past 25+ years
• Research has demonstrated that when
teachers use CBM for instructional
decision making:
♦ Students learn more
♦ Teacher decision making improves
♦ Students are more aware of their performance
3
5. Salient Features of
Mastery Measurement
• Curriculum is broken down into specific
subskills or short-term instructional
objectives
• Assess specific skill that is being taught
Example
♦ Multidigit addition, with regrouping
• Skills usually assessed using teacher-
made tests or tests in curriculum
5
6. Fourth Grade Math
Computation Curriculum
1. Multidigit addition with regrouping
2. Multidigit subtraction with regrouping
3. Multiplication facts, factors to 9
4. Multiply 2-digit numbers by a 1-digit number
5. Multiply 2-digit numbers by a 2-digit number
6. Division facts, divisors to 9
7. Divide 2-digit numbers by a 1-digit number
8. Divide 3-digit numbers by a 1-digit number
9. Add/subtract simple fractions, like denominators
10. Add/subtract whole number and mixed number
6
9. Fourth Grade Math Computation
Curriculum
1. Multidigit addition with regrouping
2. Multidigit subtraction with regrouping
3. Multiplication facts, factors to 9
4. Multiply 2-digit numbers by a 1-digit number
5. Multiply 2-digit numbers by a 2-digit number
6. Division facts, divisors to 9
7. Divide 2-digit numbers by a 1-digit number
8. Divide 3-digit numbers by a 1-digit number
9. Add/subtract simple fractions, like denominators
10. Add/subtract whole number and mixed number
9
12. Downsides to
Mastery Measurement
• Skill Hierarchies
• Teacher-Made Tests
♦ Reliability & Validity are unknown
• Retention & generalization of skills are not usually
measured
• Measurement of Short-Term Instructional Objectives
• Measurement shifts occur making it difficult to
monitor overall progress because:
1. different skills are measured at different points in
time
2. different skills are not of equal difficulty and do not
represent equal curriculum units
12
13. Most Forms of Classroom
Assessment Are Mastery
Measurement
CBM is NOT
Mastery Measurement
CBM is a General
Outcome Measure
14. Fourth Grade Math
Computation Curriculum
1. Multidigit addition with regrouping
2. Multidigit subtraction with regrouping
3. Multiplication facts, factors to 9
4. Multiply 2-digit numbers by a 1-digit number
5. Multiply 2-digit numbers by a 2-digit number
6. Division facts, divisors to 9
7. Divide 2-digit numbers by a 1-digit number
8. Divide 3-digit numbers by a 1-digit number
9. Add/subtract simple fractions, like denominators
10. Add/subtract whole number and mixed number
14
15. • Random
Sheet #1 Computation 4
Password: ARM
Name: Date
numerals A
3
7
2
7
=
B
16 + 3 =
7
C
4) 6
D
6 )7 8
E
87 5
within
x 7
problems
(considering
F G H I J
6 9 24 4
x7 x0 6 )48 5 )2 0
x 7
specifications
of problem K L M N O
types)
2 )50 61 44 33 6 7 )3 0
44 20 x 10 x0
• Random
P Q R S T
95 22 5 74 - 2=
8 )3 2 11 56 7 38
+ 75 26 8 28 24 x 33
+ 83
placement of
problem U
3 + 1
V W X Y
types on
9
5 5 = 98 2
x5
4
7 )56
97 x1
page 15
16. Sheet #2 Computation 4
Password: AIR
• Random
Name: Date
A B C D E
numerals within
9 )2 4 52 85 2 9 4 )7 2 82 85
+ 64 70 8 x0 43 04
+ 90
problems
(considering
F G H I J
6 )3 0 35 4 7 2 1
x 74 x5 x9 3 3 =
specifications of
problem types) K L M N O
32 8 5 )6 5 6 )30 34 - 1=
x 23 x6 7
• Random P Q R S T
placement of 10 7
x 3
2) 9 41 6
44
5 + 3
11 11 =
6
x2
problem types
on page U V
15 04
W X Y
41 + 6 = 14 41 9 )8 1 13 0
5 )1 0
2 x 7
16
17. Donald’s Progress in Digits Correct
Across the School Year
Instructional
Change
25
20
15
Correct Digits
10
5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
17
Weeks
18. Salient Features of
General Outcome Measurement
• General domains, not subskills
♦ Keeps global curriculum outcomes intact
and uses long-term goals
• Makes no assumptions about
instructional hierarchy for determining
measurement (i.e., CBM fits with any
instructional approach)
♦ No measurement shifts
18
19. Salient Features of
General Outcome Measurement (cont)
• Incorporates automatic tests of
retention and generalization
♦ Measurement of Long-Term Curricular
Goal Performance
• Test Construction
♦ Standardized procedures used to assess
performance on the long-term goal
♦ Reliability & validity can be determined
19
20. Downsides to
General Outcome Measurement
• Often lacks information on specific
subskills
♦ If interested in identifying specific skills to
teach, GOM not appropriate
♦ Need to use a diagnostic measure
• Fidelity of implementation is important
20
22. CBM and Math
• The number of correctly written digits
in 2 minutes from the end-of-year
curriculum
• Correct digits
♦ Not correct problems or answers
♦ 2 minutes
22
23. Math CBM
• Student(s) are given a sheet of math
problems and pencil
• Student(s) complete as many math
problems as they can in 2 minutes
• At the end of 2 minutes the number of
correctly written digits is counted
23
24. Example of a 4th grade math
curriculum
1. Multidigit addition with regrouping
2. Multidigit subtraction with regrouping
3. Multiplication facts, factors to 9
4. Multiply 3-digit numbers by a 1-digit number
5. Multiply 2-digit numbers by a 2-digit number
6. Division facts, divisors to 9
7. Divide 2-digit numbers by a 1-digit number
8. Add/subtract simple fractions, like denominators
9. Add/subtract whole number and mixed number
24
25. Sheet #1 Computation 4
Password: ARM
Name: Date
A B C D E
3 2 6
= 1 + 3=
7 4) 6 6 )7 8 87 5
7 7 x 7
F G H I J
6 9 24 4 6 )48 5 )2 0
x7 x0 x 7
K L M N O
2 )50 61 44 33 6 7 )3 0
44 20 x 10 x0
P Q R S T
95 22 5 74 - 2=
8 )3 2 11 56 7 38
+ 75 26 8 28 24 x 33
+ 83
U V W X Y
3 + 1 9
5 =
98 2 4
5 x5 7 )56
97 x1
25
26. Math scoring criteria
• If the answer is correct, the student
earns the score equivalent to the
number of correct digits written using
the “longest method” taught to solve the
problem, even if the work is not shown
• If a problem has been crossed out,
credit is given for the correct digits
written
• If the problem has not been completed,
credit is earned for any correct digits
written 26
27. A “correct digit” is the right numeral
in the right place
4507 4507 4507
2146 2146 2146
2361 2461 2441
4 3 2
correct correct correct
digits digits digits
27
31. Weekly growth rates for math
(Correct Digits [CD] in 2 minutes)
Grade Realistic Weekly Ambitious Weekly
Growth Rate Growth Rate
1 .3 CD .5 CD
2 .3 CD .5 CD
3 .3 CD .5 CD
4 .70 CD 1.15 CD
5 .75 CD 1.20 CD
6 .45 CD 1 CD
From Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)
31
32. How often?
• Progress Monitoring (Formative)
♦ 1x Week for at-risk & students with
disabilities
♦ 1x Month for typically developing students
♦ 1x Quarter for above average students
• Benchmarking/ Norming (Summative)
♦ 1x Quarter for all students
• Survey Level (Summative)
♦ 1x At the beginning of progress monitoring
♦ 1x Identify students’ instructional level
32
34. IEP Goals & Objectives
• Time (the amount of time the goal is written
for)
♦ “In 1 year…”
• Learner (the student for whom the goal is
being written)
♦ “...Jose will…”
• Behavior (the specific skill the student will
demonstrate)
♦ “…read aloud…”
• Level (the grade the content is from)
♦ “… a second-grade…”
34
35. IEP Goals & Objectives (cont)
• Content (what the student is learning about)
♦ “…reading…”
• Material (what the student is using)
♦ “…passage from ORF CBM progress-monitoring
material…”
• Criteria (the expected level of performance,
including time and accuracy)
♦ “…at 90 words read correctly in 1 minute with
greater than 95% accuracy.”
35
36. Math goals & objectives
• In 30 weeks, Larry will calculate addition and
subtraction problems from second-grade
mixed-math CBM progress-monitoring
material at 45 correct digits in 2 minutes with
greater than 95% accuracy.
• In 10 weeks, Larry will calculate addition and
subtraction problems from second-grade
mixed-math CBM progress-monitoring
material at 20 CD in 2 minutes with greater
than 95% accuracy.
36
38. Spelling CBM
1. Student(s) are given a blank sheet of
lined paper
2. Teacher dictates a spelling word every
10 seconds (grades 1-3) every 7
seconds (grades 4-8)
3. Stop at the end of 2 minutes and count
the number of correct letter sequences
(CLS)
38
40. Conducting Spelling CBM
• Say each word twice. Use homonyms in a
sentence.
Read. He read the book.
• Say a new word every 10 (or 7) seconds
♦ 12-13 words for grades 1-3
♦ 17-18 words for grades 4-8
• Dictate words for 2 minutes.
40
42. How often?
• Progress Monitoring (Formative)
♦ 1x Week for at-risk & students with
disabilities
♦ 1x Month for typically developing students
♦ 1x Quarter for above average students
• Benchmarking/ Norming (Summative)
♦ 1x Quarter for all students
42
43. Spelling CBM Goals and
Objectives
• In 30 weeks, Roberto will spell words from a
fourth-grade spelling list from Spelling CBM
progress-monitoring material at 70 correct
letter sequences in 2 minutes with greater
than 95% accuracy.
• In 10 weeks, Roberto will spell words from a
fourth-grade spelling list from Spelling CBM
progress-monitoring material at 25 correct
letter sequences in 2 minutes with greater
than 95% accuracy.
43
45. Curriculum-Based Measurement:
Written Expression
• Provides an indicator of student
performance in writing
• Three scoring methods
♦ Total Words Written (TWW)
♦ Words Spelled Correctly (WSC)
♦ Correct Writing Sequences (CWS)
• Can be group administered
• District or Classroom Norms
45
46. PRIMARY STORY STARTERS
1. The best birthday I ever had was…
2. It was a warm sunny day when the two boys…
3. A big blue turtle was coming down the street when
he…
4. Yesterday the children went on a picnic and…
5. The noise came so suddenly that…
6. It was raining with the wind blowing when…
7. The fog was so thick I could hardly see…
8. Mickey Mouse came to my birthday party and…
9. The cat climbed the telephone pole and…
10. I knew it was cold when…
46
47. INTERMEDIATE STORY
STARTERS
1. She’s gone, now I’m going to find…
2. Mary knew that if her parents found out, they…
3. Everything was just fine, until I met…
4. Something’s coming out of the sink and it’s a…
5. The children were playing on the each when they found the strange
footprints of a…
6. One day I lost my dog when…
7. “Will you keep quiet,” whispered Bob, “if you don’t someone will…
8. I opened the door and found a huge wooden crate and…
9. The magician pulled a white rabbit out of his hat instead of a…
10. A spaceship landed in my backyard and…
47
48. ADVANCED STORY STARTERS
1. At first the noise was very faint and seemed far
away, but then…
2. Mary knew that if her parents found out, they…
3. It all began in the laboratory of Professor Hall
when…
4. The teenagers were walking along the beach when
they found the strange footprints of…
5. I woke up one morning feeling very strange when I
noticed…
6. “Can you keep a secret,” whispered Joe. “No one
else knows that…
7. Buried beneath a tree with just a bit of it showing
was…
8. I knew it was going to be one of thos4 days when…
9. I suppose I shouldn’t have laughed, but I couldn’t
help myself when… 48
49. Total Words Written (TWW)
• The total number of words written regardless of
spelling or context.
• Abbreviations:
♦ Commonly used abbreviations are counted as words
• Hyphenated Words:
♦ Each morpheme separated by a hyphen(s) is counted as
an individual word if it can stand alone.
• Story Titles
♦ Words written in the title are counted as words written
49
53. Correct Writing Sequences
(CWS)
• Two adjacent writing units (word/word or
word/punctuation) that are acceptable
within the context of what is written
• Correct spelling, syntax, and semantics
are taken into account when scoring
Correct Writing Sequences
53
56. How often?
• Progress Monitoring (Formative)
♦ 1x Week for at-risk & students with
disabilities
♦ 1x Month for typically developing students
♦ 1x Quarter for above average students
• Benchmarking/ Norming (Summative)
♦ 1x Quarter for all students
56
57. Written Expression IEP Goals
and Objectives
• In 30 weeks, Jose will write from sixth-
grade writing story starter CBM
progress-monitoring material at 47
correct writing sequences in 3 minutes
with greater than 95% accuracy.
• In 10 weeks, Jose will write from sixth-
grade writing story starter CBM
progress-monitoring material at 30
correct writing sequences in 3 minutes
with greater than 95% accuracy. 57
(Introduction of Presenters….) Today we will be talking about one form of Progress Monitoring: Curriculum-Based Measurement, or CBM.
Research has demonstrated that when teachers use CBM to inform their instructional decision making, students learn more, teacher decision making improves, and students are more aware of their own performance (e.g., Fuchs, Deno, & Mirkin, 1984). CBM research, conducted over the past 30 years, has also shown CBM to be reliable and valid (e.g., Deno, 1985; Germann & Tindal, 1985; Marston, 1988; Shinn, 1989). A more in-depth bibliography of CBM research is available in the CBM manual. Resources: Deno, S.L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52, 219-232. Fuchs, L.S., Deno, S.L., & Mirkin, P.K. (1984). Effects of frequent curriculum-based measurement of evaluation on pedagogy, student achievement, and student awareness of learning. American Educational Research Journal, 21, 449-460. Germann G., & Tindal, G. (1985). An application on curriculum-based assessment: The use of direct and repeated measurement. Exceptional Children, 52 , 244-265. Marston, D. (1988). The effectiveness of special education: A time-series analysis of reading performance in regular and special education settings. The Journal of Special Education, 21, 13-26. Shinn, M.R. (Ed.). (1989). Curriculum-based measurement: Assessing special children. New York: Guilford Press.
Specific subskill testing relies on mastery measurement where small domains of test items and mastery criteria are specified for each subskill. For example, a teacher wants to increase a students spelling proficiency so she needs to do two things. One, determine the subskill hierarchy for the spelling curriculum she is using and Two, design a criterion-referenced testing procedure to match each step on the hierarchy. These skills are then taught in sequence.
Specific subskill testing relies on mastery measurement where small domains of test items and mastery criteria are specified for each subskill. For example, a teacher wants to increase a students spelling proficiency so she needs to do two things. One, determine the subskill hierarchy for the spelling curriculum she is using and Two, design a criterion-referenced testing procedure to match each step on the hierarchy. These skills are then taught in sequence.
Skill Hierarchies, use a scope and sequence for instruction (usually based on theory not empirical evidence) Teacher-Made Tests, will have no technical adequacy to support them, while commercial criterion-referenced tests are no better. Not to mention the time and money it takes for teacher to make these tests. Retention and Generalization, is not assessed because only one skill in the hierarchy is assessed at any one time. Measurement of Short-Term Instructional Objectives, are always closely linked. This is problematic because the skill is only assessed on that one task and it may not generalize to other related tasks (only near transfer and not far transfer). Measurement Shifts, must occur each time a skill is mastered. So new assessments will always need to be developed and scores will drop each time a new skill is assessed until that student has mastered the skill.
CBM is USUALLY not mastery measurement but some math probes that look at specific skills and even some early reading probes like letter sounds may be considered mastery measurement because they assess a specific skill.
Skill Domains, instead the teacher identifies the domain they want to measure throughout the year (Typically this is the curriculum content for one school year). Retention and Generalization, the skills being assessed represent the current instructional focus along with those representing past and future instructional targets. Measurement for Long-Term Curricular Goal Performance, means the assessment is less sensitive to acquisition of the specific skills currently taught BUT it is sensitive enough for instructional decision making. And because it describes student performance in terms of proficiency on the critical behavior that are broadly indicative of the annual curriculum, content and criterion validity are high relative to mastery measurement. Measurement Shifts, do not occur because the difficulty of the assessment remains constant across the school year. So there are not high and then low points the student will experience. AND student growth can be indexed over time Test Construction, general outcome measurement uses a standardized way of obtaining assessment material, administering and scoring tests and summarizing and evaluating scores. It has documented reliability and validity which mastery measurement lacks.
Math can be done really well, spelling okay too. Reading and writing—NO. Need to make sure that standardization procedures are followed. Consistency is key.
Like spelling and writing, math can be administered to a group rather than individually. Why digits & not problems? Can only fit 20 or so problems per page, whereas each problem can have 1-20 digits in the correct, longest for answer. Provides a much more sensitive measure of growth.
Identify the skills: how many of each?
Difference in errors: 3CD—borrowed correctly in part, forgot the 5 was now a 4. 2CD—doesn’t get borrowing. Show slashing through wrong digits.
Practice finding median and plotting on SLA forms in binder. X = WRC O = Errors
Survey level assessment is only done when there is a reason to- it is not done on a specific schedule.
Teachers should assess students at least 1 time per week preferably 2 to 3.
Homonym, same pronunciation, different meaning.
Survey level assessment is only done when there is a reason to- it is not done on a specific schedule.
Survey level assessment is only done when there is a reason to- it is not done on a specific schedule.