2. Test Development Process
ďŹ 1. Statement of Goals
ďŹ 2. Content Outline
ďŹ 3. Table of Specifications
ďŹ 4. Item Selection
ďŹ 5. Item Construction
ďŹ 6. Composition of Instructions
ďŹ 7. Development of Answer Sheets
ďŹ 8. Construction of Answer Keys
ďŹ 9. Test Administration
ďŹ 10. Test Revision
4. Two Important Characteristics of Tests
ďŹReliability
â consistency
⢠free from extraneous sources of error
ďŹValidity
â how well a test measures what it is
supposed to measure
5. Formative vs. Summative Tests
ďŹFormative
â monitor progress toward goals within a
course of study
ďŹSummative
â assess overall achievement of course
goals
6. A. Table of Specifications
ďŹBlue print for test
ďŹPurpose
â ensure proper emphasis given to all
elements of a course of study
â content validity
ďŹGuide for writing items
7. Table of Specifications (2)
ďŹStart with Instructional Objectives
â Biology 30: Circulation Unit Test
1. Identifies parts of circulation system.
2. Explains function of parts in relation
to whole system.
3. Distinguishes between circulation
system and other bodily systems.
9. Need to Consider and Decide:
ďŹLength of Test
ďŹWeight to be given to each objective
ďŹWeight to be given to each level of
taxonomy
ďŹEstimate number of items in each
cell
12. C. Item Construction
ďŹ Selected Response Test Items
â Item construction skills for valid and
reliable measures of student
achievement.
ďŹ Guidelines which apply to all types
ďŹ Specific Suggestions for writing each type
ďŹ Advantages and Disadvantages of Each
13. Guidelines for Writing Objective Items
1. Construct at appropriate level of
difficulty for examinees
2. Include Items at appropriate level of
difficulty for purpose of test.
3. Test significant elements of a
course.
4. Write independent items.
14. 5. Construct questions free from
extraneous reasons for problems.
6. Communicate the question in clear,
concise language.
7. In the correct alternatives,
paraphrase statements from the text.
8. Exclude clues to correct answer.
II. Guidelines for Writing Objective Items
15. III. Guidelines for Writing Objective Items
9. Provide one correct answer.
10. Edit the Items.
16. Alternate Response Items
ďŹ Involves the selection of one of two
alternatives
â true / false
â yes / no
â right / wrong
â fact / opinion
ďŹ Mainly for Knowledge & Comprehension
â Can be written at higher levels
17. True / False
ďŹ Word statements clearly. Vague or ambiguous wording
will confuse students.
ďŹ Avoid overgeneralizing.
â Poor: Heavy smoking causes lung cancer. T F
â Better: Heavy smoking often causes lung cancer. T F
ďŹ Avoid Trick questions.
â i.e., General Wolseley led Canadian troops to Manitoba in
1870.
â 3 tricks: Colonel, British, Not yet a province
ďŹ Do not use trivial statements to âpad outâ the number of
questions and marks to arrive at a predetermined level.
18. True / False
ďŹ Statements should be entirely true, or
entirely false:
â Unacceptable:
⢠In King Lear, Regan ordered Gloucesterâs eyes to be
plucked out and Gloucester died when he jumped off
the cliff of Dover.
â Acceptable:
⢠In King Lear, Regan ordered Gloucesterâs eyes to be
plucked out. (T)
⢠In King Lear, Gloucester died when he jumped off the
cliff at Dover. (F)
19. True / False
ďŹ Avoid using universal descriptors such as
âneverâ, ânoneâ, âalwaysâ, and âallâ.
â Testwise students will recognize that there are few
absolutes.
ďŹ Avoid negative words, as they are often
overlooked by students.
â Poor: It was not unheard of for Henry VIII to close
monasteries in England. T F
â Better: Henry VIII closed some monasteries in England.
(T)
20. True / False
ďŹ Do not include two ideas in one statement unless you are
evaluating studentâs understanding of cause and effect
relationships.
â Poor: Porpoises are able to communicate because
they are mammals. T F
â Better: Porpoises are mammals. T F
Porpoises are able to communicate. T F
ďŹ Provide a âTâ and âFâ beside each statement and ask
students to circle correct answer.
â Avoids problem of students writing illegible letters.
21. True / False & Variations
ďŹ Include more false than true statments in any given test
and vary the number of false statements from test to test.
â tendency to mark more statements true than false.
â discrimination between those who know the content
and those who do not is greater for false expressions.
ďŹ Avoid using negative statements.
â Under the demands of the testing situation, students
may fail to see the negative qualifier.
22. Matching Items
ďŹ Consist of
â a column of premises
â a column of responses
â directions for matching the two.
ďŹ Similar to multiple choice, but easier and more
efficient to construct
ďŹ Can be written to assess Knowledge,
Comprehension, Application, Analysis level
behaviors
23. Guidelines for Writing Matching Items
ďŹ Provide clear instructions on how to indicate the
correct answers.
ďŹ Indicate whether the same response can be used
more than once.
ďŹ Maintain grammatical consistency within and between
columns.
â within a column: either sentence or point form
â between columns: one or the other
ďŹ Ensure that any matching question appears entirely
on one page.
24. 2. Guidelines for Writing Matching Items
ďŹ Provide an unequal number of premises and
responses
â reduces guessing and elimination
â increases measure of comprehension
ďŹ Avoid designing questions which require
students to draw lines between premise and
response.
â confusing for student and marker
â provide space for letter or number answers
25. 3. Guidelines for Writing Matching Items
ďŹ Make sure lists are homogeneous.
â i.e., do not include items testing names, dates, and
events.
â Instead, make every response plausible
ďŹ Make the wording of the premises longer than
the wording of the responses.
ďŹ Identify the items in one list with numbers and
those in the second list with letters.
26. Example:
Directions: 1. On the line to the right of each phrase in
Column I, write the letter for the word in
Column II that best matches the phrase.
2. Each word in Column II may be used once,
more than once, or not at all.
Column I Column II
1. Name of the answer in addition
problems.
2. Name of the answer in
subtraction problems.
3. Name of the answer in
multiplication problems.
4. Name of the answer in division
problems.
A. Difference
B. Dividend
C. Multiplicand
D. Product
E. Quotient
F. Subtrahend
G. Sum
27. Short Answer Test Items
ďŹ Typically, the student is asked to reply with a
word, phrase, name, or sentence, rather than a
more extended response.
â Direct Questions / Short Answer
⢠Who is the current Prime Minister of Canada?
â Incomplete Sentences / Fill In the Blanks
⢠The current Prime Minister of Canada is _____?
ďŹ Items are fairly easy to construct and mark
ďŹ Assess mainly knowledge, comprehension, and
some application.
28. Guidelines for Writing Short Answer Items
ďŹ Questions must be carefully worded so that all
students understand the specific nature of the
question asked and the answer required.
â Poor: Wellington defeated Napoleon in _____ ?
â Better: In what battle fought in 1815 did Wellington
defeat Napoleon?
OR
In what year did Wellington defeat Napoleon
at Waterloo?
29. II. Guidelines for Writing Short Answer Items
ďŹ Word completion or fillin questions so that missing
information is at, or near the end of, the sentence. Makes
reading and responding easier.
â Poor: In the year ______ , Canada turned 100 years
old.
â Better: Canada turned 100 years old in the year _____
.
ďŹ Instructions and teacherâs expectations about filling in
blanks should be made clear. Indicate whether each
blank of equal length represents one word or several
words, whether long blanks require sentences or
phrases, and whether synonymous terms are accepted.
30. III. Guidelines for Writing Short Answer Items
ďŹ When an answer is to be expressed in numerical units, the
unit should be stated.
â Poor: If a room measures 7 metres by 4 metres, the
perimeter is _____ .
â Better: If a room measures 7 metres by 4 metres, the
perimeter is _____ metres (or m).
ďŹ Do not use too many blanks in completion items. The
emphasis should be on knowledge and comprehension,
not mind reading!
â Consider:
In the year _____ , Prime Minister _________ signed the
__________ , which led to a ___________ which was
__________ .
32. 1. The capital city of Canada is
a. Vancouver
b. Montreal
c. Toronto
*d. Ottawa
Terminology: Multiple Choice
1. Stem: presents the problem
2. Keyed Response: correct or best answer
3. Distracters: appear to be reasonable answers to the
examinee who does not know the content
4. Options: include the distracters and the keyed
response.
4
1
2
3
33. Ambiguity
ďŹ Extrinsic
â Desirable quality in
multiple choice
items
â Outside the item
â Allows
discrimination
between those who
know material and
those who do not.
ďŹ Intrinsic
â Undesirable
â Should be avoided
â Inside the item
⢠poor wording
⢠more than one answer
â Even those who know
content have difficulty
choosing correct answer
34. Types of Multiple Choice Items
ďŹ Correct Answer*
â Only one correct response
ďŹ Best Answer
â requires examinee to select alternative
closest to being correct
â fine distinctions
ďŹ Multiple Answer
â More than one correct or best answer
35. Interpretive Exercise
ďŹ Usually begins with verbal, tabular or graphic
information which is the basis for 1 or more
multiple choice questions.
â map, passage from a story, a poem, a cartoon
ďŹ Can challenge students at various levels of
understanding
â application, analysis, synthesis, evaluation
ďŹ Exercise contains all information needed to
answer questions
ďŹ Readily adaptive to the more important
outcomes of disciplines.
36. Interpretive Exercises (conât)
ďŹ Examples
ďŹ If student answers incorrectly it is because
they have not mastered the thinking or
reasoning required by the question, NOT
because they failed to memorize background
information.
ďŹ Math questions: give students the formulas,
test ability to apply concepts, rather than ability
to memorize formulas.
37. Guidelines for Writing:
Multiple Choice Items
ďŹ State stem in the form of a question.
â Weak
â Canada is
⢠a) a country
⢠b) where you live
⢠c) between the Pacific and Atlantic oceans
⢠d) all of the above
â Better
â Between which two oceans is Canada located?
⢠a) Atlantic and Indian
⢠b) Atlantic and Pacific
⢠c) Pacific and Indian
38. Guidelines for Writing: Stems
ďŹ Place most of the subject matter in the Stem
â ensures full statement of problem
ďŹ Eliminate extraneous material from the Stem
â goal is to measure student achievement, not to present
new material
â maximize use of time for demonstrating understanding,
not reading ability
ďŹ Avoid Negatively phrased Stems
â students may miss the qualifier
â use only when learning outcome requires this type of
differentiation
39. I. Guidelines for Writing: MC
ďŹ Ensure similarity among alternatives
with regard to:
â grammatical structure
â length
â mode of expression
ďŹ Grammatical errors provide
unintentional clues to the answer
ďŹ When in doubt, students will select the
longest alternative as the correct answer
40. Example: Length of Alternatives
Neurotics are more likely than psychotics to
a. be dangerous to society
b. have delusional symptoms
c. be dangerous to themselves
*d. have insight into their own inappropriate
behavior but nevertheless feel rather
helpless in terms of dealing with their
difficulties
41. II. Guidelines for Writing: MC
ďŹ Make one of the alternatives the most clearly
correct or best answer
â exception: multiple answer form
â reduces intrinsic ambiguity
â reduces frustration during test
ďŹ Make distracters plausible
â desire to attract students who really do NOT know the
answer to the question
â create distracters from elements of the correct response
â improves reliability of item
42. Example: Undemanding Distracters
Mickey Mouseâs two nephews are
a) Huey, Dewey and Louie
b) Clarabelle Cow
*c) Morty and Ferdy
d) Abbott and Costello
The Role of Humour?
43. III. Guidelines for Writing: MC
ďŹ Avoid parallel language between the Stem and
the Correct Response
â gives clues to keyed response
â emphasizes testwiseness, not knowledge
ďŹ Randomly distribute answers across the
alternative positions
â inexperienced test writers emphasize âbâ and âcâ
alternatives (hide the answer!!)
â do NOT use an interpretable order of keyed responses
44. IV. Guidelines for Writing: MC
ďŹ Use qualifiers such as âall of the
aboveâ and ânone of the aboveâ
sparingly
â testwise students will use process of
elimination to select answer
â do NOT use to âpad outâ the distracters
because you cannot think of another
one.
45. Advantages of Multiple Choice Items
ďŹ allow more adequate sampling of content.
ďŹ tend to more effectively structure the problem
to be addressed
ďŹ items can be more efficiently and reliably
scored than supply items
ďŹ different response alternatives can provide
diagnostic feedback (item analysis)
ďŹ items can be constructed to address various
levels of cognitive complexity
46. Disadvantages of Mult.Choice Items
ďŹ difficult & time consuming to construct good
items
â leads to emphasis on other selected response item types
ďŹ can lead the instructor to favour simple recall
of facts
ďŹ high degree of dependence on studentâs
reading and instructorâs writing ability
â can be difficult to achieve clarity of expression
ďŹ measuring synthesis and evaluation can be
difficult
ďŹ inappropriate for measuring outcomes that
require skilled performance