1. WRITING TEST
QUESTIONS FOR
ANY TEST ENGINE
Sharon Gander, M.Ed., CPT
Judith Hale, Ph.D., CPT

2. OVERVIEW
 Overview and presenter introductions
 Test engine challenges
 Conditions & scenarios
 Questions and responses
 Quality Distractors
 Quality checks
 Test Engine Debrief
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 6

3. TAKE TIME TO:
1 – OPEN A WORD PROCESSOR
2 – OPEN OR DOWNLOAD A TEST ENGINE
3 – DRAFT 2-3 TEST QUESTIONS ON ANY TOPIC
4 – CHECK THAT YOU HAVE RECEIVED THE HANDOUTS
3 – WORK WITH A PARTNER ON ACTIVITIES
– FIND YOUR PARTNER NOW!
Prepare for this
session by doing the
steps above!

4. OBJECTIVES
1. Write a one-paragraph scenario opening to a question.
2. Write a question with one right answer.
3. Write three plausible answers and one correct answer to the posed
question.
4. Identify strategies for validating the preferred response to posed
question.
5. Check answers for “tells” that give away the right or wrong answer.
6. Explain the value of using Plain Language and Global English in writing
test questions.
7. Leave with three tools to apply in writing test items for learning or
certifications.
8. Recognize test engine characteristics that impact item design, delivery
and validation.
9. Prototype one or more questions in a free test-engine.
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 8
Immediately after this session,
begin building scenario-based questions
for courses, certifications and test-outs
as opportunities arise.

5. TEST ENGINE CHALLENGES
What challenges do you experience in
writing test items for online test-
engine delivery?

6. TEST ENGINE CHALLENGES
 Developer familiarity with test engine while writing
 The need to write clear questions and responses
 It is difficult to get valid content
 Need to overcome the strong tendency to write
recall (knowledge/facts)
 Difficult to write questions that make sense to
the test candidate
 Continuous error checking dulls the senses
 Difficult to use appropriate question type
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 10

7. CONDITIONS AND SCENARIOS
How do you create performance-
based test questions that work
in any test engine?

8. PERFORMANCE-BASE QUESTIONS
The heart of a performance-based question is a condition.
You could use:
 Given a specific situation and resources…
Or
 Describe the situation in 1-2 paragraph story
(a scenario)
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 12

9. Given the fact that test engines and survey
engines handle question sets and answer
evaluation very differently and the fact that you
need to produce quality assessment questions as
quickly as possible.
Which two question types are most standard
across all engines?
a) True/false and multiple-choice.
b) Fill-in-the-blank and ordering.
c) Multi-select and ranking.
d) Essay and matching.
13THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)

10. SOAR – A SCENARIO TECHNIQUE
Situation – essential facts of the setting
Obstacles – the hurdles that will prevent action
or require unique actions; problems and
challenges
Action – the key decisions and/or action steps
are required
Result – the desired outcome
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 14

11. ACTIVITY #1: PREPARE QUESTION
1. Think of a situation of your own (or use the one below).
2. Describe the conditions using your SOAR template
3. Work with partner – 3 minutes
Write for the action. What action will overcome the obstacle to gain
the desired result
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 15
Can’t think of a situation? Try this:
You are tasked with assessing SME’s skills for writing
content. Your organization wants SMEs to apply adult
learning theory to content that will be used in e-learning
solutions. Describe a scenario that will assess this skill.

12. QUESTIONS AND RESPONSES
Why use multiple-choice questions with
only one correct response?

13. WHY ONE CORRECT ANSWER?
Clarity
 Decreases time readers spend figuring out what
the question is
 Decreases the time spent guessing at what the
question writer intended
Defensibility
 Can be supported by unambiguous documented
references
 Provides clear evidence of knowledge when
answering knowledge-type questions
 Provides clear evidence of performance when
answering performance-based questions
Technology
 Scoring and tracking answers
 Works with every test engine
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 17

14. WHAT IS A CLEAR QUESTION?
One that gives the reader clear signals about
“what to answer” by:
 Starting with a question word (5Ws/H)
 Who, what, when, where, why, or how
 Ending with a question mark
 Avoiding the hanging question that results from
using a colon that leaves the reader trying to read
your mind
 Using only one verb
 Using action verbs
 Using either the present or past tense
 Avoiding the passive “is” “was”
 Using either an actor, “you” or implied “you”
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 18

15. ACTIVITY #2: WRITE A QUESTION
1. Using your SOAR scenario and the item writing template
2. Write a question with ONE right answer
More about distractors and incorrect responses coming soon.
3. Work with partner – 3 minutes
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 19
Write for the action.
Find a way to ask, what action will overcome an
obstacle to gain the desired result?

16. WHAT IS A QUALITY DISTRACTOR
Should one response be obviously wrong?
Should I use the “ none of the above” or “all
of the above responses?
Should I use answers like “a and c only”?

17. QUALITY DISTRACTORS
Are plausible – all of them are plausible
 Real and real-life
Force discrimination
 Requires test-taker to evaluate options
Separate the less experienced from the highly
experienced
 Requires test-taker to have experience (and
education) to choose the most effective response
Technically accurate
 May distract with variance in technical details
 Forces accuracy in technical aspects
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 21

18. MULTIPLE CHOICE RESPONSES
One Correct Response
 With documented reference of correctness
 Not just an experts “feeling” that it’s right
Three Plausible Responses
 One mistake that most novices make
(a common error)
 One mistake that soon-to-be experts tend to make
 One mistake that even many experts make
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 22

19. ANSWER VALIDATION STRATEGIES
Need to validate your preferred response?
 Document references that validate the correct
response (#1)
 Ask an acknowledged expert to write a white
paper on the subject
 Poll experts and write a white paper about the
polled results
 Give question to experts and novices;
determine which answer the experts choose
 Other? Anything that documents a preferred
response other than “I believe this to be true.”
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 23

20. In a multiple choice question, the optimal answer set
is four answers with one right answer.
What should the other three answers be?
a) Two wrong answers plus one “all correct” answer.
b) Two plausible answers plus one obviously wrong answer.
c) One plausible answer, one wrong answer, plus an “all
correct” answer.
d) Three plausible answers typical of different skill levels.
24THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)

21. ACTIVITY #3: PLAUSIBLE
RESPONSES
1. Write 3 plausible and 1 correct response to your question
2. Enter your question in your test engine.
3. View your question in the test engine as though you were a test-
taker.
4. Work with partner – 5 minutes
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 25
Checkpoint
Is there a difference between what you planned
to write and what the test engine allowed or
how the question displayed? Why?

22. QUALITY ASSURANCE
How do you check your work before publishing it?

23. BASIC QUALITY ASSURANCE
• Grammar, spelling, punctuation
• Looking for “tells” (hints that give away the
correct response)
• Length of responses
• Change in verb tense
• Change in actor
• Change in technical terms
• Change in punctuation
• Ensure that the question stem
• Uses only one action verb
• Starts with question work (5Ws/H)
• Ends in question mark
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 27

24. Key words, changes in grammar style and verb
tense can provide “tells” (hints to the right answer)
among the plausible distractor answers.
Which of the following techniques would also
prevent a “tell” in your answer set?
a) Use mainly very short one or two word answers.
b) Use very precise technical terminology or phrases.
c) Use answers that are the relatively the same length.
d) Use only present tense verbs in plausible answers.
28THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)

25. In a multiple choice question, the optimal answer set
is four answers with one right answer and:
a) 3 wrong answers
b) 2 plausible answers +1 obviously wrong answr
c) Use one plausible answer, onw very wrong answer, and one
“all correct” answer.
d) Three plausible answers typical of different skill levels.
29THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)
What does the question really test?
WHAT QA ISSUES?

26. ADVANCED QUALITY ASSURANCE
Plain Language
• Active voice (avoid “is” and “was”) – go for the action
• Use strong clear noun – a person, place or thing (avoid “it” “they”)
• Use simple sentence structures – end your thought with a period and
limit the number of comma and colons.
• Use “you” or the implied “you” command when the reader is required
to take action
• Spell out acronyms
• Avoid jargon
• Organize your points
Global English
• Avoid the use of country specific terms (e.g., USDA, Rte 66, M1, Kindergarten)
• Describe common situations rather than using local terms (e.g., tsunamis,
School District, US green building code)
Reading Level (approx. 9th grade)
• Watch the long names with many capitalized nouns in a row
• Put periods at end of bullet-points
• Avoid jargon
• Limit use of technical terms
• Use short paragraphs
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 30

27. Your current project must provide an assessment with
many questions that will be used in many countries by
professionals with many language preferences.
What is the strongest reason for using Plain Language
and Global English in writing test questions?
a) Your question items will all be in one verb tense for
reading ease.
b) Your questions will be clear to anyone who reads
English.
c) Your answers are researched, justifiable and
measurable.
d) Your work can be used internationally without
translation.
31THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)

28. How did the test engine affect your question?
What happened to your quality assurance when
you entered your question into the test engine?
What test engine issues did you experience?
Did you put your questions in a work processor
first? If so, how did that impact your results?
TEST ENGINE IMPACT

29. ERROR PREVENTION
Use a QA Checklist like the one provided
 Check the writing, spelling and grammar
Review questions (offline) with an expert audience who
has not seen them
• Do they validate the question, the correct answer and the validation
resource (text, website, document)
Load your items into the online format
 Does the item display correctly?
 Does the answer correctly evaluate as right or wrong?
 Does the answer/score transfer to the LMS correctly?
Run an online pilot with both expert and non-experts
candidates
 Feedback on questions and test process
 Cut scores
Modify the test as needed
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 33

30. YOU HAVE HAD THE CHANCE TO:
1. Write a one-paragraph scenario opening to a question.
2. Write a question with one right answer.
3. Write three plausible answers and one correct answer to the posed
question.
4. Identify strategies for validating the preferred response to posed
question.
5. Check answers for “tells” that give away the right or wrong answer.
6. Explain the value of using Plain Language and Global English in
writing test questions.
7. Leave with three tools to apply in writing test items for learning or
certifications.
8. Recognize test engine characteristics that impact item design,
delivery and validation.
9. Prototype one or more questions in a free test-engine.
THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 34

31. In the next 7 days, write at least 5
scenario-based, multiple-choice questions
with 1 correct and 3 plausible responses.
QA them with the checklist and enter them
into a test engine.
THE CHALLENGE!

32. WRITING TEST
QUESTIONS FOR
ANY TEST ENGINE
Sharon Gander, M.Ed., CPT
Judith Hale, Ph.D., CPT
Contact us:
Sharon Gander at performancepi.gander@gmail.com
Judy Hale at judy@tifpi.org

33. THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 37
SESSION EVALUATION
Please take the time to provide us with your session feedback using
any of the following methods.
QR Code
1. Download a QR reader to your Smartphone
or tablet
2. Scan the QR code from this slide, your
session evaluation sign, or your program
3. Launch and complete the evaluation
URL Address
1. Locate the URL for this session from this slide, you session
evaluation sign, or your program.
2. Enter the URL Code for this session into the browser of your
laptop, Smartphone or tablet
3. Launch and complete the evaluation
URL: https://www.questionmark.com/cets07