The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams

1. The Art of Writing Good
Multiple-Choice Questions for
High-Stakes Exams
Claire Touchie, MD, FRCPC
University of Ottawa
Medical Council of Canada

2. Workshop Objectives
• Describe what can be tested with multiple-
choice items
• Define the anatomy of a multiple-choice item
• Define and identify technical flaws
• Create multiple-choice items for own stated
purpose
• Define and criticize poor performing items
2

3. How to write multiple-choice items
1. The What
2. The How
3

4. The What
• What can I test with MCQs?
• Knowledge
• Clinical decision making
• Clearly define the purpose of your exam
• Define what it is you want to test
• For the overall test
• For your specific question
4

5. The What
Prior to writing your question, ask the
following questions
• What concept do I want to test?
• Where does the learner go wrong?
• Focus on areas of “challenge” for the learner
5

6. Example of the What
Purpose: To assess the clinical clerk’s
knowledge and decision-making capability
at the end of an Internal Medicine six-week
rotation
• Concept/Objective: Management of
CHF
• Challenge to the learner: ???
6

7. The How – Anatomy of a MC item
• Stem
• Clinical vignette that describes the setting, the patient’s
age and complaint along with pertinent historical facts,
physical exam details, and/or laboratory findings
• Lead-in question
• The task
• Answer and alternatives
• The most correct answer and the plausible distractors
7

8. The How – Anatomy of a MC item
Stem
• A 58-year old man presents to the ED with
sudden onset of left-sided chest pain
associated with shortness of breath,
palpitations and dizziness. His past history is
relevant for a recent diagnosis of lung
carcinoma. His examination is only
remarkable for a heart rate of 112/minute.
8

9. The How-Anatomy of a MC item
Lead-in question
• Which one of the following diagnostic test
would be most useful to confirm the
diagnosis?
9

10. The How – Anatomy of a MC item
Correct/Best answer and distractors
1. Chest radiograph
2. CT of the chest *
3. Sputum culture
4. Electrocardiogram
5. Echocardiogram
10

11. What did you notice about this question?
Could you answer it without seeing the
alternatives?
Could you answer without being given the
diagnosis?
11

12. The Stem
• Short description of a clinical scenario
• Common or clinically important
• Clear and contains relevant information to the
clinical problem – avoid window-dressing
• Word the stem positively
• Avoid EXCEPT questions
• Use negative words with caution
• Eg: contraindication, what to avoid
12

13. The Stem
• Provide sufficient information to answer
the item
• DO NOT create tricky items by omitting
essential information
• DO NOT add extraneous information
• Stem should be a clinical vignette
13

14. Lead-in Question
• Ensure the directions are very clear with
a clear task
• Can the stem be administered in a short answer
(constructed-response) format?
• “Cover answer test”

15. Lead-in Question
• Different clinical tasks can be tested
• Can be done with the same stem (cloning of
question)
• History
• Diagnosis
• Investigations
• Management/Treatment/Drug therapy
• Counseling

16. Lead-in Question
Try asking questions that lead to clinical decision
making?
Which one of the following
– … is the most likely diagnosis?
– … investigations would you now order?
– …is the next step in the work-up and management of this
patient?
– … is the most important step in the initial management of
this patient?

17. Distractors
• Only one right choice with four distractors
• Use plausible distractor choices
• Keep distractors independent, they should not be overlapping
• E.g.: 1. 11-20; 2. 15-30
• Keep distractors homogeneous in content and grammatical
structure
• Keep the length about equal
• Avoid specific determiners such as All, Never, Always,
Completely and Absolutely
• Do not use All of the Above or None of the Above

18. Item testing clinical decision making
A 62 year-old woman with a history of confusion and constipation comes to
the office for a follow-up visit. Laboratory investigations reveal a serum
calcium of 2.9mmol/L, a creatinine of 146 µmol/L, and a hemoglobin of 108
g/L.
Which one of the following is the most likely diagnosis?
1. Hyperparathyroidism
2. Chronic renal failure
3. Multiple myeloma*
4. Vitamin D intoxication
5. Renal cell carcinoma

19. Item testing clinical decision making
A 62 year-old woman with a history of confusion and constipation
comes to the office for a follow-up visit. Laboratory investigations
reveal a serum calcium of 2.9mmol/L, a creatinine of 146 µmol/L, and
a hemoglobin of 108 g/L.
Which one of the following would help confirm the diagnosis?
1. Parathyroid hormone
2. Serum protein electrophoresis*
3. 25-OH vitamin D
4. Serum creatinine
5. Abdominal ultrasound

20. What is wrong with this item?
A previously healthy person suddenly presents with
pleuritic pain in the left chest and shortness of breath.
Which one of the following is the most likely
diagnosis?
1. Mycoplasma pneumonia
2. Spontaneous pneumothorax
3. Pulmonary embolism
4. Acute pericarditis
5. Epidemic pleurodynia

21. What is wrong with this item?
Which one of the following is true about pseudogout?
1. It occurs frequently in women.
2. Seldom associated with acute pain in a joint
3. May be associated with a finding of
chondrocalcinosis.
4. It is hereditary in all cases
5. It responds well to treatment with allopurinol
21

22. What is wrong with this item?
Aortic insufficiency may be caused by all of the
following, EXCEPT:
1. syphilis
2. Marfan’s syndrome
3. aortic dissection
4. bacterial endocarditis
5. myocardial infarction*

23. Exercise #1 – 30 minutes
Writing MCQ items
• Pair up with a partner
• Using the MCQ item development worksheet,
develop MCQ items that will be useful to you
23

24. Exercise #2 – Pre-test
A Test of General Rock and Roll
Knowledge
24

25. Technical Flaws
Violations of test item writing principles
• Flawed items are usually more difficult
• Fail more students
Downing, 2002
25

26. Technical Flaws
• Unfocused items
• Negative stem or lead-in question
• Heterogeneous options
• Logical or grammatical cues
• Long correct answer
• Word repeats
• Convergence strategy
26

27. Unfocused item
Which one of the following is true about pseudogout?
1. It occurs frequently in women.
2. Seldom associated with acute pain in a joint
3. May be associated with a finding of
chondrocalcinosis.
4. It is hereditary in all cases
5. It responds well to treatment with allopurinol
27

28. Negative stem or lead-in question
Which of the following does not cause aortic
insufficiency?
1. syphilis
2. Marfan’s syndrome
3. aortic dissection
4. bacterial endocarditis
5. myocardial infarction*
28

29. Heterogeneous option
A 24-year-old female presents to a walk-in clinic with fever,
flank pain, frequency and dysuria. The urinalysis (urine
microscopy) shows 1+proteinuria, 25 white blood cells per high
power field and a few granular casts.
Which one of the following investigations is the next best step?
1. Intravenous pyelography.
2. Intravenous antibiotics.
3. Creatinine clearance.
4. Midstream urine culture.*
5. Oral analgesia.
29

30. Logical cues
A 47-year old man present with an acute episode of
psychosis. Which one of the following treatment would
you consider prescribing?
1. Alprazolam
2. Lorazepam
3. Haloperidol*
4. Diazepam
5. Quetiapine
30

31. Grammatical cues
A 78-year old man undergoes a thoracentesis for a large
pleural effusion. Three hours later, he develops sudden
onset of shortness of breath. What is your most likely
diagnosis?
1. Reaccumulation of fluid
2. Pneumothorax*
3. Lung infection
4. Bleeding
5. Blood clot
31

32. Word repeats
Also known as “clang association”:
A 45-year-old woman presents with sudden loss of consciousness. On
exam, her vitals are normal, she is not pale and she is not diaphoretic.
Which one of the following is more typical of “fainting” as a conversion
symptom than of a syncopal attack due to orthostatic hypotension?
1. Bradycardia.
2. Muscle twitching.
3. Absence of pallor and sweating.*
4. Urinary incontinence.
5. Rapid recovery.
32

33. Convergence strategy
An 86-year-old woman fell at the local nursing home and
sustained an intertrochanteric fracture of her left hip. On clinical
examination, you would expect to find her left leg:
1. Shortened, abducted and internally rotated.
2. Lengthened, abducted and internally rotated.
3. Shortened, adducted and externally rotated.
4. Shortened, abducted and externally rotated.*
5. Lengthened, abducted and externally rotated.
33

34. Post-test
A Test of General Rock and Roll
Knowledge
34

35. Now review the items you have written
using the checklist!
35

36. Exercise #3 – Poor Performing Items
Tests used to assess those who meet the
purpose and those who don’t!
The administered test must therefore live
up to this expectation.
36

37. Poor Performing Items
When an item is either:
• Too hard (very few are getting it right)
• Too easy (very few are getting it wrong)
• Does not distinguish between an overall good
performing candidate or a poor performing
candidate (discrimination)
• Wrongly keyed
• Poor distribution of responses
37

38. Item Difficulty
Percentage or proportion of candidates that
select its correct answer
• Should be between 20-90% (0.20-0.90)
• Best if it is around the cut score but need a range
of difficulty

39. Item Discrimination
Point-biserial index of discrimination
• Correlation between candidates' performance on an item
and performance on the entire test (the latter assumed to be
a measure of a candidate’s overall ability)
• Range from –1 to +1
• Stronger candidates should perform better than weaker
candidates
• Effectively discriminating item is positive and >0.20

40. Item Discrimination
Correct answer should have positive
discrimination
• If not: is there another correct answer?
• Is there a flaw in the question?
Distractors should have negative
discrimination

41. Distribution of Responses
A report of the percentage of candidates
that select each of the options presented
A question’s distractors may need to be
assessed if no candidates are choosing
them.

42. Poor performing items
Question 1 Answer key: option 1
Item difficulty: 0.12
Item discrimination: 0.02
Distractors Response Rate Discrimination
Option 1* 0.12 0.02
Option 2 0.03 -0.12
Option 3 0.05 -0.23
Option 4 0.09 -0.17
Option 5 0.71 0.24
42

43. Poor Performing Item
Question 2 Answer key: option 1
Item difficulty: 0.97
Item discrimination: 0.28
Distractors Response Rate Discrimination
Option 1* 0.97 0.28
Option 2 0.01 -0.20
Option 3 0.01 -0.34
Option 4 0.00 0
Option 5 0.00 -0.04
43

44. Poor Performing Item
Question 3 Answer key: option 2
Item difficulty: 0.59
Item discrimination: 0.16
Distractors Response Rate Discrimination
Option 1 0.00 -0.04
Option 2* 0.59 0.16
Option 3 0.00 -0.04
Option 4 0.39 -0.40
Option 5 0.02 -0.12
44

45. Poor Performing Items
Question 4 Answer key: option 5
Item difficulty: 0.53
Item discrimination: 0.26
Distrators Response Rate Discrimination
Option 1 0.23 -0.26
Option 2 0.04 -0.24
Option 3 0.05 -0.15
Option 4 0.16 -0.33
Option 5* 0.53 0.26
45

46. In Summary
Prior to writing MCQ items:
• Determine the purpose of the test
• Determine WHAT you want to test
Write questions avoiding technical flaws
Ensure questions of are of good quality
reflecting the original purpose.
46

47. References
• Guidelines for the Development of Multiple-Choice
Questions at http://www.mcc.ca/pdf/MCQ_Guidelines_e.pdf
• Haladyna TM, Downing SM, Rodriguez MC. A Review of
Multiple-Choice Item-Writing Guidelines for Classroom
Assessment. Applied Measurement in Education
2002;15:309-334
47

48. Thank you!
Questions? Comments?
ctouchie@mcc.ca
48