Presented by David Caldwell, Director of Professional Affairs, and Adam Pate, Assistant Professor, both from University of Louisiana Monroe School of Pharmacy Although multiple-choice examinations are one of the most commonly used assessment methods in higher education, often faculty members receive little to no formal training in the discipline. This results in wide variation of item quality, which has been shown to affect student performance. This presentation will describe concepts in multiple-choice item writing including common errors, the statistical effects associated with them, and simple solutions to improve flawed items. In this interactive session presenters will describe personal struggles and successes in item writing and review published literature describing the effects on student performance.

1. EFFECTS OF QUESTION FORMATS ON
STUDENT AND ITEM PERFORMANCE
Adam Pate, PharmD
David Caldwell, PharmD

3. The fundamental problem in shock is:
a. Decreased diastolic blood pressure
b. Decreased systolic blood pressure
c. Decreased tissue perfusion
d. Increased serum lactic acid
Example

4. If you find a patient in shock, your first action
should be:
a. Begin CPR
b. Call 911
c. Circulation, airway, breathing assessment
d. Find an AED
Example

9. Haladyna, Downing, &
Rodriguez
• Developed from an analysis of 27
textbooks and 27 research studies
• Purpose was to validate each
guideline based on agreementin
studied sources
Haladyna TM, Downing SM, Rodriguez SM. Applied
Measurement in Education. 2002; 15(3):309-334.

10. Which of the following is the capital of Texas?
a. Austin*
b. Dallas
c. El Paso
d. Houston
Anatomy of a MC question

11. Guidelines: a closer look
Guideline For (%) Uncited (%) Against (%)
Use positives, no
negatives
63 19 18
Write as many plausible
distractors as you can
70 26 4
Use None of the Above
carefully
44 7 48
Avoid All of the Above 70 7 22

12. All of the following drugs are metabolized by
the P450 enzyme system except:
a. Atazanavir
b. Darunavir
c. Raltegravir
d. Tipranavir
Use positives, no negatives

13. Stem negation =
most appropriatewhen
used to measure
relevant objectives

14. Haladyna and Downing (1993)
• Two-thirds of all items had only 1-2
effectively performing distractors
• Only 1-8% of items had three
effective distractors
• No measured effect on difficulty, but
more effective distractors = more
discrimination
Write as many plausible distractors as possible
Haladyna TM, Downing SM. Education and Psychological
Measurement. 1993; 53:999-1010.

15. Effect of numerous distractorsunclear

16. Which of the following is the capital of Texas?
A. Dallas
B. El Paso
C. Houston
D. Lubbock
E. None of the above
Use “none of the above” carefully

17. 3 of 4 studies failed to demonstrate differences
in discrimination with “None of the above”

18. Which of the following is true regarding transdermal NRT?
A. These products may be more appropriate in patients who anticipate
adherence problems
B. The 16-hr patch reduces the risk of insomnia and vivid dreaming
compared to the 24-hr
C. Currently there is insufficient data to directly compare transdermal
NRT to oral NRT
D. Transdermal NRT patches should never be cut
E. All of the above
Avoid “all of the above”

19. 70% of referenced sources recommend
avoiding “all of the above”

22. 50
60
70
80
90
100
Difficulty (p)
Standard
Flawed
A retrospective study
0
0.2
0.4
0.6
0.8
1
Discrimination
(rpb)
p=0.01 p=0.64
!
48
%
52
%
Guideline # Times broken % Total
Minimize reading 24 16.9
All of the above 24 16.9
Complex item format 19 13.38
Central idea in stem 17 11.97
Edit and proof 10 7.04
Choices homogenous 8 5.63
Plausible distractors 8 5.63
Trivial Content 6 4.23
Length of choices
equal
6 4.23
None of the above 6 4.23
Other 13 9.86
Ratio Standard:Flawed
Pate AN, Caldwell DJ. CPTL. 2014; 6(1):130-134.

23. A prospective study
5 authors
Student randomization
standard, n=55
flawed, n=54
Test administration
Analysis: one-sided
independent t-tests
between the two scales
Mean
discrimination, rpb
Mean item
difficulty, p
40 exam
items
25
35
45
55
65
75
Difficulty (p)
Standard Flawed
p=0.04
0
0.1
0.2
0.3
0.4
0.5
Discrimination
(rpb)
p=0.44
Caldwell DJ, Pate AN. J Pharm Ed. 2013; 77(4):Article 71.
Investigator
review
Test form 1
20 standard items
Test form 2
20 flawed items

24. Any questions?