Presented at AMEE 2009 - Item-Writing & Standard Setting

1. Introduction of Standard Setting in undergraduate medical education: re-emphasizing the importance of item-writing. Jane Holland, Alice McGarvey, Richard Arnett, David Croke Royal College of Surgeons in Ireland

2. Standard setting – preparation for pilot Module selection Nutrition & Energy Junior Cycle Semester 1 n = 281 19 - anatomy, biochemistry pharmacology, physiology Lecturers / Judges Examination Multiple choice, n = 30 Method Modified Angoff (1971) Jane Holland, RCSI

3. Standard setting – training of judges 59.2 60.7 Workshop Dr Katherine Boursicot November 2007 Principles & specific methods Exercise 10 MCQs (JC1 NE 2007) Jane Holland, RCSI

4. JC1 NE 2008: Modified Angoff “Classic” Angoff MCQ cut score 50 58 MCQ Class mean 63.6 54.6 Overall class mean 61.1 59 Fails 12 15.4 14 judges 30 MCQs 3 ½ hours!! *All tabular values are percentages Jane Holland, RCSI

5. JC1 NE 2008: Hofstee 57 % Jane Holland, RCSI

6. RCSI Curriculum Forum, 29th March 2008 Jane Holland, RCSI

7. Item writing – training of writers Workshop Dave Swanson / Kathy Holtzman April 2008 Guidelines NBME / MRCS Jane Holland, RCSI

8. Item writing – pilot within NE module 13 unusable 64 suitable! 7 -ve 4 T/F 2 misc Call for questions 60 MCQs required 71 MCQs requested Guidelines / SBA stems 77 MCQs received 83% Jane Holland, RCSI

9. JC1 NE 2009: Modified Angoff 12 judges 30 MCQs Cut-score = 48 Item difficulties DIF IDEAL Consortium Item Analysis package Item probabilities Prob Modified Angoff (1971) Jane Holland, RCSI

10. JC1 NE 2008: DIF vs Prob Prob KR-20 = 0.735 DIF rs = 0.277 (p = 0.13) Jane Holland, RCSI

11. JC1 NE 2009: DIF vs Prob Prob KR-20 = 0.769 DIF rs = 0.599 (p = 0.001) Jane Holland, RCSI

12. Summary of results & take home message 2008 2009 MCQ cut score 58 48 MCQ Class mean 63.6 63.0 Item difficulties vs. item probabilities (rs) 0.277 (p=0.13) 0.599 (p=0.001) A well-structured item, while designed to more accurately assess the candidate’s knowledge of the subject material, also enables more accurate rating of items during the standard setting process and is an important consideration when introducing these procedures de novo. Jane Holland, RCSI