This is a presentation used to teach the basics of competency development and assessment. It demonstrates the use of the audience response system, Turning Point.
1. What is your biggest challenge in training & competency assessment?
2. Competency Assessment Boot Camp Basics for Labs Caer Vitek Education Coordinator for Graduate Medical Education Department of Laboratory Medicine and Pathology May 5, 2009
Don’t waste time with assessing knowledge of trivial facts. Focus on problems encountered in real life Stems can be long, items should be short. Patient vignettes or lab vignettes. If a question requires an examinee to reach a conclusion, make a prediction, or select a course of action, it is classified as an application of knowledge – versus assessing only remote memory of an isolated fact – this is recall. Imagine recall find answer in a paragraph of a text, versus application which would be different parts of text or pages of a text C The stem must pose a clear question and it should be possible to arrive at the answer with the options covered. To determine if the question is focused, cover up the options and see if the question is clear and if the examinees can pose an answer based on the stem. All distractors should be homogeneous. They should fall into the same category as the correct answer (eg all diagnoses,tests, treatments, prognoses, disposition alternatives). Avoid double options, do W and X, do y b/c z unless all distractors are double items.
Because item writers are more likely to pay attention to the correct answer than the distractors, grammatical errors are more likely to occur in the distractors.
Options A,B,C include all possibilities. Therefore the testwise student will know that A,B, or C must be correct whereas the non-testwise student spends time considering D and E. Often, the item writes add D and E only because they want the option list to have 5 options and therefore not paid attention to the merits of D and E.
In this option A, B and E contain terms that are less absolute than C and D. The testwise students will eliminate options C and D as possibilities because they are less likely to be true than something stated less absolutely. This flaw would not arise if the stem was focused and the options were short; it arises only when the verbs are included in the options rather than the lead-in.
C is longer than the other options, it is also the only double option. Again, we tend to spend more time on the correct answer than the distractors, and because we are teachers, we write long correct answers that include additional instructional material, parentheical information, caveats etc.
In this item the word “unreal” is used in the stem and “derealization” is the correct answer. Sometimes a word is repeated only in a metophorical sense, for example a stem mentioning bone pain, with the correct answer beginning with the prefix osteo.
So, this item illustrates a common flaw. The options are very long and complicated. Trying to decide among these options requires a significant amount of reading because the number of elements in each option. This shifts what is measured in the question from content knowledge to reading speed.
When numeric options are used, the options should be listed in numeric order and the options should be listed in a single format (as single items OR ranges). Confusion ensues when the formats are mixed. Another problem is that the options are listed in an illogical order or an inconsistent format. So for this option, A,B,C are expressed as ranges, whereas options D and E are specific percentages. Additionally, the range for C includes options D and E, which therefore rules out D and E as possible correct answers.
None of the above is problematic in items where judgement is involved and where the options are not absolutely true or false. A student can come up with a better answer than one listed and be right – for example in this question, the student may be thinking Philadelphia or Newark. None of the above can be replaced with options that mean roughly the same thing but is more specific. For exampla, in an item asking an examinee to specify the most appropriate pharmacotherapy, replacing none of the above with no drug should be given at this time eliminates some of the ambiguity.
