Examiner Training & Scoring Demonstration

1. Tammie McParland PhD
Nipissing University
Plenary IV: Examiner Training &
Scoring Demonstration
Debra Sibbald PhD
Touchstone Institute

2. The Chain Model of Educational Assessment Validation and Planning
Adapted by Sibbald from Crooks et al, 1996
Administration:
planning & development
Examiner Scoring:
THE WEAKEST
LINK
Score Aggregation
Score Generalization
Score Extrapolation
Evaluation
Decision
Educational Impact
Consequences
Participants
Perform
Tasks
Rate
Tasks
Combine
Scores
Assess
Domains
Domain
Scores
Judgement
Action
TASKS PROCESSING

3. Key question
Can training be optimized
to reduce variance in
examiner rating
processes?

4. Optimizing Training: 4 Key Elements
Key Elements of Variance
Individual Assessments:
Integration Factors
Cognitive
Processes
Internal
Factors
Objective
Tools
Operational
Logistics

5. Internal
Information
Objective
Tools
Cognitive
Processes
SCORE
EXPECTATIONS
the right things
OBSERVATIONS
the right things
done right???
Integration
Three Step Process

6. Objective
Tools:
Criteria:
• Examiner notes
• GRS Scales and rubrics
• Checklists
• Props
• Prompts
• Scoring sheets
• Tablet scoring

7. Internal
Sources of
Information
Candidate training levels vary within cohort
Rater:
– Experience – new vs experienced
– References & pattern recognition
– Internalization of external criteria:
- Interpreting guidelines & scales
- Comparing benchmarks to candidate
Internal Variance

8. Cognitive
Processes:
Rater thinking
• Weighting – good vs poor
• Looking for ….
• Comparing ……
• Making inferences
• Nuances
– “Everything was well done EXCEPT
what I think is MOST important!!!!”
Integrate Objective Tools with a Subjective Lens

9. Step 2 & 3:
Raters See Things Differently or See Different Things:
Adapted by Sibbald from Sebok, 2015:
Understanding how raters communicate in the context of medical high-stakes, performance-based assessments
Internal Information
Cherished values Subjective – social judgments?
Emotional responses Subjective – social judgments?
Impression formation Subjective – bias?
Cognitive processing
Memory models Analytic vs holistic
Mental workload capacity Task demand/Time management

10. Customized training: 5-9 hours
 3-5 videos
 Examiner calibration
– large/small groups
 Operational Logistics
– tablet scoring training etc.
Criteria &
Tools
Cognitive
Processes
Operational
Logistics
New examiners Feedback from last OSCE

11. Administration:
planning & development
Examiner Scoring:
a STRONGER LINK
Score Aggregation
Score Generalization
Score Extrapolation
Evaluation
Decision
Educational Impact
Consequences
Student
Perform
Tasks
Rate
Tasks
Combine
Scores
Assess
Domains
Domain
Scores
Judgement
Action
TASKS PROCESSING

12. 2018 Perspectives Symposium
January 22, 2018
#TSINPerspectives2018