SlideShare ist ein Scribd-Unternehmen logo
1 von 49
Downloaden Sie, um offline zu lesen
A journey towards
programmatic assessment
Medical Council of Canada’s 2015
Annual Meeting
September 27, 2015
Cees van der Vleuten
Maastricht University
The Netherlands
www.ceesvandervleuten.com
A journey towards programmatic assessment
Overview
• From practice to research
• From research to theory
• From theory to practice
• Conclusions
The Toolbox
• MCQ, MEQ, OEQ, SIMP, Write-
ins, Key Feature, Progress test,
PMP, SCT, Viva, Long case,
Short case, OSCE, OSPE,
DOCEE, SP-based test, Video
assessment, MSF, Mini-CEX,
DOPS, assessment center, self-
assessment, peer assessment,
incognito SPs, portfolio………….
Knows
Shows how
Knows how
Does
Knows
Fact-oriented assessment:
MCQ, write-ins, oral…..
Knows how
Scenario or case-based assessment:
MCQ, write-ins, oral…..
Shows how
Performance assessment in vitro:
Assessment centers, OSCE…..
Does
Performance assessment in vivo:
In situ performance assessment, 360۫ , Peer assesme
The way we climbed......
Validity
Characteristics of instruments
Reliability
Educational
impact
Acceptability
Cost
Validity
Reliability
Educational
impact
Validity: what are we assessing?
• Curricula have changed from an input
orientation to an output orientation
• We went from haphazard learning to
integrated learning objectives, to end
objectives, and now to (generic) competencies
• We went from teacher oriented programs to
learning oriented, self-directed programs
Competency-frameworks
CanMeds
 Medical expert
 Communicator
 Collaborator
 Manager
 Health advocate
 Scholar
 Professional
ACGME
 Medical knowledge
 Patient care
 Practice-based learning
& improvement
 Interpersonal and
communication skills
 Professionalism
 Systems-based practice
GMC
 Good clinical care
 Relationships with
patients and families
 Working with
colleagues
 Managing the
workplace
 Social responsibility
and accountability
 Professionalism
Knows
Shows how
Knows how
Does
Knows
Knows how
Shows how
Does
Validity: what are we assessing?
Standardized
assessment (fairly
established)
Unstandardized
assessment (emerging)
Messages from validity research
• There is no magic bullet; we need a
mixture of methods to cover the
competency pyramid
• We need BOTH standardized and non-
standardized assessment methods
• For standardized assessment quality
control around test development and
administration is vital
• For unstandardized assessment the users
(the people) are vital.
Method reliability as a function of testing time
Testing
Time in
Hours
1
2
4
8
MCQ1
0.62
0.77
0.87
0.93
Case-
Based
Short
Essay2
0.68
0.81
0.89
0.94
PMP1
0.36
0.53
0.69
0.82
Oral
Exam3
0.50
0.67
0.80
0.89
Long
Case4
0.60
0.75
0.86
0.92
OSCE5
0.54
0.70
0.82
0.90
Practice
Video
Assess-
ment7
0.62
0.77
0.87
0.93
1Norcini et al., 1985
2Stalenhoef-Halling et al., 1990
3Swanson, 1987
4Wass et al., 2001
5Van der Vleuten, 1988
6Norcini et al., 1999
In-
cognito
SPs8
0.61
0.76
0.86
0.93
Mini
CEX6
0.73
0.84
0.92
0.96
7Ram et al., 1999
8Gorter, 2002
Reliability as a function of sample size
(Moonen et al., 2013)
0.65
0.7
0.75
0.8
0.85
0.9
4 5 6 7 8 9 10 11 12
G=0.80 KPB
Mini-CEX
0.65
0.7
0.75
0.8
0.85
0.9
4 5 6 7 8 9 10 11 12
G=0.80 KPB OSATS
Mini-CEX OSATS
Reliability as a function of sample size
(Moonen et al., 2013)
0.65
0.7
0.75
0.8
0.85
0.9
4 5 6 7 8 9 10 11 12
Mini-CEX OSATS MSF
Reliability as a function of sample size
(Moonen et al., 2013)
Effect of aggregation across methods
(Moonen et al., 2013)
Method
Mini-CEX
OSATS
MSF
Sample
needed
when used
as stand-alone
8
9
9
Sample
needed
when used
as a composite
5
6
2
Messages from reliability research
• Acceptable reliability is only achieved
with large samples of test elements
(contexts, cases) and assessors
• No method is inherently better than
any other (that includes the new
ones!)
• Objectivity is NOT equal to reliability
• Many subjective judgments are pretty
reproducible/reliable.
Educational impact: How does
assessment drive learning?
• Relationship is complex (cf. Cilliers, 2011, 2012)
• But impact is often very negative
– Poor learning styles
– Grade culture (grade hunting, competitiveness)
– Grade inflation (e.g. in the workplace)
• A lot of REDUCTIONISM!
– Little feedback (grade is poorest form of feedback one can get)
– Non-alignment with curricular goals
– Non-meaningful aggregation of assessment information
– Few longitudinal elements
– Tick-box exercises (OSCEs, logbooks, work-based assessment).
• All learners construct knowledge from an inner
scaffolding of their individual and social
experiences, emotions, will, aptitudes, beliefs,
values, self-awareness, purpose, and more . . .
if you are learning ….., what you understand is
determined by how you understand things,
who you are, and what you already know.
Peter Senge, Director of the Center for
Organizational Learning at MIT (as cited in van
Ryn et al., 2014)
Messages learning impact research
• No assessment without (meaningful)
feedback
• Narrative feedback has a lot more
impact on complex skills than scores
• Provision of feedback is not enough
(feedback is a dialogue)
• Longitudinal assessment is needed.
Overview
• From practice to research
• From research to theory
• From theory to practice
• Conclusions
Limitations of the single-method
approach
• No single method can do it all
• Each individual method has
(significant) limitations
• Each single method is a
considerable compromise on
reliability, validity, educational
impact
Implications
• Validity: a multitude of methods needed
• Reliability: a lot of (combined) information is
needed
• Learning impact: assessment should provide
(longitudinal) meaningful information for
learning
Programmatic assessment
Programmatic assessment
• A curriculum is a good metaphor;
in a program of assessment:
– Elements are planned, arranged, coordinated
– Is systematically evaluated and reformed
• But how? (the literature provides extremely
little support!)
Programmatic assessment
• Dijkstra et al 2012: 73 generic
guidelines
• To be done:
– Further validation
– A feasible (self-assessment) instrument
• ASPIRE assessment criteria
Building blocks for programmatic
assessment 1
• Every assessment is but one data point (Δ)
• Every data point is optimized for learning
– Information rich (quantitative, qualitative)
– Meaningful
– Variation in format
• Summative versus formative is replaced by a
continuum of stakes (stakes)
• N data points are proportionally related to
the stakes of the decision to be taken.
Continuum of stakes,
number of data point and their function
No
stake
Very high
stake
One
Data point:
• Focused on
information
• Feedback
oriented
• Not decision
oriented
Intermediate
progress decisions:
• More data points
needed
• Focus on diagnosis,
remediation,
prediction
Final decisions on
promotion or selection:
• Many data points needed
• Focused on a (non-
surprising) heavy decision
Assessment information as pixels
Classical approach to aggregation
Method 1 to
assess skill A Σ
Method 2 to
assess skill B Σ
Σ
Σ
Method 3 to
assess skill C
Method 4 to
assess skill C
More meaningful aggregation
Method 1
Σ
Method 2
Σ
Method 3
Σ
Method 4
Σ
Skill
A
Skill B
B
Skill
C
Skill
D
A journey towards programmatic assessment
A journey towards programmatic assessment
A journey towards programmatic assessment
A journey towards programmatic assessment
Overview
• From practice to research
• From research to theory
• From theory to practice
• Conclusions
From theory back to practice
• Existing best practices:
– Veterinary education Utrecht
– Cleveland Learner Clinic, Cleveland,
Ohio
– Dutch specialty training in General
Practice
– McMaster Modular Assessment
Program in Emergency Medicine
– Graduate entry program Maastricht
From theory back to practice
• Existing best practices:
– Veterinary education Utrecht
– Cleveland Learner Clinic, Cleveland, Ohio
– Dutch specialty training in General Practice
– Canadian specialty training in General
Practice
– McMaster Modular Assessment Program in
Emergency Medicine
– Graduate entry program Maastricht
Physician-clinical investigator program
• 4 year graduate entry program
• Competency-based (Canmeds) with emphasis on
research
• PBL program
– Year 1: classic PBL
– Year 2: real patient PBL
– Year 3: clerkship rotations
– Year 4: participation in research and health care
• High expectations of students: in terms of
motivation, promotion of excellence, self-
directedness
The assessment program
• Assessment in Modules: assignments, presentations, end-examination,
etc.
• Longitudinal assessment: assignments, reviews, projects, progress
tests, evaluation of professional behavior, etc.
• All assessment is informative and low stake formative
• The portfolio is central instrument
Module-overstijgende toetsing van professioneel gedrag
Module 2 Module 3 Module 4Module 1
PT 1 PT2 PT 3 PT 4
Longitudinal Module exceeding assessment of knowledge, skills and professional behavior
portfolioCounselor
meeting
Counselor
meeting
Counselor
meeting
Counselor
meeting
Module exceeding assessment of knowledge in Progress Test
Longitudinal total test scores across 12
measurement moments and predicted future
performance
Maastricht Electronic portfolio
(ePass)
Comparison
between the score
of the student and
the average score
of his/her peers.
Every blue dot
corresponds to
an assessment
form included in
the portfolio.
Maastricht Electronic portfolio
(ePass)
Coaching by counselors
• Coaching is essential for successful use of reflective
learning skills
• Counselor gives advice/comments (whether asked or not)
• He/she counsels if choices have to be made
• He/she guards and discusses study progress and
development of competencies
Decision-making by committee
• Committee of counselors and externals
• Decision is based on portfolio information & counselor
recommendation, competency standards
• Deliberation is proportional to clarity of information
• Decisions are justified when needed; remediation
recommendation may be provided
Strategy to
establish
trustworthiness Criteria
Potential
Assessment Strategy
(sample)
Credibility Prolonged engagement Training of examiners
Triangulation Tailored volume of expert
judgment based on certainty of
information
Peer examination Benchmarking examiners
Member checking Incorporate learner view
Structural coherence Scrutiny of committee
inconsistencies
Transferability Time sampling Judgment based on broad
sample of data points
Thick description Justify decisions
Dependability Stepwise replication Use multiple assessors who have
credibility
Confirmability Audit Give learners the possibility to
appeal to the assessment
decision
Progress Test February 2012
Overview
• From practice to research
• From research to theory
• From theory to practice
• Conclusions
Conclusions 1: The way forward
• We have to stop thinking in terms of individual
assessment methods
• A systematic and programmatic approach is
needed, longitudinally oriented
• Every method of assessment may be functional
(old and new; standardized and unstandardized)
• Professional judgment is imperative (similar to
clinical practice)
• Subjectivity is dealt with through sampling and
procedural bias reduction methods (not with
standardization or objectification).
Conclusions 2: The way forward
• The programmatic approach to
assessment optimizes:
– The learning function (through
information richness)
– The pass/fail decision function
(through the combination of rich
information)
Further reading:
www.ceesvandervleuten.com

Weitere ähnliche Inhalte

Was ist angesagt?

e-Learning in medical education
e-Learning in medical educatione-Learning in medical education
e-Learning in medical educationFazlulKabir4
 
Internal assessment & formative assessment
Internal assessment  & formative assessmentInternal assessment  & formative assessment
Internal assessment & formative assessmentMunmunDasSarkar
 
Introduction to Systematic Reviews
Introduction to Systematic ReviewsIntroduction to Systematic Reviews
Introduction to Systematic ReviewsLaura Koltutsky
 
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.pptWORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.pptAbdul Jabbar Arif
 
MEU WORKSHOP Clinical Skills Education
MEU WORKSHOP Clinical Skills EducationMEU WORKSHOP Clinical Skills Education
MEU WORKSHOP Clinical Skills EducationDevan Pannen
 
Optimum assessment of cognitive domain in medical education
Optimum assessment of cognitive domain in medical educationOptimum assessment of cognitive domain in medical education
Optimum assessment of cognitive domain in medical educationK Raman Sethuraman
 
Introduction to REVISED BASIC COURSE WORKSHOP IN MED EDUCATION
Introduction to REVISED BASIC COURSE WORKSHOP IN MED EDUCATIONIntroduction to REVISED BASIC COURSE WORKSHOP IN MED EDUCATION
Introduction to REVISED BASIC COURSE WORKSHOP IN MED EDUCATIONanitasreekanth
 
Introduction of Medical Education Domain
Introduction of Medical Education Domain Introduction of Medical Education Domain
Introduction of Medical Education Domain Md Yunus
 
Newer Methods of Assessment in Medical Education
Newer Methods of Assessment in  Medical EducationNewer Methods of Assessment in  Medical Education
Newer Methods of Assessment in Medical EducationSwati Deshpande
 
Teaching clinical skills slideshare
Teaching clinical skills slideshareTeaching clinical skills slideshare
Teaching clinical skills slideshareMahmoud Abdel-Aleem
 
The Evaluation Checklist
The Evaluation ChecklistThe Evaluation Checklist
The Evaluation Checklistwmartz
 

Was ist angesagt? (20)

e-Learning in medical education
e-Learning in medical educatione-Learning in medical education
e-Learning in medical education
 
Curriculum development in medical education
Curriculum development in medical educationCurriculum development in medical education
Curriculum development in medical education
 
Internal assessment & formative assessment
Internal assessment  & formative assessmentInternal assessment  & formative assessment
Internal assessment & formative assessment
 
Chossing right assessment method
Chossing right assessment methodChossing right assessment method
Chossing right assessment method
 
Choosing right teaching methods
Choosing right teaching methodsChoosing right teaching methods
Choosing right teaching methods
 
Introduction to Systematic Reviews
Introduction to Systematic ReviewsIntroduction to Systematic Reviews
Introduction to Systematic Reviews
 
ATCOM Booklet
ATCOM BookletATCOM Booklet
ATCOM Booklet
 
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.pptWORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
 
MEU WORKSHOP Clinical Skills Education
MEU WORKSHOP Clinical Skills EducationMEU WORKSHOP Clinical Skills Education
MEU WORKSHOP Clinical Skills Education
 
Optimum assessment of cognitive domain in medical education
Optimum assessment of cognitive domain in medical educationOptimum assessment of cognitive domain in medical education
Optimum assessment of cognitive domain in medical education
 
Introduction to REVISED BASIC COURSE WORKSHOP IN MED EDUCATION
Introduction to REVISED BASIC COURSE WORKSHOP IN MED EDUCATIONIntroduction to REVISED BASIC COURSE WORKSHOP IN MED EDUCATION
Introduction to REVISED BASIC COURSE WORKSHOP IN MED EDUCATION
 
AETCOM ppt by prof satya 2020
AETCOM ppt by prof satya 2020 AETCOM ppt by prof satya 2020
AETCOM ppt by prof satya 2020
 
Introduction of Medical Education Domain
Introduction of Medical Education Domain Introduction of Medical Education Domain
Introduction of Medical Education Domain
 
Newer Methods of Assessment in Medical Education
Newer Methods of Assessment in  Medical EducationNewer Methods of Assessment in  Medical Education
Newer Methods of Assessment in Medical Education
 
MEDICAL ELECTIVES.pptx
MEDICAL ELECTIVES.pptxMEDICAL ELECTIVES.pptx
MEDICAL ELECTIVES.pptx
 
Critical Appraisal
Critical Appraisal Critical Appraisal
Critical Appraisal
 
Osce and ospe
Osce and ospeOsce and ospe
Osce and ospe
 
Teaching clinical skills slideshare
Teaching clinical skills slideshareTeaching clinical skills slideshare
Teaching clinical skills slideshare
 
The Evaluation Checklist
The Evaluation ChecklistThe Evaluation Checklist
The Evaluation Checklist
 
Assessment and evaluation
Assessment and evaluationAssessment and evaluation
Assessment and evaluation
 

Andere mochten auch

Developing Milestones for GI
Developing Milestones for GIDeveloping Milestones for GI
Developing Milestones for GIjakinyi
 
Competency-based Medical Education
Competency-based Medical EducationCompetency-based Medical Education
Competency-based Medical EducationJanet Corral
 
How to Create a Global PBI Lesson Plan
How to Create a Global PBI Lesson Plan How to Create a Global PBI Lesson Plan
How to Create a Global PBI Lesson Plan Participate
 
Competency-based assessment: The good, the bad, and the puzzling
Competency-based assessment:The good, the bad, and the puzzlingCompetency-based assessment:The good, the bad, and the puzzling
Competency-based assessment: The good, the bad, and the puzzlingMedCouncilCan
 
Technology to Personalize Learning for Gifted Kids
Technology to Personalize Learning for Gifted KidsTechnology to Personalize Learning for Gifted Kids
Technology to Personalize Learning for Gifted KidsBrian Housand
 

Andere mochten auch (6)

Developing Milestones for GI
Developing Milestones for GIDeveloping Milestones for GI
Developing Milestones for GI
 
Competency-based Medical Education
Competency-based Medical EducationCompetency-based Medical Education
Competency-based Medical Education
 
McGill Workshop
McGill WorkshopMcGill Workshop
McGill Workshop
 
How to Create a Global PBI Lesson Plan
How to Create a Global PBI Lesson Plan How to Create a Global PBI Lesson Plan
How to Create a Global PBI Lesson Plan
 
Competency-based assessment: The good, the bad, and the puzzling
Competency-based assessment:The good, the bad, and the puzzlingCompetency-based assessment:The good, the bad, and the puzzling
Competency-based assessment: The good, the bad, and the puzzling
 
Technology to Personalize Learning for Gifted Kids
Technology to Personalize Learning for Gifted KidsTechnology to Personalize Learning for Gifted Kids
Technology to Personalize Learning for Gifted Kids
 

Ähnlich wie A journey towards programmatic assessment

Theoretical Constructs and Best Practice in Assessment
Theoretical Constructs and Best Practice in Assessment Theoretical Constructs and Best Practice in Assessment
Theoretical Constructs and Best Practice in Assessment Touchstone Institute
 
Maggie Cruickshank - Optional work place assessment of training
Maggie Cruickshank - Optional work place assessment of training   Maggie Cruickshank - Optional work place assessment of training
Maggie Cruickshank - Optional work place assessment of training triumphbenelux
 
Keys to success with assessment and evaluation
Keys to success with assessment and evaluationKeys to success with assessment and evaluation
Keys to success with assessment and evaluationFrank Cervone
 
Don't tell me show me ! - Lambert, Foley, Moore and Hinds
Don't tell me show me ! - Lambert, Foley, Moore and HindsDon't tell me show me ! - Lambert, Foley, Moore and Hinds
Don't tell me show me ! - Lambert, Foley, Moore and Hindsmdxaltc
 
ASSESSMENT IN MEDICAL EDUCATION07122022.pptx
ASSESSMENT IN MEDICAL EDUCATION07122022.pptxASSESSMENT IN MEDICAL EDUCATION07122022.pptx
ASSESSMENT IN MEDICAL EDUCATION07122022.pptxMuralidharanp13
 
Introduction to assessment
Introduction to assessmentIntroduction to assessment
Introduction to assessmentRitu71
 
Assessment planning and quality assurence by Dr. Debjani Sengupta
Assessment planning and quality assurence by Dr. Debjani SenguptaAssessment planning and quality assurence by Dr. Debjani Sengupta
Assessment planning and quality assurence by Dr. Debjani SenguptaDrAnindyaDasgupta
 
Evaluation in Education
Evaluation in EducationEvaluation in Education
Evaluation in EducationKusum Gaur
 
Moderation workshop dietetics april 2017b
Moderation workshop dietetics april 2017bModeration workshop dietetics april 2017b
Moderation workshop dietetics april 2017bRita Ndagire Kizito
 
Alicia O' Cathain: qualitative research and evaluation
Alicia O' Cathain: qualitative research and evaluationAlicia O' Cathain: qualitative research and evaluation
Alicia O' Cathain: qualitative research and evaluationNuffield Trust
 
ASSESSMENT and EVALUATION.pptx
ASSESSMENT and EVALUATION.pptxASSESSMENT and EVALUATION.pptx
ASSESSMENT and EVALUATION.pptxGIRLYCAMACHO
 
Class 6 research quality in qualitative methods 3 2-17
Class 6 research quality in qualitative methods 3 2-17Class 6 research quality in qualitative methods 3 2-17
Class 6 research quality in qualitative methods 3 2-17tjcarter
 
How to Assess Students in Virtual Environment?
How to Assess Students in Virtual Environment?How to Assess Students in Virtual Environment?
How to Assess Students in Virtual Environment?Attiya Karim Balghari
 
Mentoring in practice portfolio guidance
Mentoring in practice portfolio guidanceMentoring in practice portfolio guidance
Mentoring in practice portfolio guidanceGemma Ryan
 
Evaluation the many faces
Evaluation   the many facesEvaluation   the many faces
Evaluation the many facesleesha roberts
 

Ähnlich wie A journey towards programmatic assessment (20)

Theoretical Constructs and Best Practice in Assessment
Theoretical Constructs and Best Practice in Assessment Theoretical Constructs and Best Practice in Assessment
Theoretical Constructs and Best Practice in Assessment
 
Chapter 14
Chapter 14Chapter 14
Chapter 14
 
Maggie Cruickshank - Optional work place assessment of training
Maggie Cruickshank - Optional work place assessment of training   Maggie Cruickshank - Optional work place assessment of training
Maggie Cruickshank - Optional work place assessment of training
 
Keys to success with assessment and evaluation
Keys to success with assessment and evaluationKeys to success with assessment and evaluation
Keys to success with assessment and evaluation
 
Don't tell me show me ! - Lambert, Foley, Moore and Hinds
Don't tell me show me ! - Lambert, Foley, Moore and HindsDon't tell me show me ! - Lambert, Foley, Moore and Hinds
Don't tell me show me ! - Lambert, Foley, Moore and Hinds
 
ASSESSMENT IN MEDICAL EDUCATION07122022.pptx
ASSESSMENT IN MEDICAL EDUCATION07122022.pptxASSESSMENT IN MEDICAL EDUCATION07122022.pptx
ASSESSMENT IN MEDICAL EDUCATION07122022.pptx
 
Introduction to assessment
Introduction to assessmentIntroduction to assessment
Introduction to assessment
 
ARLG 2019: Myer have we made a difference
ARLG 2019: Myer have we made a differenceARLG 2019: Myer have we made a difference
ARLG 2019: Myer have we made a difference
 
Assessment planning and quality assurence by Dr. Debjani Sengupta
Assessment planning and quality assurence by Dr. Debjani SenguptaAssessment planning and quality assurence by Dr. Debjani Sengupta
Assessment planning and quality assurence by Dr. Debjani Sengupta
 
Evaluation in Education
Evaluation in EducationEvaluation in Education
Evaluation in Education
 
Moderation workshop dietetics april 2017b
Moderation workshop dietetics april 2017bModeration workshop dietetics april 2017b
Moderation workshop dietetics april 2017b
 
Finalreview
FinalreviewFinalreview
Finalreview
 
Alicia O' Cathain: qualitative research and evaluation
Alicia O' Cathain: qualitative research and evaluationAlicia O' Cathain: qualitative research and evaluation
Alicia O' Cathain: qualitative research and evaluation
 
CoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
CoLT 2017 - Prof. Dr. Vishna Devi NadarajahCoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
CoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
 
ASSESSMENT and EVALUATION.pptx
ASSESSMENT and EVALUATION.pptxASSESSMENT and EVALUATION.pptx
ASSESSMENT and EVALUATION.pptx
 
Class 6 research quality in qualitative methods 3 2-17
Class 6 research quality in qualitative methods 3 2-17Class 6 research quality in qualitative methods 3 2-17
Class 6 research quality in qualitative methods 3 2-17
 
Principles of student_assessment_satya_
Principles of student_assessment_satya_Principles of student_assessment_satya_
Principles of student_assessment_satya_
 
How to Assess Students in Virtual Environment?
How to Assess Students in Virtual Environment?How to Assess Students in Virtual Environment?
How to Assess Students in Virtual Environment?
 
Mentoring in practice portfolio guidance
Mentoring in practice portfolio guidanceMentoring in practice portfolio guidance
Mentoring in practice portfolio guidance
 
Evaluation the many faces
Evaluation   the many facesEvaluation   the many faces
Evaluation the many faces
 

Mehr von MedCouncilCan

Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
Peering through the Looking Glass: Towards a Programmatic View of the Qualify...Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
Peering through the Looking Glass: Towards a Programmatic View of the Qualify...MedCouncilCan
 
Pushing the Boundaries of Medical Licensing
Pushing the Boundaries of Medical Licensing Pushing the Boundaries of Medical Licensing
Pushing the Boundaries of Medical Licensing MedCouncilCan
 
Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...MedCouncilCan
 
Séance d'orientation pour les candidats de l'examen de la CNE 2015
Séance d'orientation pour les candidats de l'examen de la CNE 2015Séance d'orientation pour les candidats de l'examen de la CNE 2015
Séance d'orientation pour les candidats de l'examen de la CNE 2015MedCouncilCan
 
2015 NAC candidate orientation presentation
2015 NAC candidate orientation presentation2015 NAC candidate orientation presentation
2015 NAC candidate orientation presentationMedCouncilCan
 
Surveying the landscape: An overview of tools for direct observation and asse...
Surveying the landscape: An overview of tools for direct observation and asse...Surveying the landscape: An overview of tools for direct observation and asse...
Surveying the landscape: An overview of tools for direct observation and asse...MedCouncilCan
 
The New Blueprint: challenging the comfort zone
The New Blueprint: challenging the comfort zoneThe New Blueprint: challenging the comfort zone
The New Blueprint: challenging the comfort zoneMedCouncilCan
 
Séance d'orientation pour les candidats de l'examen de la CNE de 2014
Séance d'orientation pour les candidats de l'examen de la CNE de 2014Séance d'orientation pour les candidats de l'examen de la CNE de 2014
Séance d'orientation pour les candidats de l'examen de la CNE de 2014MedCouncilCan
 
2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentationMedCouncilCan
 
NAC PRA update - 2014 Ottawa Conference
NAC PRA update - 2014 Ottawa ConferenceNAC PRA update - 2014 Ottawa Conference
NAC PRA update - 2014 Ottawa ConferenceMedCouncilCan
 
Blueprinting update - 2014 Ottawa Conference
Blueprinting update - 2014 Ottawa ConferenceBlueprinting update - 2014 Ottawa Conference
Blueprinting update - 2014 Ottawa ConferenceMedCouncilCan
 
Orientation à l’intention des candidats au format ECOS (2014)
Orientation à l’intention des candidats au format ECOS (2014)Orientation à l’intention des candidats au format ECOS (2014)
Orientation à l’intention des candidats au format ECOS (2014)MedCouncilCan
 
2014 Candidate Orientation Presentation - Certification Examination in Family...
2014 Candidate Orientation Presentation - Certification Examination in Family...2014 Candidate Orientation Presentation - Certification Examination in Family...
2014 Candidate Orientation Presentation - Certification Examination in Family...MedCouncilCan
 
Présentation d'orientation de 2014 - EACMC, partie II
Présentation d'orientation de 2014 - EACMC, partie IIPrésentation d'orientation de 2014 - EACMC, partie II
Présentation d'orientation de 2014 - EACMC, partie IIMedCouncilCan
 
Présentation d'orientation - Examen de la CNE 2014
Présentation d'orientation - Examen de la CNE 2014Présentation d'orientation - Examen de la CNE 2014
Présentation d'orientation - Examen de la CNE 2014MedCouncilCan
 
2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation 2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation MedCouncilCan
 
Pre-Exam Orientation for Candidates taking the Certification Examination in F...
Pre-Exam Orientation for Candidates taking the Certification Examination in F...Pre-Exam Orientation for Candidates taking the Certification Examination in F...
Pre-Exam Orientation for Candidates taking the Certification Examination in F...MedCouncilCan
 
Pre-Exam Candidate Orientation
Pre-Exam Candidate OrientationPre-Exam Candidate Orientation
Pre-Exam Candidate OrientationMedCouncilCan
 
Practice Ready Assessment for IMG Physicians
Practice Ready Assessment for IMG PhysiciansPractice Ready Assessment for IMG Physicians
Practice Ready Assessment for IMG PhysiciansMedCouncilCan
 

Mehr von MedCouncilCan (20)

Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
Peering through the Looking Glass: Towards a Programmatic View of the Qualify...Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
Peering through the Looking Glass: Towards a Programmatic View of the Qualify...
 
Pushing the Boundaries of Medical Licensing
Pushing the Boundaries of Medical Licensing Pushing the Boundaries of Medical Licensing
Pushing the Boundaries of Medical Licensing
 
Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...
 
Séance d'orientation pour les candidats de l'examen de la CNE 2015
Séance d'orientation pour les candidats de l'examen de la CNE 2015Séance d'orientation pour les candidats de l'examen de la CNE 2015
Séance d'orientation pour les candidats de l'examen de la CNE 2015
 
2015 NAC candidate orientation presentation
2015 NAC candidate orientation presentation2015 NAC candidate orientation presentation
2015 NAC candidate orientation presentation
 
Surveying the landscape: An overview of tools for direct observation and asse...
Surveying the landscape: An overview of tools for direct observation and asse...Surveying the landscape: An overview of tools for direct observation and asse...
Surveying the landscape: An overview of tools for direct observation and asse...
 
The New Blueprint: challenging the comfort zone
The New Blueprint: challenging the comfort zoneThe New Blueprint: challenging the comfort zone
The New Blueprint: challenging the comfort zone
 
The
The The
The
 
Séance d'orientation pour les candidats de l'examen de la CNE de 2014
Séance d'orientation pour les candidats de l'examen de la CNE de 2014Séance d'orientation pour les candidats de l'examen de la CNE de 2014
Séance d'orientation pour les candidats de l'examen de la CNE de 2014
 
2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation
 
NAC PRA update - 2014 Ottawa Conference
NAC PRA update - 2014 Ottawa ConferenceNAC PRA update - 2014 Ottawa Conference
NAC PRA update - 2014 Ottawa Conference
 
Blueprinting update - 2014 Ottawa Conference
Blueprinting update - 2014 Ottawa ConferenceBlueprinting update - 2014 Ottawa Conference
Blueprinting update - 2014 Ottawa Conference
 
Orientation à l’intention des candidats au format ECOS (2014)
Orientation à l’intention des candidats au format ECOS (2014)Orientation à l’intention des candidats au format ECOS (2014)
Orientation à l’intention des candidats au format ECOS (2014)
 
2014 Candidate Orientation Presentation - Certification Examination in Family...
2014 Candidate Orientation Presentation - Certification Examination in Family...2014 Candidate Orientation Presentation - Certification Examination in Family...
2014 Candidate Orientation Presentation - Certification Examination in Family...
 
Présentation d'orientation de 2014 - EACMC, partie II
Présentation d'orientation de 2014 - EACMC, partie IIPrésentation d'orientation de 2014 - EACMC, partie II
Présentation d'orientation de 2014 - EACMC, partie II
 
Présentation d'orientation - Examen de la CNE 2014
Présentation d'orientation - Examen de la CNE 2014Présentation d'orientation - Examen de la CNE 2014
Présentation d'orientation - Examen de la CNE 2014
 
2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation 2014 NAC candidate orientation presentation
2014 NAC candidate orientation presentation
 
Pre-Exam Orientation for Candidates taking the Certification Examination in F...
Pre-Exam Orientation for Candidates taking the Certification Examination in F...Pre-Exam Orientation for Candidates taking the Certification Examination in F...
Pre-Exam Orientation for Candidates taking the Certification Examination in F...
 
Pre-Exam Candidate Orientation
Pre-Exam Candidate OrientationPre-Exam Candidate Orientation
Pre-Exam Candidate Orientation
 
Practice Ready Assessment for IMG Physicians
Practice Ready Assessment for IMG PhysiciansPractice Ready Assessment for IMG Physicians
Practice Ready Assessment for IMG Physicians
 

Kürzlich hochgeladen

How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17Celine George
 
Philosophy of Education and Educational Philosophy
Philosophy of Education  and Educational PhilosophyPhilosophy of Education  and Educational Philosophy
Philosophy of Education and Educational PhilosophyShuvankar Madhu
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17Celine George
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptxmary850239
 
How to Print Employee Resume in the Odoo 17
How to Print Employee Resume in the Odoo 17How to Print Employee Resume in the Odoo 17
How to Print Employee Resume in the Odoo 17Celine George
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 
M-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxM-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxDr. Santhosh Kumar. N
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17Celine George
 
3.19.24 Urban Uprisings and the Chicago Freedom Movement.pptx
3.19.24 Urban Uprisings and the Chicago Freedom Movement.pptx3.19.24 Urban Uprisings and the Chicago Freedom Movement.pptx
3.19.24 Urban Uprisings and the Chicago Freedom Movement.pptxmary850239
 
Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesMohammad Hassany
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxKatherine Villaluna
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.EnglishCEIPdeSigeiro
 
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfP4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfYu Kanazawa / Osaka University
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationMJDuyan
 
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxPISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxEduSkills OECD
 
How to Filter Blank Lines in Odoo 17 Accounting
How to Filter Blank Lines in Odoo 17 AccountingHow to Filter Blank Lines in Odoo 17 Accounting
How to Filter Blank Lines in Odoo 17 AccountingCeline George
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxraviapr7
 
How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17Celine George
 

Kürzlich hochgeladen (20)

How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17
 
Philosophy of Education and Educational Philosophy
Philosophy of Education  and Educational PhilosophyPhilosophy of Education  and Educational Philosophy
Philosophy of Education and Educational Philosophy
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptx
 
How to Print Employee Resume in the Odoo 17
How to Print Employee Resume in the Odoo 17How to Print Employee Resume in the Odoo 17
How to Print Employee Resume in the Odoo 17
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 
M-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxM-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptx
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17
 
3.19.24 Urban Uprisings and the Chicago Freedom Movement.pptx
3.19.24 Urban Uprisings and the Chicago Freedom Movement.pptx3.19.24 Urban Uprisings and the Chicago Freedom Movement.pptx
3.19.24 Urban Uprisings and the Chicago Freedom Movement.pptx
 
Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming Classes
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptx
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.
 
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfP4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive Education
 
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxPISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
 
How to Filter Blank Lines in Odoo 17 Accounting
How to Filter Blank Lines in Odoo 17 AccountingHow to Filter Blank Lines in Odoo 17 Accounting
How to Filter Blank Lines in Odoo 17 Accounting
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptx
 
How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17
 

A journey towards programmatic assessment

  • 1. A journey towards programmatic assessment Medical Council of Canada’s 2015 Annual Meeting September 27, 2015 Cees van der Vleuten Maastricht University The Netherlands www.ceesvandervleuten.com
  • 3. Overview • From practice to research • From research to theory • From theory to practice • Conclusions
  • 4. The Toolbox • MCQ, MEQ, OEQ, SIMP, Write- ins, Key Feature, Progress test, PMP, SCT, Viva, Long case, Short case, OSCE, OSPE, DOCEE, SP-based test, Video assessment, MSF, Mini-CEX, DOPS, assessment center, self- assessment, peer assessment, incognito SPs, portfolio………….
  • 5. Knows Shows how Knows how Does Knows Fact-oriented assessment: MCQ, write-ins, oral….. Knows how Scenario or case-based assessment: MCQ, write-ins, oral….. Shows how Performance assessment in vitro: Assessment centers, OSCE….. Does Performance assessment in vivo: In situ performance assessment, 360۫ , Peer assesme The way we climbed......
  • 7. Validity: what are we assessing? • Curricula have changed from an input orientation to an output orientation • We went from haphazard learning to integrated learning objectives, to end objectives, and now to (generic) competencies • We went from teacher oriented programs to learning oriented, self-directed programs
  • 8. Competency-frameworks CanMeds  Medical expert  Communicator  Collaborator  Manager  Health advocate  Scholar  Professional ACGME  Medical knowledge  Patient care  Practice-based learning & improvement  Interpersonal and communication skills  Professionalism  Systems-based practice GMC  Good clinical care  Relationships with patients and families  Working with colleagues  Managing the workplace  Social responsibility and accountability  Professionalism
  • 9. Knows Shows how Knows how Does Knows Knows how Shows how Does Validity: what are we assessing? Standardized assessment (fairly established) Unstandardized assessment (emerging)
  • 10. Messages from validity research • There is no magic bullet; we need a mixture of methods to cover the competency pyramid • We need BOTH standardized and non- standardized assessment methods • For standardized assessment quality control around test development and administration is vital • For unstandardized assessment the users (the people) are vital.
  • 11. Method reliability as a function of testing time Testing Time in Hours 1 2 4 8 MCQ1 0.62 0.77 0.87 0.93 Case- Based Short Essay2 0.68 0.81 0.89 0.94 PMP1 0.36 0.53 0.69 0.82 Oral Exam3 0.50 0.67 0.80 0.89 Long Case4 0.60 0.75 0.86 0.92 OSCE5 0.54 0.70 0.82 0.90 Practice Video Assess- ment7 0.62 0.77 0.87 0.93 1Norcini et al., 1985 2Stalenhoef-Halling et al., 1990 3Swanson, 1987 4Wass et al., 2001 5Van der Vleuten, 1988 6Norcini et al., 1999 In- cognito SPs8 0.61 0.76 0.86 0.93 Mini CEX6 0.73 0.84 0.92 0.96 7Ram et al., 1999 8Gorter, 2002
  • 12. Reliability as a function of sample size (Moonen et al., 2013) 0.65 0.7 0.75 0.8 0.85 0.9 4 5 6 7 8 9 10 11 12 G=0.80 KPB Mini-CEX
  • 13. 0.65 0.7 0.75 0.8 0.85 0.9 4 5 6 7 8 9 10 11 12 G=0.80 KPB OSATS Mini-CEX OSATS Reliability as a function of sample size (Moonen et al., 2013)
  • 14. 0.65 0.7 0.75 0.8 0.85 0.9 4 5 6 7 8 9 10 11 12 Mini-CEX OSATS MSF Reliability as a function of sample size (Moonen et al., 2013)
  • 15. Effect of aggregation across methods (Moonen et al., 2013) Method Mini-CEX OSATS MSF Sample needed when used as stand-alone 8 9 9 Sample needed when used as a composite 5 6 2
  • 16. Messages from reliability research • Acceptable reliability is only achieved with large samples of test elements (contexts, cases) and assessors • No method is inherently better than any other (that includes the new ones!) • Objectivity is NOT equal to reliability • Many subjective judgments are pretty reproducible/reliable.
  • 17. Educational impact: How does assessment drive learning? • Relationship is complex (cf. Cilliers, 2011, 2012) • But impact is often very negative – Poor learning styles – Grade culture (grade hunting, competitiveness) – Grade inflation (e.g. in the workplace) • A lot of REDUCTIONISM! – Little feedback (grade is poorest form of feedback one can get) – Non-alignment with curricular goals – Non-meaningful aggregation of assessment information – Few longitudinal elements – Tick-box exercises (OSCEs, logbooks, work-based assessment).
  • 18. • All learners construct knowledge from an inner scaffolding of their individual and social experiences, emotions, will, aptitudes, beliefs, values, self-awareness, purpose, and more . . . if you are learning ….., what you understand is determined by how you understand things, who you are, and what you already know. Peter Senge, Director of the Center for Organizational Learning at MIT (as cited in van Ryn et al., 2014)
  • 19. Messages learning impact research • No assessment without (meaningful) feedback • Narrative feedback has a lot more impact on complex skills than scores • Provision of feedback is not enough (feedback is a dialogue) • Longitudinal assessment is needed.
  • 20. Overview • From practice to research • From research to theory • From theory to practice • Conclusions
  • 21. Limitations of the single-method approach • No single method can do it all • Each individual method has (significant) limitations • Each single method is a considerable compromise on reliability, validity, educational impact
  • 22. Implications • Validity: a multitude of methods needed • Reliability: a lot of (combined) information is needed • Learning impact: assessment should provide (longitudinal) meaningful information for learning Programmatic assessment
  • 23. Programmatic assessment • A curriculum is a good metaphor; in a program of assessment: – Elements are planned, arranged, coordinated – Is systematically evaluated and reformed • But how? (the literature provides extremely little support!)
  • 24. Programmatic assessment • Dijkstra et al 2012: 73 generic guidelines • To be done: – Further validation – A feasible (self-assessment) instrument • ASPIRE assessment criteria
  • 25. Building blocks for programmatic assessment 1 • Every assessment is but one data point (Δ) • Every data point is optimized for learning – Information rich (quantitative, qualitative) – Meaningful – Variation in format • Summative versus formative is replaced by a continuum of stakes (stakes) • N data points are proportionally related to the stakes of the decision to be taken.
  • 26. Continuum of stakes, number of data point and their function No stake Very high stake One Data point: • Focused on information • Feedback oriented • Not decision oriented Intermediate progress decisions: • More data points needed • Focus on diagnosis, remediation, prediction Final decisions on promotion or selection: • Many data points needed • Focused on a (non- surprising) heavy decision
  • 28. Classical approach to aggregation Method 1 to assess skill A Σ Method 2 to assess skill B Σ Σ Σ Method 3 to assess skill C Method 4 to assess skill C
  • 29. More meaningful aggregation Method 1 Σ Method 2 Σ Method 3 Σ Method 4 Σ Skill A Skill B B Skill C Skill D
  • 34. Overview • From practice to research • From research to theory • From theory to practice • Conclusions
  • 35. From theory back to practice • Existing best practices: – Veterinary education Utrecht – Cleveland Learner Clinic, Cleveland, Ohio – Dutch specialty training in General Practice – McMaster Modular Assessment Program in Emergency Medicine – Graduate entry program Maastricht
  • 36. From theory back to practice • Existing best practices: – Veterinary education Utrecht – Cleveland Learner Clinic, Cleveland, Ohio – Dutch specialty training in General Practice – Canadian specialty training in General Practice – McMaster Modular Assessment Program in Emergency Medicine – Graduate entry program Maastricht
  • 37. Physician-clinical investigator program • 4 year graduate entry program • Competency-based (Canmeds) with emphasis on research • PBL program – Year 1: classic PBL – Year 2: real patient PBL – Year 3: clerkship rotations – Year 4: participation in research and health care • High expectations of students: in terms of motivation, promotion of excellence, self- directedness
  • 38. The assessment program • Assessment in Modules: assignments, presentations, end-examination, etc. • Longitudinal assessment: assignments, reviews, projects, progress tests, evaluation of professional behavior, etc. • All assessment is informative and low stake formative • The portfolio is central instrument Module-overstijgende toetsing van professioneel gedrag Module 2 Module 3 Module 4Module 1 PT 1 PT2 PT 3 PT 4 Longitudinal Module exceeding assessment of knowledge, skills and professional behavior portfolioCounselor meeting Counselor meeting Counselor meeting Counselor meeting Module exceeding assessment of knowledge in Progress Test
  • 39. Longitudinal total test scores across 12 measurement moments and predicted future performance
  • 40. Maastricht Electronic portfolio (ePass) Comparison between the score of the student and the average score of his/her peers.
  • 41. Every blue dot corresponds to an assessment form included in the portfolio. Maastricht Electronic portfolio (ePass)
  • 42. Coaching by counselors • Coaching is essential for successful use of reflective learning skills • Counselor gives advice/comments (whether asked or not) • He/she counsels if choices have to be made • He/she guards and discusses study progress and development of competencies
  • 43. Decision-making by committee • Committee of counselors and externals • Decision is based on portfolio information & counselor recommendation, competency standards • Deliberation is proportional to clarity of information • Decisions are justified when needed; remediation recommendation may be provided
  • 44. Strategy to establish trustworthiness Criteria Potential Assessment Strategy (sample) Credibility Prolonged engagement Training of examiners Triangulation Tailored volume of expert judgment based on certainty of information Peer examination Benchmarking examiners Member checking Incorporate learner view Structural coherence Scrutiny of committee inconsistencies Transferability Time sampling Judgment based on broad sample of data points Thick description Justify decisions Dependability Stepwise replication Use multiple assessors who have credibility Confirmability Audit Give learners the possibility to appeal to the assessment decision
  • 46. Overview • From practice to research • From research to theory • From theory to practice • Conclusions
  • 47. Conclusions 1: The way forward • We have to stop thinking in terms of individual assessment methods • A systematic and programmatic approach is needed, longitudinally oriented • Every method of assessment may be functional (old and new; standardized and unstandardized) • Professional judgment is imperative (similar to clinical practice) • Subjectivity is dealt with through sampling and procedural bias reduction methods (not with standardization or objectification).
  • 48. Conclusions 2: The way forward • The programmatic approach to assessment optimizes: – The learning function (through information richness) – The pass/fail decision function (through the combination of rich information)