1. ALERT Presentation:
Using simulation for competency-based assessment of
pediatric residents within the Milestones Project
Leah Mallory, MD
The Barbara Bush Children’s Hospital
at Maine Medical Center
International Network for Simulation-based Pediatric Innovation, Research and Education

2. • More and more pediatric residency programs are developing
simulation curricula.
• ACGME has moved toward competency-based assessment with
the Milestones Project
• Educators are seeking objective ways to evaluate learners
within these milestones
• Simulation (manikin, standardized patient, skills trainer and
computer-based) offers rich opportunities for structured
assessment
International Network for Simulation-based Pediatric Innovation, Research and Education
Background

3. • P: Pediatric residency programs looking to use simulation as an
assessment tool
• I: Survey experts in pediatric residency training and simulation
based education to identify gaps in existing assessment methods
and opportunities for simulation-based assessment
• C:
• O: Generate a prioritized list of Milestones sub-competencies
ripe for simulation-based assessment as well as a review of the
literature of current simulation-based assessment tools
International Network for Simulation-based Pediatric Innovation, Research and Education
PICO Question

4. • Initial plan to use Delphi with group of pedi sim experts and program
directors to come to consensus regarding milestone-aligned priorities
for sim-based assessment
• Plan revised after INSPIRE meeting January 2014
– Survey to pedi program directors to identify gaps in current assessment methodologies
– Companion survey to pedi sim experts to identify areas best suited to sim-based
assessment
• Collaboration with Simulation Pediatric Education Group within APPD
– Presented idea and survey for feedback at April 2014 APPD meeting
– Concerns that asking PD’s to go through each milestone sub-competency would lead
to survey fatigue and weak data (lots of 2’s and 3’s on a likert scale)
– Better to return to consensus exercise with smaller, invested group of people who are
involved in pediatric resident education and simulation?
International Network for Simulation-based Pediatric Innovation, Research and Education
Approach / Design

5. 1. Two main questions to answer:
1. Where are the gaps in existing assessment methodologies with regards to pediatric milestone sub-competencies?
2. Which sub-competencies are best suited to simulation-based assessment?
2. Identify group of experts in Pediatric resident education and pediatric simulation
1. Back to delphi or other consensus building technique
2. In person exercise (AAP meeting in San Diego in October?)
3. Results define a prioritized list of areas for simulation-based assessment opportunities tied to pediatric milestones
4. Evidence-based review of current validated simulation-based assessment tools
5. Gap between identified simulation-based assessment priorities and existing simulation-based assessment tools defines
research agenda for development of new simulation-based assessment tools
International Network for Simulation-based Pediatric Innovation, Research and Education
Approach / Design

6. • Spring/Summer 2014
• Identify “experts” and invite to Consensus building exercise at AAP
• Design process for consensus building- use pediatric milestone sub-competencies
to identify gaps in existing assessment methods and opportunities for simulation-
based assessment
• October 2014
• Consensus building exercise in San Diego around AAP meeting
• November 2014
• Analyze results
• January 2015
• Developing manuscript for publication
International Network for Simulation-based Pediatric Innovation, Research and Education
Timeline

7. 1. Is this return to the “consensus building” study design (based
on APPD Sim PEG feedback) the right way to go?
2. If so, how do we define our experts?
3. If so, what does our consensus building exercise design look
like?
4. Should we also be more granular than the milestone sub-
competenies? (e.g. Patient Care #8 “Prescribe and perform all
medical procedures”- which procedures?)
International Network for Simulation-based Pediatric Innovation, Research and Education
3 Questions to improve study

8. Name Leah Mallory, MD
Institution The Barbara Bush Children’s Hospital at
Maine Medical Center
E-mail mallol@mmc.org, Phone 207-662-1504
Website / QR Code http://simulation.mmc.org/home
International Network for Simulation-based Pediatric Innovation, Research and Education
Contact Information