1. Assessing Procedural
Competencies
Amy S. Oxentenko, MD, FACP, FACG
Associate Professor of Medicine
Fellowship Director, Mayo Clinic-Rochester

2. Outline:
Procedural Competency
• The Numbers
• The Skills
• The Future

3. Competent or Not?
• 3rd year fellow • 3rd year fellow
• 285 colonoscopies • 427 colonoscopies
• Cecal intubation 95% • Cecal intubation 69%
• Adenoma detection 35% • Adenoma detection 12%
• Withdrawal 10 min • Withdrawal 6 min
• Patient tolerance 97% • Patient tolerance 85%
fair to excellent fair to excellent
YES NO

4. What Needs to be
Measured?
Competency
Minimum Defined
Procedure level of
Number ability

5. The Numbers

6. Minimum Cited Numbers
• Flex sigs 25-30
• EGDs 130
• Colons 140
– Based on goal of cecal intubation > 90%
– Others found competence at 275 cases1
– Others found competence at 500 cases2
1
Using Sedlack data
2
Using Spier data

7. Cecal Intubation Rate
Sedlack RE. Gastrointest Endosc 2011;74:355-66.

8. Spier BJ, et al. Gastrointest Endosc 2010;71:319-24.

9. Procedure Logs:
Not Just Numbers Anymore!
• “A skilled preceptor must be available to
teach and supervise the fellows in the
performance and interpretation of
procedures, which must be documented
in each fellow's record, including
indications, outcomes, diagnoses, and
supervisor(s).”
IV.A.6.d).(2) on Page 19, GI Program Requirements, “Tracked Changes” document

10. The Skills

11. Procedure Logging
• “Assessment of procedural competence should include
a formal evaluation process and NOT be based solely
on a minimum number of procedures performed.
• Each program must define criteria for competence for
all required and elective procedures.
• The record of evaluation must include the fellow’s
logbook or an equivalent method to demonstrate that
each fellow has achieved competence in the
performance of required procedures.”
V.A.1.a).(2) and 1.b).(1).(a) on Page 20-21, GI Requirements, “Tracked Changes” document

12. Multi-Society Evaluation Form
(MSEF)
• AASLD, ACG, AGA, ASGE
• Part of the GI Core Curriculum
• Third Edition, May 2007
http://www.asge.org/WorkArea/showcontent.aspx?id=3584

14. Any Downfalls of the MSEF?
• Lacks anchoring characteristics for all points
• Not validated for continuous assessment
• Grade inflation (our problem, not the form’s)
• Compare graduates across programs?
• What constitutes competent?

15. Mayo Colonoscopy Skills
Assessment Tool (MCSAT)
• 13-item survey
• Staff completed on each colon
• Took < 1 minute to complete
• Embedded in MERGE database
– Allowed for recording of procedure
# for fellow, fellow name, etc.
Sedlack RE. Gastrointest Endosc 2010;72:1125-33.

16. Sedlack RE. Gastrointest Endosc 2010;72:1125-33.

17. Overall Skill in Colonoscopy
Sedlack RE. Gastrointest Endosc 2011;74:355-66.

18. Other Date
Retrieved Per
Trainee

19. Barriers of the MCSAT or
Similar Systems?
• Many procedures performed
– Assessment needs to be quick/simple
• Differing procedures performed
– Similar models needed: EGD, PEGs, capsules, etc
• Compliance with completion
– Too easy for staff to forget or not take the time
• Differing endoscopy database systems
– No communication across programs

20. Sunrise: ProVation
October 2012
Sunset: MERGE
October 2012

21. ProVation Tracking: EGD

22. ProVation Tracking: Colons

23. Pros/Cons of ProVation
• PROS:
• Compliance with completion
– Automatic pop-up on all fellow EGDs and colons
– Staff cannot sign off until complete
• CONS:
• Yet another database
– How long with it be around?
– Not everyone has it
– Dependent on others to add features desired
• Detail desired
– Has to fit into radio buttons, brief, succinct

24. The Future

25. The Game Has Changed in
the Setting of NAS
• No longer a numbers game
• No longer a competency yes/no game
• Now it is all about meeting milestones
on the way to becoming competent
http://www.acgme-nas.org/assets/pdf/NEJMfinal.pdf

26. A Blueprint for Milestones
or Competency?
Unacceptable Competent Ideal
Adenoma >20%
Detection Rate
Colonoscopy > 6 min
Withdrawal Time
Cecal Intubation > 95%
Rate
Complication Rate < 1/200 bleed
<1/1000 perf
Polyp Retrieval > 95% > 10 mm
Rate > 80% < 10 mm
Patient Tolerance > 90% fair to
excellent

27. A Blueprint for Milestones
or Competency?
Unacceptable Competent Ideal
Adenoma > 20% >20%
Detection Rate
Colonoscopy 7-15 min > 6 min
Withdrawal Time
Cecal Intubation > 90% > 95%
Rate
Complication Rate < 1/200 bleed
<1/1000 perf
Polyp Retrieval > 95% > 10 mm
Rate > 80% < 10 mm
Patient Tolerance > 90% fair to
excellent

28. A Blueprint for Milestones
or Competency?
Unacceptable Competent Ideal
Adenoma < 15% > 20% >20%
Detection Rate
Colonoscopy > 20 min 7-15 min > 6 min
Withdrawal Time
Cecal Intubation < 80% > 90% > 95%
Rate
Complication Rate < 1/200 bleed
<1/1000 perf
Polyp Retrieval > 95% > 10 mm
Rate > 80% < 10 mm
Patient Tolerance > 90% fair to
excellent

29. Milestones and Narratives

30. Summary
• Procedure numbers are not enough
– An anchor at which competency
assessment should begin
• Procedure details are now needed
– Indication, findings, complications
• Competency tracking is required
– Milestones will pave the way, and they
need to be carefully developed

31. Thank you!
oxentenko.amy@mayo.edu