David Theobald Clinical Director National Endoscopy Quality Improvement Programme (Thursday, 11.15am, Forum Room)

1. Does ProVationtm MD
capture colonoscopy KPIs?
David Theobald
Clinical Director
National Endoscopy Quality Improvement Programme

2. National Endoscopy Quality
Improvement Programme
• Catalyst - Bowel Screening Pilot
• Appointment of NCLGIE and wider team
• Colonoscopy
• Widened brief

3. National Endoscopy Quality
Improvement Programme
Goal
The primary aim is to have every
endoscopy unit in the country
providing a patient focused service.

4. The strategy to reach this
goal will be
total service improvement
encompassing individual
performance, unit
performance and training

5. The methodology to reach
this goal will be
change based on evidence
(principles of the ‘Audit
Cycle’)

6. The primary tool to reach
this goal will be
the endoscopy
Global Rating Scale (GRS)

7. What is the Global Rating Scale (GRS)?
The GRS is a web-based self assessment tool
that provides a standard for accreditation and a
quality framework for service improvement

8. GRS Standards
1. Clinical Quality 2. Quality of the Patient Experience
1. Information/consent 1. Equality of access
2. Safety 2. Timeliness
3. Comfort 3. Choose and book
4. Quality 4. Privacy and dignity
5. Appropriateness 5. Aftercare
6. Results to referrer 6. Patient feedback

9. GRS Standards
1. Clinical Quality 2. Quality of the Patient Experience
1. Information/consent 1. Equality of access
2. Safety 2. Timeliness
3. Comfort 3. Choose and book
4. Quality 4. Privacy and dignity
5. Appropriateness 5. Aftercare
6. Results to referrer 6. Patient feedback
3. Workforce
1. Skill mix review and recruitment
2. Orientation and training
3. Assessment and appraisal
4. Staff are cared for
5. Staff are listened to

10. GRS Standards
1. Clinical Quality 2. Quality of the Patient Experience
1. Information/consent 1. Equality of access
2. Safety 2. Timeliness
3. Comfort 3. Choose and book
4. Quality 4. Privacy and dignity
5. Appropriateness 5. Aftercare
6. Results to referrer 6. Patient feedback
3. Workforce 4. Training
1. Skill mix review and recruitment 1. Environment and opportunity
2. Orientation and training 2. Endoscopy trainers
3. Assessment and appraisal 3. Assessment and appraisal
4. Staff are cared for 4. Equipment and materials
5. Staff are listened to

11. GRS achievement levels
• Units score themselves on several items for
each standard
• This gives a level attained and, thus, monitors
progress
• Levels D through A
• These levels can later be used for
accreditation purposes

12. Audit Cycle

13. National Endoscopy Quality
Improvement Programme
• Total service improvement
• Approach is that of the audit cycle
• Principal tool is the Global Rating Scale
• Standards
– Quality standards
– Auditable outcomes

14. Auditable Outcomes and Quality
Standards
An auditable outcome refers to an outcome, which is
important to monitor and review, for which it is not
possible to assign a standard. Examples of this might be
use of reversal agents for over sedation, minimum number
of procedures required to maintain competence,
or outcome of endoscopic therapy for variceal bleeding

15. Auditable Outcomes and Quality
Standards
A quality standard is an auditable outcome for which
there is an evidence base that can recommend a minimum
standard, for example completion rates for colonoscopy or
bleeding rates for sphincterotomy. As the evidence base
improves it is expected that it will be possible to convert
auditable outcomes into quality standards.

16. Auditable Outcomes and Quality
Standards
• Both require acquisition of large amounts of
data
• This must be accurate if important decisions
are to be made on it
• Clinical KPIs are part of this data
• Data entry should be as effortless as it is
accurate

17. Audit Cycle
• Standards – initially, for the Development Trial,
most are UK standards
• Reliable Data – various sources
• Schedule of audits - EUG
• Action plans - EUG

18. Clinical data as a KPI
• Choice of data
• Do not collect data because it can be collected
• Using data suggested by British Society of
Gastroenterology (BSG)

19. KPI’s as per BSG

20. Requirements of a clinical data
collection solution
• Accurate
• Require minimal extra effort over and above
reporting input
• Collected in a format that is usable
• Is connected to a system that can use it
• Exportable

21. Endoscopy Reporting in NZ
• Mixture
• Dictation
• Hand written
• Legacy standalone electronic reporting
systems
• In some locations more than one method on
the same site

22. Acquisition of ProVationtm MD
• Joint purchase after extensive tender process
by the three Auckland DHBs
• Auckland Installation 2010
• Christchurch installation 2011
• Other DHBs at various stages of engagement

23. ProvationtmMD
produces
terrific
endoscopy reports

24. How easy is it to get data into
ProVationtmMD?
• Menu structure is regarded as cumbersome by
clinicians
• Leads to more than desirable us of free text
fields
• Structure undeniably US biased with many
redundant fields for a NZ setting
• Changes possible but difficult

25. How easy is it to get data from
ProVationtm MD?
• All keystrokes are recorded
• Theoretically everything entered can be
extracted
• Database queries regarded as cumbersome by
clinicians
• Macros

26. Which BSG KPIs could be
extracted from ProvationtmMD

27. Initial appraisal of Provationtm MD
and clinical KPIs

28. Initial appraisal of Provationtm MD and
clinical KPIs
• Proof of concept investigation to assess
whether standard KPIs could be extracted
from ProVation reports. KPIs were chosen in
line with NHS, ASGE & NZ standards

29. Initial appraisal of Provationtm MD and
clinical KPIs
• Prospective audit across three Auckland DHBs
• Seven KPIs assessed
• Two six-week cycles with a planned preliminary analysis after week six
• Analysis performed using the data export function built into ProVation
(“automated”) and compared to results collated from individual reports
(“manual”).
• Feedback provided after the first round via department meetings or by
email Minor changes made after first round
– Index colonoscopy field added, bowel preparation field made mandatory
– Endoscopists asked to use post-surgical note for appropriate patients

30. EM Johns1, PD Frankish1, RS Walmsley1, DS Rowbotham2, RK Ogra3, DR Theobald1
Departments of Gastroenterology, Waitemata1, Auckland2 & Counties Manukau3 District Health Boards
Introduction Methods
Quality colonoscopy is integral to the success of bowel cancer screening programs. Key
Prospective audit across three Auckland DHBs
performance indicators (KPIs) are well established but require meticulous collection of a
Initial appraisal of Provationtm MD
Seven KPIs assessed
large volume of information. The endoscopy database program ProVation has recently been
Two six-week cycles with a planned preliminary analysis after week six
introduced across the Auckland region. Its potential role as a quality assurance tool was a
Analysis performed using the data export function built into ProVation (“automated”)
key reason for its implementation, and its future use as a nationwide audit tool is under
and compared to results collated from individual reports (“manual”).
consideration.
Feedback provided after the first round via department meetings or by email
Aim and clinical KPIs
Proof of concept investigation to assess whether standard KPIs could be extracted from
Minor changes made after first round
Index colonoscopy field added, bowel preparation field made mandatory
Endoscopists asked to use post-surgical note for appropriate patients
ProVation reports. KPIs were chosen in line with NHS, ASGE & NZ standards1-3.
Results Hospital X Hospital Y Hospital Z
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2
Automated Manual Automated Manual Automated Manual Automated Manual Automated Manual Automated Manual
Caecal intubation rate 208/226 199/209 225/243 216/229 245/264 231/243 268/282 253/263 200/239 202/230 226/257 228/245
92% 95% 93% 94% 93% 95% 95% 96% 84% 88% 88% 93%
Bowel preparation 81% 83% 94% 96% 64% 65% 99% 100% 80% 82% 99% 100%
quality documented
Withdrawal times no 53% 43% 60% 46% 35% 40% 37% 37% 40% 31% 28% 29%
manoeuvre#: proportion
>6minutes
Withdrawal times no 7:41 6:20 7:15 6:02 5:57 6:04 5:46 5:11 5:49 5:11 5:24 4:52
manoeuvre#: mean
Polyp detection rate age Insufficient data Insufficient data 39/86 42/95 Insufficient Insufficient data 37/103 32/101 Insufficient data Insufficient data 45/101 33/98
50-80 (index exams) 45% 44% data 37% 32% 45% 34%
Adenoma detection rate Not supported Insufficient data 26/86* 28/95 Not supported Insufficient data 24/103* 24/101 Not supported Insufficient data 26/101* 24/98
age 50-80 (index exams) 30% 29% 23% 24% 26% 24%
Polyp recovery rate Not supported 165/175 Not supported 190/215 Not supported 150/159 Not supported 159/183 Not supported 243/261 Not supported 191/221
94% 88% 94% 87% 93% 86%
Complications Not supported None Not supported 2 readmissions Not supported None Not supported 2 perforations Not supported 3 perforations Not supported 2 readmissions
1 readmission 1 readmission
*histology manually retrieved
#no polypectomy, biopsy etc

31. Initial appraisal of Provationtm MD
and clinical KPIs
KPI Problem Suggested solution
Caecal Intubation rate Difficulty identifying intact New mandatory field for
colons post surgical patients
Bowel preparation quality Variably reported when not Made mandatory
mandatory
No standardization Need agreement on
definitions of prep quality

32. Initial appraisal of Provationtm MD
and clinical KPIs
KPI Problem Suggested solution
Withdrawal times if no Times overestimated if Should improve with
manouevre manouevres not familiarity
documented
Polyp and adenoma Easily retrieved if Link to a pathology program
detection rates polypectomy documented (currently not possible)
Histology not linked Manual linkage

33. Initial appraisal of Provationtm MD
and clinical KPIs
KPI Problem Suggested solution
Polyp recovery rate Provation records Requires software
qualitatively and not modification
quantitavely
Complications Manual entry by person Needs dedicated audit
with admin privileges personnel

34. Does ProVationtm MD
capture colonoscopy KPIs?
Well, sort of -ish

35. Can ProVationtm MD
capture colonoscopy KPIs?
Well, yes

36. Where are we now?
Suggested solution Suggested solution Suggested solution
New mandatory field for Should improve with Requires software
post surgical patients familiarity modification
Made mandatory Link to a pathology program Needs dedicated audit
(currently not possible) personnel
Need agreement on
definitions of prep quality

37. Where are we now?
Suggested solution Suggested solution Suggested solution
New mandatory field for
post surgical patients
Made mandatory
These are done and were easy requiring just
a software tweak within the existing program.
This was done by clinicians

38. Where are we now?
Suggested solution Suggested solution Suggested solution
Should improve with
familiarity
Needs dedicated audit
personnel
Need agreement on
definitions of prep quality These are partly done or partly not done
and require behavioural change from clinicians

39. Where are we now?
Suggested solution Suggested solution Suggested solution
Requires software
modification
Link to a pathology program
(currently not possible)
These are not done and require major software
work and behavoural change from clinicians

40. The way forward
• Progress must be clinician led
• Clinicians must collaborate across the entire
country to formulate an agreed set of clinical
requirements
• This requires strong clinical leadership
• A NZ ProVation Users Group is required for this
• So far abortive attempts to establish this

41. The way forward
• The NZ ProVation Users Group then
approaches company for software updates
(NZ cf US market share)
• The NZ ProVation Users Group needs to have
ongoing close ties with the National
Endoscopy Quality Improvement Programme

42. The way forward
• Should all DHBs purchase ProvationtmMD?
• IT Board of the NHB supports a single software
solution without naming a specific product
• The clinical information required for KPIs is
not very complex……
• …..but it does need to be collected with
something more reliable than a pencil and a
piece of paper

43. Does ProVationtm MD
capture colonoscopy KPIs?
Well, sort of - ish

44. Can ProVationtm MD
capture colonoscopy KPIs?
Definitley