Martin Utley, Director of the Clinical Operational Research Unit at University College London, reflects upon his involvement in the launch of specific tools to monitor care quality for paediatric cardiac surgery.

1. Engaging with data
Martin Utley, Director of the Clinical Operational Research Unit,
University College London
Chair: Chris Sherlaw-Johnson, Senior Research Analyst,
Nuffield Trust

2. Building tools with clinical
teams for use in monitoring
outcomes.
Martin Utley
UCL Clinical Operational Research Unit

3. I received royalties from UCL Business in relation to co-authorship of the
PRAiS software tool for monitoring 30-day survival following paediatric cardiac surgery.
Declaration

4. An insentient machine

5. ?
One can monitor outcomes to identify problems
One has absolute standards (some components to ¼ inch, others to a thousandth)
If there is a problem, the machine can be fixed.
To diagnose and fix problem, one has to stop production and open the box.
Stopping production comes at a cost, so one has to have a degree of
confidence that there is a genuine problem.
To what extent do these conditions apply to health care?
An insentient machine

6. Survival to 30 days following
paediatric cardiac surgery

7. Variable Life Adjusted Display (VLAD)
with Partial Risk Adjustment in Surgery (PRAiS)

8. Variable Life Adjusted Display (VLAD)
with Partial Risk Adjustment in Surgery (PRAiS)
Pagel C, Utley M, Crowe S, Witter T, Anderson D, Samson R, McLean A, Banks V, Tsang V, Brown K, Real time
monitoring of risk-adjusted paediatric cardiac surgery outcomes using variable life-adjusted display: implementation
in three UK centres, Heart (2013) 99:1445-50.
In use for local
monitoring of
outcomes at UK
surgical centres
since 2013.

9. In use for local
monitoring of
outcomes at UK
surgical centres
since 2013.
Dissemination to
wider clinical community
Clinical consultation / contribution / leadership
Development of risk
model
Design of data displays

10. In use for local
monitoring of
outcomes at UK
surgical centres
since 2013.
Dissemination to
wider clinical community
Clinicians led
engagement with surgical units,
National audit body & other stakeholders

11. All episodes in a 30% validation
sample within development set:
Mortality: 3.5%, N=7890
High risk diagnoses
Mortality: 8.8%, N=667
HLHS
Interrupted aortic arch
Pulmonary atresia (inc PA+IVS)
Comorbidity
Low risk diagnoses
Mortality: 1.4%, N=3196
Fallot/DORV-Fallot type
VSD
Pulmonary stenosis
Aortic arch obst +/- VSD/ASD
ASD
Subaortic stenosis (isolated)
Aortic regurgitation
Procedure
Medium risk diagnoses
Mortality: 4.2%, N=4027
PDA
Acquired
EMPTY/Unknown
Functionally uni-ventricular heart
TGA+VSD/DORV-TGA type
Pulmonary atresia+VSD (inc fallot type)
Tricuspid valve abnormality (inc Ebstein)
Mitral valve abnormality (inc supra, sub)
TAPVC
AVSD
Aortic valve stenosis
Misc Cong
TGA & IVS
Common arterial truncus
Normal
All episodes in a 30% validation
sample within development set:
Mortality: 3.5%, N=7890
High risk diagnoses
Mortality: 8.8%, N=667
HLHS
Interrupted aortic arch
Pulmonary atresia (inc PA+IVS)
Comorbidity
Low risk diagnoses
Mortality: 1.4%, N=3196
Fallot/DORV-Fallot type
VSD
Pulmonary stenosis
Aortic arch obst +/- VSD/ASD
ASD
Subaortic stenosis (isolated)
Aortic regurgitation
Procedure
Medium risk diagnoses
Mortality: 4.2%, N=4027
PDA
Acquired
EMPTY/Unknown
Functionally uni-ventricular heart
TGA+VSD/DORV-TGA type
Pulmonary atresia+VSD (inc fallot type)
Tricuspid valve abnormality (inc Ebstein)
Mitral valve abnormality (inc supra, sub)
TAPVC
AVSD
Aortic valve stenosis
Misc Cong
TGA & IVS
Common arterial truncus
Normal
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
Percentage episodes with empty co-morbidity fields
New ways of using diagnostic codes
clinically led / meaningful
then simplified for purposes of
risk model
Any old way of including comorbidity
knowingly included poor quality data
acknowledging clinical nous
commitment to long game
Development of risk
model Old ways of using operation codes
accepted existing schemes
impact on model “performance”
impact of model use
Year
% missing
comorbidity
data

12. PRAIS 1:
• 29 Specific procedures
• 3 diagnosis groups (low, medium, high)
• UVH
• Yes/no comorbidity
• Bypass/non-bypass
• Continuous weight
• Continuous age
• Age band (neonate/infant/child)
• Epoch effect
38 parameters
PRAIS 2:
• 16 spec proc groupings
• 11 diagnosis groupings
• UVH
• 4 “other illness” indicators
• Bypass/non-bypass
• Continuous nonlinear weight
• Continuous nonlinear age
• Epoch effect
36 parameters
PRAiS2 – developed on 21838 surgical episodes 2009-2014 from NCHDA data.
Recent update

13. VLADs do not give an evaluation of the
effectiveness / benefit of a surgical
programme to the patient population.
VLADs are not evidence of good or bad
performance.
DATA
patient case-mix
structure / resources
care processes
professionals
Queensland
pyramid model of
investigation
Worked with 3 sites on design of software
We improved ease of data entry / use, shared strengths and
limitations, listened to concerns.

14. Consultation on design of data displays
Added after consultation with units
Secondary outcomes we would have discarded
if not for discussions with clinical team

15. Weeks following release of PRAiS 2013

16. Christina Pagel - How can we better support appropriate interpretation
and use of risk adjusted survival data?
PRAiS 1 launch

17. What?
How?
Why?
PRAiS2
Research team worked with clinicians, family
representatives, press officers and journalists on
resources to support understanding and use of
risk-adjusted mortality data.

18. childrensheartsurgery.info

19. Launch
childrensheartsurgery.info

20. Launch
childrensheartsurgery.info

21. What?
How?
Why?
Funding NIHR HSDR PI Victor Tsang
Health Foundation PI MU
NICOR
Funding NIHR HSDR PI Christina Pagel
Funding NIHR HSDR PI Victor Tsang & Kate Brown
Funding GOSH Charity PI Kate Brown
None of the views expressed are those of DH, NIHR, Health Foundation, GOSH Charity etc.

23. 2014/15 - 4,207 surgical episodes with 97 deaths (mortality rate 2.3%) – 89 deaths predicted
Area under ROC curve 0.86 [0.82, 0.89]
PRAiS2 performance in test data

24. Brown KL, Crowe S, Franklin R, McLean A, Cunningham D, Barron D, Tsang V, Pagel C, Utley M, Trends in 30-day mortality
rate and case mix for paediatric cardiac surgery in the UK between 2000 and 2010, Open Heart. 2015 Feb 14;2(1):e000157.
doi: 10.1136/openhrt-2014-000157.
Absolute survival rates completely absent from debate in 2013
Kate Brown

25. PRAIS 2:
• 16 spec proc groupings
• 11 diagnosis groupings
• UVH
• 4 “other illness” indicators
• Bypass/non-bypass
• Continuous nonlinear weight
• Continuous nonlinear age
• Epoch effect
36 parameters
Performance assessed using cross-validation in main data set and prospective use in
2014/15 national data set.
PRAiS risk model recently updated
Close collaboration with an
expert panel including of
intensive care consultants,
surgeons, cardiologists, data
experts, analysts