Investigating the impact of the Quality and Outcomes Framework on the quality of primary care

1. UK pay-for-performance scheme
Methods
Results
Investigating the impact of the Quality and
Outcomes Framework on the quality of primary
care
Evan Kontopantelis Tim Doran* David Reeves
Jose Valderas Martin Roland
National Primary Care Research and Development Centre
University of Manchester England
RSS, 9th September 2009
Kontopantelis The QOF impact

2. UK pay-for-performance scheme
Methods
Results
Outline
1 UK pay-for-performance scheme
The idea
Framework details
2 Methods
The design
The General Practice Research Database
Analysis
3 Results
Short term QOF effect (2004/05)
Long term QOF effect (2004/05-2006/07)
Summary
Kontopantelis The QOF impact

3. UK pay-for-performance scheme
Methods
Results
The idea
Framework details
Improving quality of care.
A (very) juicy carrot...
A P4P program kicked off in April 2004 with the
introduction of a new GP contract.
General practices are rewarded for achieving a set of
quality targets for patients with chronic conditions.
The aim was to increase overall quality of care and to
reduce variation in quality between practices.
The incentive scheme for payment of GPs was named
Quality and Outcomes Framework (QOF).
QOF is reviewed at least every two years.
Kontopantelis The QOF impact

4. UK pay-for-performance scheme
Methods
Results
The idea
Framework details
Quality and Outcomes Framework.
Indicator details relate to Year 1.
Estimated cost of £1.8b, over 3 years (escalated to £2.8b).
GP income increased by up to 25%, dependent on
performance.
146 quality indicators.
Clinical care for 10 chronic diseases (76 indicators).
Organisation of care (56 indicators).
Additional services (10 indicators).
Patient experience (4 indicators).
Implemented simultaneously in all practices (a control
group was out of the question).
Kontopantelis The QOF impact

5. UK pay-for-performance scheme
Methods
Results
The idea
Framework details
Performance.
Incentivised aspects keep improving but what about the non-icentivised ones?
55
60
65
70
75
80
85
90
%
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
year
Coronary heart disease Asthma Diabetes
40, 42 and 42 practices respectively in total
Clinical performance
70
80
90
100
%
2004/05 2005/06 2006/07 2007/08
QOF year
Asthma(3) CHD(9) Mental Health(3) DM(15)
Epilepsy(3) Heart failure(2) Stroke(7) COPD(2)
based on data from all English practices
number of indicators used in the averages, displayed in brackets
Clinical domain averages
Overall performance of incentivised care
Quality scores for all QOF
clinical indicators have been
improving.
They are only a small
proportion of all clinical care.
There are concerns that
quality for non-incentivised
aspects of care may have
been neglected.
How to measure performance
on the non-incentivised
aspects though?
Kontopantelis The QOF impact

6. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
Clinical indicators.
..and research questions.
Two aspects to clinical indicators:
a disease condition (e.g. diabetes, CHD).
a care activity (e.g. influenza vaccination, BP control).
Condition
Incentivised
(QOF)
Non-
incentivised
Incentivised activity (QOF) A: 30 B: 9
Non-incentivised activity C: 7 D: 8
We aimed to compare the four classes on changes in
quality from pre-QOF to post-QOF.
Would class A indicators show most improvement?
Would classes B and C show some ‘halo’ effects since they
involve either a QOF condition or activity?
Has quality for class D indicators declined?
Kontopantelis The QOF impact

7. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
Enter GPRD...
Patient level data!
GPRD holds event data for more than 270 English
practices, from 1999 (488 active practices in 2007 and
around 10m patients).
Database is broken down to numerous tables, because of
the volume of data.
Final sample was 600,000 patients from 150 practices
(over 24GB in size).
Allows us to:
Investigate quality of care at the patient level.
Model and measure non incentivised aspects of care.
Kontopantelis The QOF impact

8. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
GPRD details.
Event files.
Clinical: all medical history data (symptoms, signs and
diagnoses).
Referral: information on patient referrals to external care
centres.
Immunisation: data on immunisation records.
Therapy: data relating to all prescriptions issued by a GP.
Test: data on test records.
Lookup files.
Medical codes: READ codes, 111,865 available.
Product codes: 77,198 available.
Test codes: 304 available.
Kontopantelis The QOF impact

9. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
The process.
Step 1: searching in the dark...
Size of the tables prohibits looking at codes one by one.
Instead we use search terms to identify potentially relevant
codes in the lookup tables and create draft lists.
Example (Search terms for diabetes)
String search in Medical codes: ’diab’ ’mell’ ’iddm’ ’niddm’.
READ code search in Medical codes file: ’C10’ ’XaFsp’.
String search in Product codes file: ’insulin’ ’sulphonylurea’
’chlorpropamide’ ’glibenclamide’.
Kontopantelis The QOF impact

10. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
The process.
Step 2: clinicians brawl over the codes (statisticians place bets).
Clinicians go through the draft lists and select the relevant
codes.
Three sets of codes are created, that correspond to:
QOF criteria.
Conservative criteria.
Speculative criteria.
Using the finalised code lists we search for events in the
Clinical, Referral, Immunisation, Therapy and Test files.
Kontopantelis The QOF impact

11. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
The process.
Step 3: extracting the events of interest, for each indicator.
For each indicator and year, we used the associated
clinical rules and retrieved events to construct a % score:
(cases where indicator was met / eligible patients)*100%.
Example (C4 indicator, QOF year 06/07: 01Apr06-31Mar07)
Percentage of patients with hypertension who have a
record of blood glucose once since diagnosis.
Denominator: Patients registered for the full year with the
practice, diagnosed with hypertension at any point until the
31st of March 2007 (excluding resolved cases).
Numerator: Patients in the denominator who have had at
least one record of blood glucose, postdating the first
(unresolved) hypertension diagnosis.
Kontopantelis The QOF impact

12. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
Issues to tackle.
Is the analysis ever easy...?
Indicator classes are imbalanced.
Three different types of activities:
clinical processes related to measurement (PM/R).
clinical processes related to treatment (PT).
intermediate outcome measures (I).
Condition
Incentivised (QOF) Non-incentivised
PM/R PT I PM/R PT I
Inc. activity (QOF) 18 8 4 8 1 0
Non-inc. activity 4 3 0 2 6 0
Quality of care was already improving (prior to QOF).
The ceiling has been reached for certain ‘easy’ indicators.
Kontopantelis The QOF impact

13. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
The approach.
Interrupted Time Series.
ITS multi-level multivariate regression analyses, allowed us
to compare the four indicator classes on the basis of two
outcome measures:
20
40
60
80
100
Performance(%)
00−01
01−02
02−03
03−04
04−05
05−06
06−07
QOF years
Observations Regression line, pre−QOF
Regression line, post−QOF pre−QOF line extended
level change
The level difference
between the estimated pre-
and post-QOF trends, in
2004/05.
The change in slope from
the pre- to the post-QOF
trend.
Kontopantelis The QOF impact

14. UK pay-for-performance scheme
Methods
Results
The design
The General Practice Research Database
Analysis
The approach.
Some details.
The main analysis used logit-transformed scores.
Regression models run with xtmixed in STATA (mle).
Practices were treated as crossed with indicators and
random intercepts were used.
Created variable combining class & type into 9 ind groups.
Analysed 3 models, for each of the outcome measures:
Uncontrolled
Controlled pre-QOF slope & mean of each indicator - RE*.
Controlled pre-QOF slope & mean of each indicator - FE.
Using the estimated means for the 9 indicator groups we
compared across classes on: (i) PM/R types only
(ii) PT types only (iii) PM/R & PT types combined.
Kontopantelis The QOF impact

15. UK pay-for-performance scheme
Methods
Results
Short term QOF effect (2004/05)
Long term QOF effect (2004/05-2006/07)
Summary
Still...
Kontopantelis The QOF impact

16. UK pay-for-performance scheme
Methods
Results
Short term QOF effect (2004/05)
Long term QOF effect (2004/05-2006/07)
Summary
Change in level immediately following QOF.
Compared to expectations.
Level change - logit, controlled, RE
-1.37
1.23
-0.83
-0.54
0.35
0.68
-0.30
0.24
0.13
0.45
-0.74
0.32
-0.33
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
A-I
A-PM/R
B-PM/R
C-PM/R
D-PM/R
A-PT
B-PT
C-PT
D-PT
A-P
B-P
C-P
D-P
Indicator groupLevel change - logit, uncontrolled
-0.14
0.49
0.03
-0.02
0.41
-0.10
-0.04
0.65
-0.06
-0.05
-0.06
0.89
-0.08
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
A-I
A-P
M/R
B-P
M/R
C-P
M/R
D-P
M/R
A-P
T
B-P
T
C-P
T
D-P
T
A-P
B-P
C-P
D-P
Indicator group
Type PM/R indicators of class
A significantly increased in
level above expectation
post-QOF.
Type PM/R indicators of
classes B & D significantly
decreased in level below
expectation post-QOF.
Similar results for classes A &
D (but not B) using the
combined process type
indicators.
Kontopantelis The QOF impact

17. UK pay-for-performance scheme
Methods
Results
Short term QOF effect (2004/05)
Long term QOF effect (2004/05-2006/07)
Summary
Change in level immediately following QOF.
Comparing ordered means across classes, using the Newman-Keuls procedure.
A-I 0.32
C-PM/R B-PM/R D-PM/R
A-PM/R 1.23 * *** ***
C-PM/R 0.45 *** ***
B-PM/R -0.74 ns
D-PM/R -1.37
A-PT D-PT B-PT
C-PT 0.24 ns ns ns
A-PT 0.13 ns ns
D-PT -0.30 ns
B-PT -0.33
C-P B-P D-P
A-P 0.68 ns *** ***
C-P 0.35 * ***
B-P -0.54 ns
D-P -0.83
Ordered Means For PM/R indicators all four
classes were significantly
different from each other in
mean level change, bar B & D.
For PT indicators there were
no significant differences.
For combined P indicators,
pairs A & C and B & D were
not significantly different, but
A & C had significantly greater
level changes than B & D.
Kontopantelis The QOF impact

18. UK pay-for-performance scheme
Methods
Results
Short term QOF effect (2004/05)
Long term QOF effect (2004/05-2006/07)
Summary
Change in slope from pre- to post-QOF.
Change in slope - logit, controlled, RE
-0.21
-0.12
-0.23
0.06
-0.18
-0.17
-0.21
-0.18
-0.05
-0.22
-0.25
-0.17
-0.29
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
A-I
A-PM/R
B-PM/R
C-PM/R
D-PM/R
A-PT
B-PT
C-PT
D-PT
A-P
B-P
C-P
D-P
Indicator group
Change in slope - logit, uncontrolled
0.03
-0.32
0.00
-0.10
-0.16
-0.21
-0.02
-0.03
-0.09
-0.29
-0.12
-0.09
-0.08
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
A-I
A-P
M/R
B-P
M/R
C-P
M/R
D-P
M/R
A-P
T
B-P
T
C-P
T
D-P
T
A-P
B-P
C-P
D-P
Indicator group
Slope declined significantly:
for the A-I group.
for all classes of PM/R
indicators, bar class C.
for all classes of PT
indicators.
for all classes of the
combined P indicators, bar
class C.
Kontopantelis The QOF impact

19. UK pay-for-performance scheme
Methods
Results
Short term QOF effect (2004/05)
Long term QOF effect (2004/05-2006/07)
Summary
Change in slope from pre- to post-QOF.
Comparing ordered means across classes, using the Newman-Keuls procedure.
A-I -0.12
A-PM/R B-PM/R D-PM/R
C-PM/R 0.06 *** *** ***
A-PM/R -0.17 ns
B-PM/R -0.23
D-PM/R -0.29
A-PT D-PT B-PT
C-PT -0.17 ns
A-PT -0.18
D-PT -0.21
B-PT -0.21
A-P B-P D-P
C-P -0.05 ** ** ***
A-P -0.18 ns
B-P -0.22
D-P -0.25
Ordered Means For the PM/R, only class C
differed significantly from the
other classes (in mean slope
change).
For the PT indicators there
were no significant
differences.
For the combined P
indicators, only class C
emerged as significantly
different.
Kontopantelis The QOF impact

20. UK pay-for-performance scheme
Methods
Results
Short term QOF effect (2004/05)
Long term QOF effect (2004/05-2006/07)
Summary
Conclusions.
...results are still being interpreted!
Short term:
Only the PM/R indicators within class A showed greater
than expected improvement.
PM/R indicators in classes B and D demonstrated
significantly lower than expected gains.
Class C (QOF-incentivised patient group) PM/R indicators
significantly out-performed classes B and D.
Long term:
All types and classes, with the exception of C-PM/R,
experienced a significant drop in the rate of improvement.
QOF had an immediate ’halo’ effect on non-QOF activities
for patient groups targeted by QOF (class C).
Kontopantelis The QOF impact

21. Thank you Not too much information, I hope!
Comments and questions:
e.kontopantelis@manchester.ac.uk
Kontopantelis The QOF impact