CHE Seminar 6 July 2017, Speaker: Rachel Meacock, Research Fellow in Health Economics, University of Manchester,

1. METHODS FOR THE ECONOMIC EVALUATION
OF CHANGES TO THE ORGANISATION AND
DELIVERY OF HEALTH SERVICES
Rachel Meacock
Centre for Health Economics, University of York
6th July 2017

2. Background
• Established methods exist for the economic evaluation of
new health technologies seeking NHS funding in England
• Treatments go through mandatory NICE appraisal process
• Changes to the organisation and delivery of health services,
including policy changes, (service interventions) are funded
from the same NHS budget, but not covered by this process
• Often rolled out without supportive evidence or evaluation

3. Background
• Inconsistent approaches - differing levels of scrutiny likely to
result in allocative inefficiency in the health system
• Resulted in a lack of methodological development and cost-
effectiveness evidence
• Whilst principle of assessing cost-effectiveness should apply
to all NHS spending, methods will need adapting in places to
enable evaluation of service interventions

4. Aim
To contribute to the development of methods for the economic
evaluation of service interventions
1. Method to quantify effects of service interventions in terms
of QALYs in absence of primary data collection on HRQoL
2. Demonstrate how survival analysis can be used to improve
treatment effect estimates associated with service
interventions (length of life component of QALY)

5. Part 1:
Quantifying the effects of service interventions in terms of
QALYs in the absence of primary data collection on HRQoL

6. Introduction
• Often estimate the impact of service changes on mortality
• Useful indicator of the impact of a programme, but tells us
nothing about the intervention’s value
• To assess cost-effectiveness, we need to estimate the
impact of a programme in terms of QALYs
• Problem = usually evaluate using administrative data which
does not contain information on HRQoL

7. Proposed method
• A QALY ‘tariff’ applied to mortality outcomes to estimate the
QALY gains associated with detected mortality reductions
• Discounted and quality-adjusted life expectancy (DANQALE)
tariff
• Stratified by single year of age (18 - 100) and sex
• Represents the average stream of remaining QALYs for an
individual i in each age-sex group a from general population

8. DANQALE
Two components of the QALY:
• Length of life component: Sex-specific life expectancy
estimates at each single year of age taken from 2008-10
ONS interim life tables
• QoL adjustment: Age-sex specific mean EQ-5D values from
2006 wave of the Health Survey for England

9. We calculate the DANQALE (𝑄𝑖𝑎) for each individual i in each
age-sex group a as:
𝑄𝑖𝑎 = 1 − 𝑚𝑖
𝑘=𝑎
𝐿 𝑎
𝑞 𝑘 1 + 𝑟 − 𝑘−𝑎
• 𝑚𝑖 equals 1 if the individual dies within 30days and 0
otherwise
• k indexes ages from a to the life expectancy of an individual
currently aged a(𝐿 𝑎)
• 𝑞 𝑘 is HRQoL at age k
• r is the discount rate (3.5%)

10. Analysis
• Attach DANQALE to each individual in the data
• Perform analysis as normal, but on DANQALE variable
• Can then compare to costs of the programme, either at the
individual or total programme level
Mortality Discounted QALYs
(DANQALE)
Total QALYs
AQ -0.9** [-1.4, -0.4] 0.07** [0.04, 0.11] 5,227

11. Limitations
• Likely to over-estimate health gains enjoyed by additional
survivors as assumes those surviving past 30days experience:
– Average life expectancy of the general population
– Average QoL of the general population
BUT
• Only captures QALYs gained due to mortality reductions i.e.
deaths averted
• Does not capture pure QoL improvements

12. Potential extensions
• Might want to update QoL values – later waves of HSE, other
data sources (e.g. SF-6D from Understanding Society)
• Could use condition-specific QoL estimates from audits if
available
Reference
Meacock R, Kristensen S, Sutton M. (2014). The cost-
effectiveness of using financial incentives to improve provider
quality: a framework and application. Health Economics, 23, 1-
13.

13. Part 2:
Using survival analysis to improve estimates of life year
gains in policy evaluations

14. Introduction
• Focus on methodology for estimating the impact of service
interventions on length of life
• Length of life is a key outcome in cost-effectiveness analysis:
– Cost per life year gained
– Cost per QALY
• Evaluations attempting to take a lifetime horizon can use
admin data to estimate changes in short-term mortality
• Convert these to projected life year gains using published
estimates of life expectancy from the general population

15. Previous approach
• Estimate the impact of a programme in terms of 30 day
mortality (binary outcome)
• Estimated reductions in this mortality are then translated into
life years gained
– Patients dying within 30 days are attributed no survival days
(effectively assumed to die instantly)
– Patients surviving past 30 days are assigned the remaining
age/gender-specific life expectancy of the general population

16. Limitations to previous approach
• Length of life of patients affected by interventions is likely to
differ from the general population – may lead to incorrect
estimations of the impact on life years gained
• True impact of policies on survival may be more complex,
potentially impacting survival over the whole life course
• Such longer-term effects not captured by evaluations
focusing solely on mortality rates within e.g. 30 days

17. Proposed solution
• Even with minimal data of 1 financial year available in many
administrative data sets, possible to observe most patients
for longer than 30 days
• Prolonged follow-up often ignored in policy evaluations
• Survival analysis is commonly used in clinical trials to
extrapolate gains in life expectancy from observed trial data
• Utilises all available follow-up information on patients rather
than applying an arbitrary cut-off window

18. Aim
• Examine whether the additional information available within
admin data sets on survival beyond usual 30 days
considered, albeit censored, can be used to improve
accuracy of estimated life year gains
• Demonstrate the feasibility and materiality of using
parametric survival models commonly employed in clinical
trials analysis to extrapolate future survival in policy evals

19. Motivating example
• Previous CEA of Advancing Quality (AQ) P4P programme
• Examine pneumonia patients only
• Consider a typical situation – data on dates of admission and
death are available for 1 financial year pre and post AQ
• Parametric survival models to estimate the effect of AQ on
survival over lifetime horizon
• Compare to results obtained using previous method

20. Data
• Hospital Episode Statistics linked to ONS death records
• Patients admitted to hospital in England for pneumonia:
– 2007/08 (pre-AQ)
– 2009/10 (post-AQ)
• Data period: 1st April 2007 – 31st March 2011
• Risk-adjustment: primary & secondary diagnoses (ICD-10),
age, gender, financial quarter of admission, provider,
admitted from own home vs institution, emergency vs
transfer

21. Methods
1. Comparison of methods on a development cohort
• Cohort of patients admitted to any hospital in England prior
to introduction of AQ (2007/08)
• Compare 3 methods for estimating remaining life years using
data from 2007/08 only
• Compare to observed survival of this cohort now available up
to 31st March 2011

22. Purpose of development cohort
• Illustrate difference in magnitude of estimated remaining life
years of a patient population when:
– Additional information available on survival past 30 days is utilised
– Risk of death is taken from the population under investigation rather
than general population figures
• Exercise also used to select the most appropriate functional
form for the survival models later used to evaluate AQ

23. Method i
Simple application of published life expectancy tariffs
• Simplified version of DANQALE applied in original evaluation
(does not incorporate discounting or QoL)
• 30 day mortality assessed as a binary outcome
• Gender-specific general population life expectancy estimates
at each single year of age (18 – 100) attached to patients
surviving beyond 30 days to estimate remaining life
expectancy

24. Method i
Remaining life expectancy:
𝐿𝑖
𝑔𝑎
= 𝑠𝑖
30
∙ 𝐿 𝑔𝑎
𝐿 𝑔𝑎 is the life expectancy of an individual of gender g who is
currently aged a
𝑠𝑖
30
equals 1 if individual i survives more than 30 days and 0
otherwise

25. Method i
Implicitly assumes that individuals surviving beyond 30 days
after admission survive, on average, the life expectancy of the
general population
Will produce an inaccurate estimate of the actual life
expectancy:
a) Period of survival within 30 days is not incorporated
b) Assumes life expectancy of individuals surviving beyond 30 days
after admission will be equal to that of the general population of their
age and gender
Ignores information on observed survival available in data

26. Method ii
Short-term observed survival plus application of published life
expectancy estimates
• Extend method i to utilise all information on mortality available
within year of data (2007/08)
• Can follow patients for between 1 – 365 days depending on
admission date
• For those that died during the period, number of days survived
between admission and death are counted
• Life expectancy again applied to those surviving past the end of
the financial year

27. Method ii
Improves on method i by:
• Eliminating problem a) period of survival within 30 days is not
incorporated
• Reducing, but not eliminating, inaccuracies due to problem
b) assuming life expectancy of those surviving beyond 30
days after admission will be equal to that of the general
population of their age and gender

28. Method iii
Extrapolation using survival models
• Improve on method used for extrapolation by estimating
parametric survival models on the observed year of data
• Predict lifetime survival based on mortality rates of the
population of interest
• Considered six standard parametric models (exponential,
Weibull, Gompertz, log-logistic, log-normal, generalised
gamma)

29. Method iii
Model fit assessed using:
• AIC
• Tests of whether restrictions on the parameters in the
generalised gamma model suggest it could be reduced to the
simpler models it nests
• Examination of residual plots
External validity of extrapolations assessed by comparing
proportion of the cohort predicted to be alive at annual intervals
to the observed survival now available to 31st March 2011

30. Method iii
• In our case, while standard parametric models were able to
fit the observed data well, the tails of these distributions did
not correctly represent the pattern of future mortality
• Hazard rates experienced by our patient cohort changed
over time – extremely high-risk period shortly after
emergency hospital admission not representative of lifetime
risk of those surviving past this period

31. Method iii
Solution = estimate survival in 2 separate models:
• Short-term survival during the first year estimated on the
observed 1 year of data
• Extrapolation of long-term survival (1 year + after admission)
based on a model estimated on data excluding first 30 days
following admission
Long-term models represent hazards experienced by our
patient cohort after the initial high-risk period following
emergency admission – still much higher than general
population but significantly lower than when first admitted

32. Allowed us to estimate the effect of covariates on survival in both the observed
and extrapolated period

33. Method iii
Improves on method i by:
• Again eliminating problem a) period of survival within 30
days is not incorporated
• Using information on mortality risk from the patient
population under investigation rather than general population
estimates
We compare results at each stage as assumptions are dropped
– illustrates materiality of these developments

34. Application
2. Application to the evaluation of AQ
• Stage 1 demonstrates the materiality of the difference
survival analysis makes to the estimated life years remaining
of our patient cohort
• Stage 2 illustrates how these models can be used in an
applied programme evaluation

35. Dichotomous difference-in-differences (DiD) design:
𝐿𝑖𝑗𝑡 = 𝑓(𝑎 + 𝑋′ 𝑏 + 𝑢𝑗 + 𝑣 𝑡 + 𝛿𝐷𝑗
1
∙ 𝐷𝑡
2
+ 𝜀𝑖𝑗𝑡)
• 𝐿𝑖𝑗𝑡 is the life expectancy of individual i treated in hospital j at
time t
• f(∙) is the link function
• X is the vector of case-mix covariates
• 𝑢𝑗 are provider fixed effects
• 𝑣 𝑡 are time fixed effects
• 𝐷𝑗
1
is a dummy = 1 for hospitals that become part of AQ
• 𝐷𝑡
2
is a dummy = 1 in the periods after the introduction of AQ
• 𝜀𝑖𝑗𝑡 is an individual-specific error terms
• 𝛿 is the DiD term, which is our coefficient of interest

36. Application
First, consider situation where data on admissions and deaths
are available for 1 financial year pre and post AQ
• Pre AQ (2007/08)
• Post AQ (2009/10)
BUT, survival analysis can utilise additional follow-up on the
pre-intervention group collected during same period as initial
follow-up of the post-intervention group
• Examine how life expectancy estimates are affected when including
additional follow-up available (2008/09 – 2009/10) on pre-AQ group –
should improve accuracy of estimates

37. Application
Use average partial effects to calculate the effect of AQ on life
expectancy
Estimate life expectancy for individuals admitted to AQ
hospitals in the post-AQ period under 2 scenarios:
– DiD term set = 0 (absence of AQ)
– DiD term set = 1 (presence of AQ)
Compare results to those obtained using methods i and ii
(linear regression on gen pop life expectancy estimates)

38. RESULTS
Part 1: Development cohort

39. Annual mortality rates for females
Age General
population
Patients admitted for pneumonia
2007/08
years % % (n deaths)
20 0.02 6.12 (98)
30 0.04 4.71 (191)
40 0.10 11.33 (309)
50 0.24 17.53 (291)
60 0.56 27.23 (584)
70 1.46 42.78 (783)
80 4.52 59.63 (1,469)
90 14.60 77.50 (1,142)
100 39.19 89.90 (109)
• Highlights importance of using information on the risk of death from the patient
cohort under investigation rather than general population figures when
estimating remaining life years
• Using gen pop figures would underestimate the annual mortality rate by a factor
of between 2 (age > 100 years) and over 300 (age 20)

40. Exponential Weibull Gompertz Log-
normal
Log-
logistic
Generalised
gamma
Internal
validity:
AIC 326,943 288,141 291,563 283,531 285,139 283,386
External
validity:
Time point Predicted survival, % Observed
survival, %
31/03/08 56.76 60.02 60.02 60.10 59.78 60.21 61.05
31/03/09 36.02 46.51 52.87 49.08 47.63 48.96 49.73
31/03/10 25.93 39.26 52.37 43.92 41.77 43.67 43.86
31/03/11 20.04 34.30 52.24 40.49 37.92 40.14 39.31
Internal and external validity of different parametric survival functions
• Lowest AIC
• Wald test confirmed does not reduce to the log-normal
• Best performance on external validity – predicted proportion of cohort alive to
within 1% of observed survival at each of 4 annual time points available

41. Method Assessment period Extrapolation
method
Those alive at end
of assessment
period, n (%)
Estimated life years
remaining, mean
i Admission to 30 days Gen pop life
expectancy
82,208 (72.56%) 13.15
ii Admission to end of
financial year
Gen pop life
expectancy
69,158 (61.05%) 11.98
iii Admission to end of
financial year
Parametric survival
models
69,158 (61.05%) 9.19
Comparison of estimates of remaining life years for patients admitted
for pneumonia 2007/08 (n = 113,289)
• Taking account of additional information on survival past 30 days reduced the
estimate of average remaining life years by 9% (method ii)
• Using survival models to extrapolate future survival reduced original estimate by
30% (method iii)

42. RESULTS
Part 2: Application to the evaluation of AQ

43. Patients admitted in 2007/08 Patients admitted in 2009/10
North West Rest of England North West Rest of England
n 17,993 95,296 19,946 106,365
Age at admission 71.7 72.2 71.9 72.8
Female, % 49.8% 48.7% 50.3% 49.1%
Comorbidities, n 1.79 1.65 1.99 1.92
Unadjusted
mortality within 30
days
28.4% 27.3% 25.6% 26.0%
Dead by end of
the financial year
40.7% 38.6% 37.3% 37.3%
Descriptive statistics for patients admitted for pneumonia, by region
and time period
• Pre-AQ the unadjusted mortality rate was higher in the North West within 30
days of admission and persisted in the longer-term to end of financial year
• Unadjusted mortality rates decreased in both regions, with a greater reduction in
the North West – positive effect of AQ on mortality previously detected

44. Estimated effect of AQ on the remaining life expectancy of
patients admitted to hospitals in the North West in 2009/10

45. Method i Method ii Method iii
Source of life expectancy
estimates
Gen pop life
tables
Gen pop
life tables
Survival analysis using 1 financial year of
follow-up
Short-term model:
Entry time =
admission
Long-term model:
Entry time = 31 days
post admission
Estimation method OLS OLS Generalised gamma Generalised gamma
DiD coefficient
(robust t stat)
0.154
(2.39)
0.221
(3.04)
0.103
(2.64)
0.089
(1.71)
Observations 239,600 239,600 239,600 156,860
Deaths, n 63,845 91,272 91,272 26,785
Life expectancy for patients
admitted in North West
2009/10
13.218 11.982 9.041
Counterfactual estimate, life
expectancy for patients in
North West in absence of AQ
13.064 11.761 8.730
Effect of AQ on life
expectancy for those
admitted in North West
0.154 0.221 0.311

46. Interpretation
• Lower absolute estimates of life expectancy both in the
presence and absence of AQ were expected from methods ii
and iii – additional deaths were observed and risk taken from
patient cohort under investigation
• Despite lower absolute estimates of life expectancy, estimate
of the effect of AQ increased – indicates that AQ impacted on
survival beyond 30 day post-admission window
• Generalised gamma parameterized in the AFT metric –
coefficients < 1 indicate time passes more slowly – failure
(death) expected to occur later as a result of AQ

47. Method iii
Source of life
expectancy estimates
Survival analysis using 1 financial
year of follow-up
Survival analysis using all available
follow-up ( to 31/03/10)
Short-term
model:
Entry time =
admission
Long-term model:
Entry time = 31
days post
admission
Short-term
model:
Entry time =
admission
Long-term model:
Entry time = 31
days post
admission
DiD coefficient
(robust t stat)
0.103
(2.64)
0.089
(1.71)
0.142
(3.62)
0.101
(2.26)
Observations 239,600 156,860 239,600 164,438
Deaths, n 91,272 26,785 110,747 45,290
Life expectancy for
patients admitted in
North West 2009/10
9.041 8.439
Counterfactual estimate,
life expectancy for
patients in North West in
absence of AQ
8.730 8.059
Effect of AQ on life
expectancy for those
admitted in North West
0.311 0.380

48. Interpretation
Utilising additional follow-up data available on pre-AQ group:
• Increased precision of estimates
• Slightly decreased estimated remaining life expectancy for
the cohort both in the presence and absence of AQ
• Further increased estimated treatment effect of AQ

49. Discussion
Demonstrated:
– Feasibility of using parametric survival models to extrapolate future
survival in policy evaluations
– Materiality of the impact this has on estimates of remaining life years of a
patient cohort and a policy treatment effect
Detected impact of AQ beyond 30 day window usually assessed
shows advantage of survival analysis – ability to capture effects of
policies over the whole life course
In pre- and post- evaluation design, survival models can be
developed on the pre-intervention population and predictive
performance evaluated against observed follow-up available during
post-intervention period – external validity

50. Future work
• For estimates of life years gained to be used in CEA, the
stream of remaining life years need to be adjusted for QoL
and discounted – quite simple extensions
• More sophisticated survival models
Reference
Meacock, Sutton, Kristensen, Harrison. (2017). Using survival
analysis to improve estimates of life year gains in policy
evaluations. Medical Decision Making, 37, 415-426.

51. Overall conclusions
• Development of methods and applications of economic
evaluation of service interventions has the potential to
improve allocative efficiency
• Still a LONG way to go, but (hopefully) offered some useful
approaches
• Both strands of health economics have made impressive
methodological progress in different aspects of evaluation –
could learn a lot from each other

52. THANK YOU
Contact details:
rachel.meacock@manchester.ac.uk
@RachelMeacock

55. Method i
Remaining life expectancy:
𝐿𝑖
𝑔𝑎
= 𝑠𝑖
30
∙ 𝐿 𝑔𝑎
𝐿 𝑔𝑎 is the life expectancy of an individual of gender g who is
currently aged a
𝑠𝑖
30
equals 1 if individual i survives more than 30 days and 0
otherwise

56. Method ii
𝐿𝑖
𝑔𝑎
= 𝑠𝑖
𝑡∗
∙ 𝐿 𝑔𝑎
+ (1 - 𝑠𝑖
𝑡∗
) ∙ (𝑡𝑖
ϯ
- 𝑡𝑖
0
)
Where 𝑠𝑖
𝑡∗
is a binary indicator equal to 1 if individual i survives
to the end of the observation period 𝑡∗
𝑡𝑖
ϯ
is the date of death for individuals who die before the end of
the observation period
𝑡𝑖
0
is the date of admission

57. Method iii

58. Following estimation of survival models, created additional rows
of data for each individual for each possible future year up to
age 100
Estimated the probability of surviving to that year, allowing for
the progression of time and increments in age – analogous to
estimating transition probabilities in a Markov model:
𝑚𝑖
𝑡
(𝑎𝑖0, 𝑥𝑖) =
𝑠 𝑖𝑡 (𝑎 𝑖𝑡,𝑥 𝑖)
𝑠 𝑖,𝑡−1 (𝑎 𝑖𝑡,𝑥 𝑖)
– 1
𝑚𝑖
𝑡
is the probability that individual i will die by time t, given that
they have survived to time t-1
𝑠𝑖𝑡 is the probability that individual i will survive to time t, given
the values of their covariates x and their age 𝑎𝑖 at the time of
their admission

59. Then calculated the individual’s life expectancy using the sum of
the probability of surviving to the end of the first year and the
survival rates for each subsequent year, to a max age of 100:
𝐿𝑖
= 1 − 𝑚𝑖
1
∙ 𝑡∗
− 𝑡𝑖
0
+
𝑗= 𝑎 𝑖0+1
𝐴
𝑠𝑖,𝑗−𝑎 𝑖0
𝑎𝑖𝑗, 𝑥𝑖 ∙ (1 −
1
𝑚𝑖
𝑗+1− 𝑎 𝑖0
)
𝐿𝑖 is the life expectancy of individual i
𝑚𝑖
1
is the probability that individual i will die by the end of the
first year
𝑡∗ - 𝑡𝑖
0
is the length of time between the individual’s admission
date and the end of the first year