1. Introduction
Methodology
Simulations
Application
Conclusion
Robust Methods for Health-related
Quality-of-life Assessment
Ian McCarthy
Baylor Scott & White Health
Center for Clinical Effectiveness
Utah Health Services Research Conference
April 30, 2014
Robust Methods for Health-related Quality-of-life Assessment

2. Introduction
Methodology
Simulations
Application
Conclusion
Background
Cost- and comparative-effectiveness studies becoming
increasingly important
Require assessment of health-related quality-of-life (HRQoL)
outcomes and quality-adjusted life-years (QALYs)
Common approach first collapses the multi-dimensional
HRQoL profile into a one-dimensional QALY (Drummond
et al., 2005; Brazier et al., 2002; Brazier & Ratcliffe, 2007)
EQ-5D
SF-6D
HUI
Robust Methods for Health-related Quality-of-life Assessment

3. Introduction
Methodology
Simulations
Application
Conclusion
Problem
Loss of information when reducing HRQoL profile into QALY, with
potentially biased and inconsistent marginal effects estimates
(Mortimer & Segal, 2008; Devlin et al., 2010; Parkin et al., 2010;
Gutacker et al., 2012):
1 Floor and ceiling effects not present in the underlying domains
but imposed by the scoring algorithm.
2 Nonlinearities in the relationship between the outcome and
independent variables which are difficult to approximate using
the summary score.
Robust Methods for Health-related Quality-of-life Assessment

4. Introduction
Methodology
Simulations
Application
Conclusion
Current Study
1 Monte Carlo study showing the bias of the estimated
coefficients when relying solely on QALYs or some other
summary score based on several ordered outcome variables.
2 Propose new two-step methodology that first estimates
coefficients in each HRQoL domain and then transforms the
coefficients and marginal effects into the QALY domain based
on predicted values from the first-stage regressions.
Robust Methods for Health-related Quality-of-life Assessment

5. Introduction
Methodology
Simulations
Application
Conclusion
Estimating QALYs
Marginal Effects: Standard Approach
Marginal Effects: Proposed Methodology
The SF-6D
Developed by John Brazier and other, the SF-6D is formed from a
subset of questions from the SF-36 or SF-12 and is a common
HRQoL outcome intended to provide a general measure of a
patient’s health status (Brazier et al., 2002; Brazier & Ratcliffe,
2007).
Six dimensions/domains of health: (Physical functioning, role
limitations, social functioning, pain, mental health, and
vitality)
Each domain characterized numerically with a range of
integers. Best value is 1, and worst value ranges from 4 to 6.
Scoring algorithm developed in Brazier et al. (2002) and
Brazier & Ratcliffe (2007) for calculating a population-based
index score from the SF-6D questionnaire
Robust Methods for Health-related Quality-of-life Assessment

6. Introduction
Methodology
Simulations
Application
Conclusion
Estimating QALYs
Marginal Effects: Standard Approach
Marginal Effects: Proposed Methodology
Scoring the SF-6D
Physical Functioning (PF)
PF=2 or PF=3 -0.035
PF=4 -0.044
PF=5 -0.056
PF=6 -0.117
Role Limitations (RL)
RL=2 or RL=3 or RL=4 -0.053
Social Functioning (SF)
SF=2 -0.057
SF=3 -0.059
SF=4 -0.072
SF=5 -0.087
Pain (P)
P=2 or P=3 -0.042
P=4 -0.065
P=5 -0.102
P=6 -0.171
Mental Health (MH)
MH=2 or MH=3 -0.042
MH=4 -0.100
MH=5 -0.118
Vitality (V)
V=2 or V=3 or V=4 -0.071
V=5 -0.092
Combination of Domains
“Most Severe” -0.061
Robust Methods for Health-related Quality-of-life Assessment

7. Introduction
Methodology
Simulations
Application
Conclusion
Estimating QALYs
Marginal Effects: Standard Approach
Marginal Effects: Proposed Methodology
Focus on QALYs
By far the most common methodology for estimating
coefficients and ultimately marginal effects is to first reduce
the multi-dimensional health profile to a one-dimensional
QALY (Austin et al., 2000; Austin, 2002; Richardson &
Manca, 2004; Manca et al., 2005; Basu & Manca, 2012)
Recent literature on how best to accommodate distributional
features somewhat specific to QALYs (Austin, 2002; Basu &
Manca, 2012), including a censored least absolute deviation
model and a Beta MLE approach
Robust Methods for Health-related Quality-of-life Assessment

8. Introduction
Methodology
Simulations
Application
Conclusion
Estimating QALYs
Marginal Effects: Standard Approach
Marginal Effects: Proposed Methodology
First Stage Regression
1 Estimate an ordered probit model separately for each domain,
d = 1, ..., 6, with the follow-up HRQoL response (yid,t1 )
modeled as a function of person-specific variables (xi ),
baseline HRQoL response (yid,t0 ), and treatment status (Ti ).
2 Form predicted probabilities of every possible response, j, in
each domain, d, denoted ˆpd
j .
The regression results provide a predicted (marginal) probability for
each of 31 possible outcomes for each person.
Robust Methods for Health-related Quality-of-life Assessment

9. Introduction
Methodology
Simulations
Application
Conclusion
Estimating QALYs
Marginal Effects: Standard Approach
Marginal Effects: Proposed Methodology
“Most Severe” Category
1 Defined as any one of the following (Brazier et al., 2002): 4
or more in the physical functioning, social functioning, mental
health, or vitality domains; 3 or more in the role limitation
domain; or 5 or more in the pain domain
2 Since the probabilities, Pd
ij , are potentially correlated across
domains, the probability of a “most severe” health status can
be calculated following the principle of inclusion and exclusion
for probability:
P (A1 ∪ A2 ∪ ... ∪ AN) = P (A1) + ... + P (AN) +
N
n=2
(−1)n+1
P (∩ n events) .
Robust Methods for Health-related Quality-of-life Assessment

10. Introduction
Methodology
Simulations
Application
Conclusion
Estimating QALYs
Marginal Effects: Standard Approach
Marginal Effects: Proposed Methodology
Estimate QALYs
QALY i = 1 − 0.035 × ˆPPF
i2 + ˆPPF
i3 − 0.044 × ˆPPF
i4 − 0.056 × ˆPPF
i5 − 0.117 × ˆPPF
i6
− 0.053 × ˆPRL
i2 + ˆPRL
i3 + ˆPRL
i4
− 0.057 × ˆPSF
i2 − 0.059 × ˆPSF
i3 − 0.072 × ˆPSF
i4 − 0.087 × ˆPSF
i5
− 0.042 × ˆPPain
i2 + ˆPPain
i3 − 0.065 × ˆPPain
i4 − 0.102 × ˆPPain
i5 − 0.171 × ˆPPain
i6
− 0.042 × ˆPMH
i2 + ˆPMH
i3 − 0.100 × ˆPMH
i4 − 0.118 × ˆPMH
i5
− 0.071 × ˆPV
i2 + ˆPV
i3 + ˆPV
i4 − 0.092 × ˆPV
i5
− 0.061 × ˆP (Most Severe) .
Robust Methods for Health-related Quality-of-life Assessment

11. Introduction
Methodology
Simulations
Application
Conclusion
Marginal Effects on QALYs
Treatment Effects with Selection
Data Generating Processes
The D × 1 vector of latent HRQoL values, y∗
i , is simulated as
follows:
y∗
i = γ + βxi + εi , where
ε ∼ N (0D×1, ID×D) ,
x ∼ U[0, 1],
γ = ID×1, and
β = 1.5 × ID×1.
Discrete HRQoL values are generated based on the value of the
latent value, y∗
id , relative to the Jd × 1 vector of threshold values in
each domain.
Robust Methods for Health-related Quality-of-life Assessment

12. Introduction
Methodology
Simulations
Application
Conclusion
Marginal Effects on QALYs
Treatment Effects with Selection
Simulated QALY Distributions
01020304050
Frequency
.4 .6 .8 1
SF-6D Index Score
010203040
Frequency
.2 .4 .6 .8 1
SF-6D Index Score
01020304050
Frequency
.4 .6 .8 1
SF-6D Index Score
020406080
Frequency
.3 .4 .5 .6 .7 .8
SF-6D Index Score
050100150200
Frequency
.4 .6 .8 1
SF-6D Index Score
Robust Methods for Health-related Quality-of-life Assessment

13. Introduction
Methodology
Simulations
Application
Conclusion
Marginal Effects on QALYs
Treatment Effects with Selection
Monte Carlo Results
Model Incremental Effect St. Dev. Mean % Bias Lower % Bias Upper % Bias RMSE
DGP 1:
True Effect 0.070 0.002
Two-stage Approach 0.070 0.003 -0.73% -11.85% 11.64% 0.0827
OLS 0.073 0.004 3.79% -8.89% 17.18% 0.0828
Beta MLE 0.077 0.004 9.49% -4.84% 25.44% 0.0830
Beta QMLE 0.075 0.004 6.27% -6.66% 19.96% 0.0829
DGP 2:
True Effect 0.093 0.003
Two-stage Approach 0.092 0.005 -0.64% -12.62% 11.48% 0.1041
OLS 0.089 0.005 -3.84% -15.36% 8.39% 0.1043
Beta MLE 0.142 0.010 52.57% 28.34% 76.59% 0.1115
Beta QMLE 0.102 0.006 10.14% -4.26% 25.24% 0.1043
DGP 3:
True Effect 0.076 0.003
Two-stage Approach 0.075 0.005 -1.34% -15.60% 15.21% 0.0916
OLS 0.065 0.004 -15.02% -29.91% -1.40% 0.0923
Beta MLE 0.075 0.008 -1.01% -23.44% 23.44% 0.0935
Beta QMLE 0.086 0.006 12.71% -5.97% 32.68% 0.0917
Robust Methods for Health-related Quality-of-life Assessment

14. Introduction
Methodology
Simulations
Application
Conclusion
Marginal Effects on QALYs
Treatment Effects with Selection
Monte Carlo Results
Model Incremental Effect St. Dev. Mean % Bias Lower % Bias Upper % Bias RMSE
DGP 4:
True Effect 0.075 0.002
Two-stage Approach 0.075 0.003 -0.22% -10.58% 11.14% 0.0966
OLS 0.083 0.004 10.32% -2.40% 24.52% 0.0968
Beta MLE 0.083 0.005 10.71% -2.67% 25.71% 0.0969
Beta QMLE 0.082 0.004 9.20% -3.23% 22.88% 0.0968
DGP 5:
True Effect 0.062 0.002
Two-stage Approach 0.061 0.003 -0.28% -11.20% 11.19% 0.0916
OLS 0.072 0.004 16.70% 2.21% 32.65% 0.0920
Beta MLE 0.070 0.004 13.03% -1.05% 28.53% 0.0919
Beta QMLE 0.070 0.004 13.46% -0.26% 28.56% 0.0919
Robust Methods for Health-related Quality-of-life Assessment

15. Introduction
Methodology
Simulations
Application
Conclusion
Marginal Effects on QALYs
Treatment Effects with Selection
General Case of Selection on Observable
Variables
-.0050.005.01.015
DeviationfromTrueEffect
0 10 20 30 40 50
Degree of Selection
-.01-.0050.005.01
DeviationfromTrueEffect
0 10 20 30 40 50
Degree of Selection
OLS 2SE
DGP 1: Homogeneous Effects across Domains
-.0050.005.01.015.02
DeviationfromTrueEffect
0 10 20 30 40 50
Degree of Selection
-.01-.0050.005.01
DeviationfromTrueEffect
0 10 20 30 40 50
Degree of Selection
OLS 2SE
DGP 2: Heterogeneous Effects across Domains
Robust Methods for Health-related Quality-of-life Assessment

16. Introduction
Methodology
Simulations
Application
Conclusion
Marginal Effects on QALYs
Treatment Effects with Selection
With Simulated Data based on the SF-6D
Random Treatment Assignment Selection on Observed Variables
Model Treatment Effect St. Dev. RMSE Treatment Effect St. Dev. RMSE
DGP 1: δ = 1.5 × I6×1
True Effect 0.142 0.005 0.142 0.005
2SE 0.143 0.006 0.054 0.143 0.007 0.054
OLS 0.143 0.007 0.066 0.151 0.010 0.068
Beta MLE 0.169 0.012 0.082 0.174 0.021 0.080
Beta QMLE 0.143 0.007 0.067 0.146 0.011 0.066
DGP 2: δ = 3 × I6×1
True Effect 0.264 0.007 0.264 0.007
2SE 0.264 0.007 0.046 0.263 0.009 0.046
OLS 0.265 0.008 0.077 0.284 0.010 0.091
Beta MLE 0.296 0.010 0.075 0.378 0.018 0.067
Beta QMLE 0.264 0.008 0.061 0.320 0.013 0.056
DGP 3: δ = [2, 1, 0.5, 2.5, 0, 1]
True Effect 0.104 0.004 0.104 0.004
2SE 0.104 0.005 0.055 0.104 0.007 0.055
OLS 0.104 0.006 0.063 0.088 0.009 0.064
Beta MLE 0.117 0.012 0.087 0.083 0.023 0.083
Beta QMLE 0.104 0.006 0.070 0.079 0.011 0.070
Robust Methods for Health-related Quality-of-life Assessment

17. Introduction
Methodology
Simulations
Application
Conclusion
Marginal Effects on QALYs
Treatment Effects with Selection
With Simulated Data based on the SF-6D
Random Treatment Assignment Selection on Observed Variables
Model Treatment Effect St. Dev. RMSE Treatment Effect St. Dev. RMSE
DGP 4: interaction terms with δ = 1.5 × I6×1
True Effect 0.122 0.006 0.122 0.006
2SE 0.122 0.007 0.048 0.122 0.010 0.048
OLS 0.122 0.008 0.084 0.137 0.014 0.094
Beta MLE 0.133 0.011 0.096 0.234 0.023 0.085
Beta QMLE 0.122 0.008 0.074 0.165 0.015 0.073
DGP 5: interaction terms with δ = 3 × I6×1
True Effect 0.220 0.007 0.220 0.007
2SE 0.220 0.007 0.043 0.220 0.010 0.043
OLS 0.220 0.008 0.096 0.266 0.014 0.132
Beta MLE 0.231 0.011 0.081 0.332 0.022 0.080
Beta QMLE 0.220 0.008 0.068 0.272 0.015 0.065
DGP 6: interaction terms with δ = [2, 1, 0.5, 2.5, 0, 1]
True Effect 0.102 0.005 0.102 0.005
2SE 0.102 0.006 0.047 0.102 0.009 0.047
OLS 0.102 0.007 0.078 0.098 0.013 0.079
Beta MLE 0.114 0.012 0.109 0.210 0.024 0.090
Beta QMLE 0.102 0.008 0.081 0.137 0.015 0.081
Robust Methods for Health-related Quality-of-life Assessment

18. Introduction
Methodology
Simulations
Application
Conclusion
Data Summary
Results
Data
Data collected prospectively on adult scoliosis patients from over
10 participating members of the International Spine Study Group
(ISSG).
Variable Mean Standard
Deviation
Age 56.76 14.51
BMI 26.59 5.84
Baseline SF-6D 0.61 0.12
Follow-up SF-6D 0.66 0.12
Count Percent
Operative 193 53%
Female 309 85%
Robust Methods for Health-related Quality-of-life Assessment

19. Introduction
Methodology
Simulations
Application
Conclusion
Data Summary
Results
Summary Statistics
Baseline Follow-up
Count Percent Count Percent
Physical Functioning Domain
PF=1 0 0% 0 0%
PF=2 35 10% 54 15%
PF=3 117 32% 121 33%
PF=4 96 27% 83 23%
PF=5 100 28% 95 26%
PF=6 14 4% 9 2%
Role Limitations Domain
RL=1 41 11% 53 15%
RL=2 115 32% 144 40%
RL=3 10 3% 11 3%
RL=4 196 54% 154 42%
Social Functioning Domain
SF=1 110 30% 156 43%
SF=2 72 20% 77 21%
SF=3 99 27% 86 24%
SF=4 56 15% 30 8%
SF=5 25 7% 13 4%
Robust Methods for Health-related Quality-of-life Assessment

20. Introduction
Methodology
Simulations
Application
Conclusion
Data Summary
Results
Summary Statistics
Baseline Follow-up
Count Percent Count Percent
Pain Domain
P=1 5 1% 19 5%
P=2 34 9% 47 13%
P=3 79 22% 123 34%
P=4 85 23% 88 24%
P=5 109 30% 66 18%
P=6 50 14% 19 5%
Mental Health Domain
MH=1 76 21% 130 36%
MH=2 127 35% 132 36%
MH=3 89 25% 61 17%
MH=4 53 15% 32 9%
MH=5 17 5% 7 2%
Vitality Domain
V=1 13 4% 15 4%
V=2 73 20% 123 34%
V=3 107 30% 108 30%
V=4 94 26% 74 20%
V=5 75 21% 42 12%
Robust Methods for Health-related Quality-of-life Assessment

21. Introduction
Methodology
Simulations
Application
Conclusion
Data Summary
Results
Average Treatment Effect of Surgery
OLS Beta Beta 2SE
MLE QMLE
Outcome: QALY QALY QALY PF RL SF P MH V
Surgery 0.03*** 0.17*** 0.15*** -0.06 -0.06 0.14 0.54*** 0.28** 0.26**
(0.01) (0.05) (0.05) (0.12) (0.12) (0.12) (0.12) (0.12) (0.12)
Age 0.00* 0.00 0.00* -0.00 -0.01 0.00 0.01** 0.00 0.00
(0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00) (0.00)
Female -0.02 -0.10 -0.09 -0.12 -0.27 0.07 -0.09 -0.47*** -0.31*
(0.01) (0.07) (0.07) (0.16) (0.17) (0.17) (0.16) (0.18) (0.17)
BMI -0.00 -0.00 -0.00 -0.00 -0.02** 0.01 -0.02 -0.00 0.00
(0.00) (0.00) (0.00) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01)
ATE on QALY 0.033*** 0.038*** 0.032*** 0.029***
(0.011) (0.011) (0.011) (0.010)
RMSE 0.098 0.111 0.098 0.097
Robust Methods for Health-related Quality-of-life Assessment

22. Introduction
Methodology
Simulations
Application
Conclusion
Intuition
Collapsing multi-dimensional profile into a single summary
measure introduces floor/ceiling effects and nonlinearities that
are difficult to accommodate in a single equation framework.
With selection into treatment (whether on observables or
unobservables), standard methods relying only on QALYs
provide biased estimates of true treatment effect.
An alternative approach is to estimate coefficients based on
the full health profile and then re-interpret effects in the
QALY domain based on predicted probabilities in the
first-stage regressions.
Robust Methods for Health-related Quality-of-life Assessment

23. Introduction
Methodology
Simulations
Application
Conclusion
Thank You
Robust Methods for Health-related Quality-of-life Assessment

24. Introduction
Methodology
Simulations
Application
Conclusion
Bibliography I
Austin, P.C. 2002. A comparison of methods for analyzing health-related quality-of-life measures. Value in Health,
5(4), 329–337.
Austin, P.C., Escobar, M., & Kopec, J.A. 2000. The use of the Tobit model for analyzing measures of health
status. Quality of Life Research, 9(8), 901–910.
Basu, A., & Manca, A. 2012. Regression Estimators for Generic Health-Related Quality of Life and
Quality-Adjusted Life Years. Medical Decision Making, 32(1), 56–69.
Brazier, J., & Ratcliffe, J. 2007. Measuring and valuing health benefits for economic evaluation. Oxford University
Press, USA.
Brazier, J., Roberts, J., & Deverill, M. 2002. The estimation of a preference-based measure of health from the
SF-36. Journal of health economics, 21(2), 271–292.
Devlin, N.J., Parkin, D., & Browne, J. 2010. Patient-reported outcome measures in the NHS: new methods for
analysing and reporting EQ-5D data. Health economics, 19(8), 886–905.
Drummond, M.F., Sculpher, M.J., & Torrance, G.W. 2005. Methods for the economic evaluation of health care
programmes. Oxford University Press, USA.
Gutacker, N., Bojke, C., Daidone, S., Devlin, N., & Street, A. 2012. Analysing Hospital Variation in Health
Outcome at the Level of EQ-5D Dimensions.
Manca, A., Hawkins, N., & Sculpher, M.J. 2005. Estimating mean QALYs in trial-based cost-effectiveness analysis:
the importance of controlling for baseline utility. Health economics, 14(5), 487–496.
Mortimer, D., & Segal, L. 2008. Comparing the incomparable? A systematic review of competing techniques for
converting descriptive measures of health status into QALY-weights. Medical decision making, 28(1), 66.
Parkin, D., Rice, N., & Devlin, N. 2010. Statistical analysis of EQ-5D profiles: does the use of value sets bias
inference? Medical Decision Making, 30(5), 556–565.
Richardson, G., & Manca, A. 2004. Calculation of quality adjusted life years in the published literature: a review of
methodology and transparency. Health economics, 13(12), 1203–1210.
Robust Methods for Health-related Quality-of-life Assessment