1. © Nuffield Trust
Evaluation methods – where can
predictive risk models help?
Adam Steventon
Nuffield Trust
8 July 2013

2. © Nuffield Trust
The problem with observational studies
Eligible patients
All patients Intervention
patients
n 54,990 556
% aged 85+ 21.6 46.2
Prior emergency admissions 0.5 1.4
Number chronic conditions 1.0 1.1
Predictive risk score 22.3 33.6
Intervention
patients
Source: Steventon et al (2012)

3. © Nuffield Trust
Solutions, 1) before-after study

4. © Nuffield Trust
Solutions, 2) regression adjustment
Y = f(age, number of chronic conditions,
prior emergency admissions,
intervention status)

5. © Nuffield Trust
Eligible patients
Intervention
patients
Matched
controls
All patients Matched
controls
Intervention
patients
n 54,990 556 556
% aged 85+ 21.6 46.2 46.2
Prior emergency
admissions
0.5 1.4 1.4
Number chronic
conditions
1.0 1.3 1.1
Predictive risk score 22.3 33.5 33.6
Solutions, 3) Matched controls
Source: Steventon et al (2012)

6. © Nuffield Trust
How to select matched controls
Propensity score (Rosenbaum and Rubin 1983)
- Predictive risk of receiving the intervention
Prognostic score (Hansen 2008)
- Predictive risk of experiencing the outcome (e.g. emergency
hospitalisation), in the absence of the intervention
Genetic matching (Sekhon and Grieve 2012)

7. © Nuffield Trust
Advantages / disadvantages
Disadvantage – only allows for observed variables
But
Matching as ‘data pre-processing’ – reduces dependence of estimated
intervention effects on regression model specification
Intuitive?

8. © Nuffield Trust
Overcoming regression to the mean using a control
group
Start of intervention

9. © Nuffield Trust
Overcoming regression to the mean using a control
group
Start of intervention

10. © Nuffield Trust
Overcoming regression to the mean using a control
group
Start of intervention

11. © Nuffield Trust
Overcoming regression to the mean using a control
group
Start of intervention

12. © Nuffield Trust
Solutions, 4) regression discontinuity
Winningthenextelection
Fraction of votes awarded to Democrats in the previous election
Source: Lee and Lemieux (2009)

13. © Nuffield Trust
What is being done at the moment?
Telehealth studies in Pubmed, 2006-2012
Descriptive Before after
Dose
response
Controlled All
Number of studies 3 24 1 16 44
Median number of patients in
telemonitored group (range)
45
(40 to 851)
35
(7 to 17,025)
246
102
(19 to 1,767)*
45
(7 to 17,025)*
Endpoints
Mortality 2 - 1 3 6
Hospital use (or costs) 3 6 - 12 21
Clinical (e.g. HbA1c) - 17 - 4 21
Patient reported outcomes
(e.g. quality of life)
1 10 - 3 14
Source: Steventon, Krief and Grieve (work in progress)

14. © Nuffield Trust
References
Lee DS, Lemieux T. Regression discontinuity designs in economics. 2009.
Available from: http://www.nber.org/papers/w14723.pdf?new_window=1
Sekhon JS, Grieve RD. A matching method for improving covariate balance in
cost-effectiveness analyses. Health economics 2012;21:695–714.
Rosenbaum P, Rubin D. The central role of the propensity score in observational
studies for causal effects. Biometrika 1983;70:41–55.
Hansen BB. The prognostic analogue of the propensity score. Biometrika
2008;95:481–8.

15. © Nuffield Trust8 July 2013
www.nuffieldtrust.org.uk
Sign-up for our newsletter:
www.nuffieldtrust.org.uk/newsletter
Follow us on Twitter: http://twitter.com/NuffieldTrust
© Nuffield Trust
adam.steventon@nuffieldtrust.org.uk