Databases of electronic medical records and in particular primary care databases (PCDs) are increasingly used in research. The largest PCDs contain full data on all primary care consultations by millions of patients over two or more decades. They provide a means for investigating important healthcare questions which cannot be practically addressed in a Randomised Controlled Trial. However, concerns remain about the validity of studies based on data from PCDs. Most work around validity has attempted to confirm individual data values within a dataset. We take a different approach and instead replicate published PCD studies in a second, independent, PCD. Agreement of results then implies that the conclusions drawn are independent of the data source (though this doesn’t rule out that such as confounding by indication are commonly influencing both). We replicated two previous PCD studies using the Clinical Practice Research Datalink (CPRD). The first was a retrospective cohort study of the effect of Beta-blocker therapy on survival in cancer patients using DIN-LINK. The second was a nested case-control analysis of the effects of Statins on mortality of patients with ischaemic heart disease using QRESEARCH. Our analyses produced several important quantitative differences compared to the original studies, altering conclusions. These could not be fully explained by either demographic differences in the patient samples or structural differences between the datasets. Our study highlights both the caution that needs to be applied when assessing the findings from analysis of just a single database and the difficulties in performing replications of existing PCD studies.

1. Using primary care databases to
evaluate drug benefits and harms:
are the results valid and
replicable?
David A. Springate, University of Manchester
Centres for Primary Care/Biostatistics

2. Outline
1. Primary Care Database (PCD) study validity
2. PCD Replications
– Statins and Ischaemic heart disease
– β-blockers and Cancer
3. Lessons to be learned

3. PCD studies are all the rage. . .
Number of UK PCD
publications is rapidly
increasing
1990 1995 2000 2005 2010
050100150
PCD articles in pubmed
year
Numberofarticles
There is global interest in UK PCD
research
Institutions affiliated with UK PCD publications
xx
x x
x
xxxxxx
x x x
x
xx
x
x
x
xx
x
xx
x
xxx xx
xxx
x
xx
xxxxxx
xx
xx
x
xx
x
x
x xx x
x
x
xx
x
x
xxxxxx
x
x
x
x
x
x
xx
x xxx
x
xxxxx
xxx
xxx
x
x
x
x
xx
xxx
xx
x xx
x
x
xxx
xx
x
x
x
x
x
x
x
x
x
x
xx xx
x
xxx
x x
x
x
x
x
x
x
xx
xx
x
xx
xxxx
x
x
x
x
xx
x
xx
x
xxx
x xx
xx
xx
x
x
xx
x
xxxxxx x
x
x
xx
x
xxx
x
x
xxx
x
x
x
x
xxxxx
x
xx
x
xx
xxxxxx
xx
xx
x
x
x
x
xx x
xxx
x
x
xx
xx
x xxxxx
x
xxxxx
x
xx
xxx
x
x
x
xx
xx
xxx
x
x
x
x
xx
x
xx
xx
x
x
x
x
xx
xx
xxxxxx x
x
x
xx
x
x
x
x
x
x
x
xx
x x
x
x
x
xx
x
x
x
xxx
x
xxxxx
x
x
x xxxx
x x
xxxxx
xx
xx
x
x
xxxxxxxx xxx
x
xxxxx
x
x xx xxx x
x
xx xxxx
x
x
xxx
xx
x
xx
xxxxx
x
xx
x
x
xx
x
x
x
xx
x
x
x
xxx
x
xx
x
xxxx
xx
xxx
xx
x
xx x
xx
x
xxxx
xxx
x
x
xxx
x
x xxxx
x
x
x
x
x
x
x
x
x
x
x
x
x
x xxxxxx xxxx xx
x
xxx
x
x
x
x
x x
x x
xx
x
x
x
x
x
xxx
x
x
x
xx
x
xxx
x
x
x
x
x
x
x
xx x
x
x
x
x
x
xx
xx
xxxx
x
x
x
x
xxx
x
x
xx
xxx
x
x xxx
x
x
x
xx
xxxxxxxxx
xx
xx
x
xxxx
xx
x
xxxx
x
x
xx
x
x
x
x xx
x
xxx
x
xx
xxxxxx xx
x
xx
x
x
x
xxx
x
x
x
xxxxx
xx
xx
x
x
x
x
x
x
x
x
x x
x
xxxxx
x
xx xxx
x
xxx
x
x
x
x
x
x
x
xx
x
x
x
x
x
xxx
xx
x
xxxxx
x x
xx
xx
x
x
x
xxxxxxx
xx
x
x
xxxx
xx xx
x
x x x
xxxx
xx
xx
xxx
xxx
x xx
xx
x
xxx
x
x
x
x
x
xxx
x
x
x
x
xxxx
x
x
x x
xxxx
xxx
xxxxxxxx x
xx
xx
xx
x
xxxx x
x
x
xxxx
x
x
x
xx
xxxx
xx
x xx
xxx
xxx
x
x
x xxx
xxxx xxxx
x
xx
x
x
x
x
xx
x
x x
x
xx
xxx
x
x
x
x
x
x
x
x
xxxxxxxx
x
xxx
xx
x
xxx
x
xx xxxx
xx
x
xxxxxxxx xxxxx
x
xx
xx
x
xxxxxxx
x
x
xx
xxx
x
x xx
x
xx
xx
x
x
xx
x
x
x
xxx
x x
x
xxx
x
x
xx
xx
xx
xxx
x
x
x
xx x
x
xxx x
x
x x
xx
x
x
xxx
x
xx
xxxxxxxx
x x
x
x
x
x
xx
x
xxxxxx
x
x
xxxx
xxx
x
xxx
x
x
x
x xx
x
x
xx
x
x
x
x
x
x
x
x
x
x x
x
x x
xx x
xx
xx x
x
x
xx
x
x
x
xxx
x
xxx
x
xx
xx
x
x
x
x
x
xx
xx
xx
x
x
xx
x
xxxxxxx
x
xxxxxxxxxxx xxxxxxxxxxx
xxx x
x
x
xxxx
x
xxxx
xxxxxxxxxxxxxxxxxx xxxxx

4. Uses of Primary care databases. . .
• Prevalence / incidence studies
• Associations between conditions
• Harms and risks of treatments
• Comparative effectiveness
• RCT comparisions / replications (and replacements?)

5. BUT, There are still concerns about the
validity of PCD-based studies. . .
Threat Refs
Data quality Herrett 2009,
Khan 2010, Jordan 2004
Data completeness Marston 2010, Delaney 2007,
Collins 2010
Confounding Tannen 2008, Lewis 2007
Clinical coding www.ClinicalCodes.org

6. PCD Replications
“Non-reproducible single
occurrences are of no
significance to science.”
—– Karl Popper (1959)
An approach to validity that asks
whether flaws and differences in
the data make any difference to
the ultimate conclusions rather
than looking at validity and
completeness of the underlying
individual data
http://xkcd.com/242

7. Replicating studies in another, independent
PCD
• Agreement implies that conclusions are not dependent
on data source
• Some factors could influence both (such as
confounding by indication)
• First completely independent PCD replication (but see
Vinogradova (BMJ 2013): relationship between
bisphosphate exposure and cancer in QResearch and
CPRD)

8. Replications were performed in CPRD
• Largest UK primary care database (CPRD-GOLD)
• ˜ 14 million patients
• ˜ 650 practices across the UK
• Uses the Vision GP computer system

9. Criteria for replication
1. Effectiveness studies
2. Different GP computer system from CPRD (not
Vision)
3. No practice overlap with CPRD
4. Representative coverage
5. Primary Care Database (Not integrated
primary/secondary/pharmacy)

10. PCD replications
Hippisley-Cox and Coupland (2006) Effect of statins on the
mortality of patients with ischaemic heart disease: population
based cohort study with nested case-control analysis. Heart
92:752-758 (QResearch)
Shah, Carey et al. (2011) Does β-adrenoceptor blocker
therapy improve cancer survival? Findings from a
population-based retrospective cohort study
Br J Clin Pharmacol 72:157-161 (DIN-LINK)
We then compare
• summary statistics
• mortality rates
• model coefficients and standard errors

11. Effect of statins on the mortality of patients
with ischaemic heart disease (QResearch)
Objective To measure the effect of statins on mortality for
community based patients with IHD
Design Cohort survival analysis and nested case-control
Setting 1.18 million patients in 89 practices
Subjects Patients with first diagnosis of IHD between
January 1996 and December 2003
Outcomes - Cohort: Adjusted hazard ratios (+/-
95%CI) for all-cause mortality
- Case-control: Odds ratio (+/-95%CI) for
current use of statins, previous use and
duration of use

12. Effect of statins on the mortality of patients
with ischaemic heart disease
Summary statistics
Measure Analysis CPRD Qresearch
Number of practices Cohort 661 89
Number of patients Cohort 91589 13029
Cases 15591 2266
Controls 62356 9064
Median age Cases 80 80
Controls 79 80
Percent female Cases 45.5% 44.3%
Controls 45.5% 44.3%
Median followup (months) Cases 22.1 20.3
Controls 22.5 21
Percent on statins Cases 17% 19.6%
Controls 23.6 25.4%

13. Mortality rates for patients with IHD
Age Comorbidity Sex
0
100
200
300
400
Total0−4445−5455−6465−7475−8485−94
95_plus
N
o
diabetes
D
iabetes
N
o
hypertension
H
ypertensionN
o
C
C
F
C
C
F
Fem
ale
M
ale
Patient group
Mortalityrate(per1000personyears)
PCD
CPRD
QResearch

14. Survival analyses for patients on Statins
QResearch
Adjusted HR 0.47 (0.41 to 0.53)
CPRD
0 2 4 6 8
Time since diagnosis of IHD (Years)
Survival
0.000.250.500.751.00
Adjusted HR 0.43 (0.40 to 0.46)

15. Case-control analysis: Odds ratios for effects
of Statins on mortality in IHD patients
Odds are relative to
patients not on statins.
Dotted line represents
1:1 odds
Previously on statins Currently on statins
0.0
0.5
1.0
1.5
2.0
AllStatins
Atorvastatin
C
erivastatinFluvastatinPravastatinSim
vastatin
AllStatins
Atorvastatin
C
erivastatinFluvastatinPravastatinSim
vastatin
Statin type
Oddsratio
PCD
CPRD
Qresearch

16. Misleading pooled odds ratios
Combining effects of e.g. two drugs of the same BNF
chapter
Group Y N Odds ratio
Drug 1 Cases 600 200
3/3 = 1
Drug 1 Controls 75 25
Drug 2 Cases 10 30
0.333/1 = 0.333
Drug 2 Controls 30 30
Pooled Cases 610 230
2.65/1.91 = 1.39
Pooled Controls 105 55

17. Case-control analysis: Adjusted OR for
duration of use of statins on survival
Odds are relative to
patients not on statins.
Dotted line represents
1:1 odds
0.00
0.25
0.50
0.75
1.00
0−12
13−24
25−36
37−48
49−60
>60
Duration (months)
AdjustedOddsratio PCD
CPRD
Qresearch

18. Summary — Statins study
1. Strikingly similar results in the two studies, despite
different GP computer systems (Vision vs EMIS)
2. As expected, narrower confidence intervals due to
larger study
3. Original study was well designed (Matching,
appropriate analyses etc.)
4. Given the results, pooling of “all statins” is
questionable

19. Does β-adrenoceptor blocker therapy improve
cancer survival?(DIN-LINK)
Objective To examine the effect of β-blocker treatment on
cancer survival
Design Survival analyses for 9 cancer types
Setting 3462 cancer patients on β-blocker or other
antihypertensive therapy
Subjects Patients 40-85 with first cancer diagnosis
between 1997 and 2006
Outcomes - Adjusted hazard ratios (+/- 95%CI) for
all-cause mortality in each cancer type
- Pooled hazard ratio (random effects)

20. Comparison of summary statistics
Cohort size:
CPRD 11316
DIN-LINK 3462
BPLM = Blood pressure
lowering medicines
Deaths Alive at 1 year Diagnosed 1997−2001 Diagnosed 2002−2006
0
25
50
75
100
Beta−blockersO
therBPLM
Beta−blockersO
therBPLM
Beta−blockersO
therBPLM
Beta−blockersO
therBPLM
Drug combination
Percentageofpatients
PCD
CPRD
DIN−LINK

21. Comparison of patient samples by cancer site
BPLM = Blood pressure
lowering medicines
0
10
20
30
40
breast
colon
lungoesophagus
ovarian
pancreas
prostate
renal
stom
ach
Cancer site
Percentageofpatients
PCD
CPRD
DIN−LINK

22. Survival analyses: comparison of hazard ratios
Cancer type
breast
colon
lung
oesophagus
ovarian
pancreas
prostate
renal
stomach
Overall
Hazard ratio
1.09 (0.8, 1.49)
1 (0.77, 1.3)
1.12 (0.89, 1.41)
1.05 (0.69, 1.6)
1.14 (0.63, 2.06)
1.88 (1.09, 3.25)
1.54 (1.13, 2.09)
1.14 (0.52, 2.52)
1.44 (0.76, 2.74)
1.18 (1.04, 1.33)
% weight
14.85
20.93
25.7
8.17
4.27
4.91
15.19
2.37
3.61
100
0 1 2 3 4
all Beta−blockers vs controls
DIN−LINK
Cancer type
breast
colon
lung
oesophagus
ovarian
pancreas
prostate
renal
stomach
Overall
Hazard ratio
1.19 (1.03, 1.37)
0.85 (0.74, 0.97)
1.04 (0.91, 1.19)
1.27 (1.01, 1.59)
1.05 (0.74, 1.5)
0.94 (0.74, 1.21)
1.03 (0.92, 1.15)
0.46 (0.26, 0.83)
1.03 (0.78, 1.36)
1.01 (0.91, 1.13)
% weight
14.87
15.11
15.36
10.7
6.32
9.76
16.27
2.89
8.71
100
0 1 2 3 4
all Beta−blockers vs controls
CPRD

23. Summary — β-blocker study
1. Different individual cancer HR’s and overall
conclusions
2. Important differences in some cohort statistics
3. Differences remain after correcting to give the same
patient:practice
4. Differences remain after reducing the size of the
CPRD study
5. Databases appear to be demographically similar (Carey
et al. 2004)

24. Summary — β-blocker study
1. Different individual cancer HR’s and overall
conclusions
2. Important differences in some cohort statistics
3. Differences remain after correcting to give the same
patient:practice
4. Differences remain after reducing the size of the
CPRD study
5. Databases appear to be demographically similar (Carey
et al. 2004)
WHY?

25. ”an experiment is reproducable until another
laboratory tries to repeat it.” — Alexander Kohn

26. ”an experiment is reproducable until another
laboratory tries to repeat it.” — Alexander Kohn
• Artifact of differences in GP computer systems?
DIN-LINK uses Torex/iSoft systems (See
Kontopantelis et al 2013)

27. ”an experiment is reproducable until another
laboratory tries to repeat it.” — Alexander Kohn
• Artifact of differences in GP computer systems?
DIN-LINK uses Torex/iSoft systems (See
Kontopantelis et al 2013)
• Analysis methods?
– No matching - potential confounding
– No clustering by practice
– Limited control for covariates
– Is meta-analysis the most appropriate method
(Assumes independence)?
• Data quality?

28. Conclusions / recommendations
These replications add to the evidence
that PCD results are valid

29. Conclusions / recommendations
These replications add to the evidence
that PCD results are valid
• PCD Replication is hard!

30. Conclusions / recommendations
These replications add to the evidence
that PCD results are valid
• PCD Replication is hard!
– Methods details are inadequate for
replication
– Clinical codes not provided with
the original article
– Relies on active cooperation of
authors of original studies
– Even then, ambiguity can remain

31. Conclusions / recommendations
These replications add to the evidence
that PCD results are valid
• PCD Replication is hard!
– Methods details are inadequate for
replication
– Clinical codes not provided with
the original article
– Relies on active cooperation of
authors of original studies
– Even then, ambiguity can remain
• Publish full methods (in online
appendix?)
• Publish full clinical code lists...
ClinicalCodes.org

32. Thanks. . .
Research team
David Reeves
Evan Kontopantelis
Ivan Olier
Darren Ashcroft
Authors of the original
studies
Iain Carey (St. Georges
University, London)
Carol Coupland
(University of
Nottingham)
Contact: david.springate@manchester.ac.uk