Dr Yen-Fu Chen's presentation on publication bias in service delivery research for the CLAHRC WM Scientific Advisory Group, 10th June 2015, Birmingham, UK

1. Publication bias in health service
delivery research
12/06/2015
Yen-Fu Chen, Richard Lilford
Warwick Centre for Applied Health Research & Delivery (W-CAHRD)
University of Warwick
CLAHRC West Midlands International Scientific Advisory Group

2. Publication and related bias
• Publication bias
The tendency on the parts of investigators, reviewers,
and editors to submit or accept manuscripts for
publication based on the direction or strength of the
study findings (Dickersin 1990)
• Dissemination bias
- Outcome reporting bias
- Citation bias
- Language bias
- Media attention bias
• P-hacking

3. Publication bias in biomedical
literature (and beyond)
• Well established in biomedical research
e.g. Song et al. Health Technol Assess 2010;14(8)
• Led to the creation and enforcement of clinical trial
registries
• Schmucker et al. PLoS ONE 2014;9(12): e114023
- REC: 46% (prediction interval 22% to 72%) published
- Trial registries: 54% (prediction interval 13% to 90%) published
• Increasing evidence in other scientific disciplines: social
sciences, management, economics, ecology and
evolution

4. Publication bias in health
services & delivery research
(HSDR)
• Scoping search identified very limited literature
‘publication bias’ and key terms such as ‘health services research’
or ‘health services management’ or ‘service delivery’ or ‘quality
improvement’ or ‘patient safety’
• Why documentation of empirical evidence is lacking?
- Not an issue in HSDR
- Lack of awareness
- Choose to ignore
- Difficult to locate
- Difficult to study

5. Nature of HSDR
• Definition and boundary
- difficulties in searching the literature
• Multiple variables and relationship between them
• Diverse research methods

6. Publication and related bias
in HSDR: study design
Experimental
studies
(e.g. RCTs)
Quasi-
experimental
studies
(e.g. natural
experiments,
instrumental variable
analyses, ITS, CBA)
Non-
experimental
(observational)
studies
(e.g. uncontrolled
before-and-after
study, analysis of
routine data)
Quality
improvement
projects
ResearchPublication bias?
Scientific
enquiry
Practical
instrument

7. Methods for investigating
publication bias
• Direct evidence
1. Following a cohort of studies over time
2. Survey of researchers
• Indirect evidence – examining the literature
3. Assessing publication and related bias in systematic reviews
4. Methodological (case) studies to explore methods for detecting
and/or mitigating publication and related bias
• Empirical evidence in HSDR & proposed research

8. • Time-sharing Experiments in the Social Sciences (TESS)
• Peer-reviewed, competitive process; surveys conducted by the same
research firm
Empirical evidence 1:
inception cohort of HSDR studies
Franco et al. Science 2014;345:1502-5
Null
(n=48)
Mixed
(n=82)
Strong
(n=91)
Not written 65% 12% 4%
Written but not published 15% 39% 34%
Published (non-top-tier) 10% 38% 38%
Published (top-tier) 10% 11% 23%
2 (6) = 80.3, P < 0.0001

9. • Review of the effects of mass mailings on influenza vaccination
uptake among Medicare beneficiaries
• Medicare Peer Review Organization Health Care Quality
Improvement Project (HCQIP) database
• Six controlled trials identified
• Only one study reporting modest but statistically significant results
was published (increase of 8.7% vs 6.5% vs 4.4%)
• The remaining did not observe clinically meaningful improvement –
none published
Empirical evidence 1: inception
cohort of HSDR studies
Maglione et al. Am J Prev Med 2002;23(1):43–46

10. Proposed research 1: Following
an inception cohort of HSDR
studies over time
• Identification of a suitable cohort
- Lack of comprehensive, accessible registries for HSDR
- Ethics committees
- Papers presented in conferences
- Manuscripts submitted to journals
• Existing evidence suggests non-submission by
investigators is the main cause of non-publication

11. Empirical evidence 2:
Interrogation of HSDR stakeholders
1. Planned or in preparation
“May publish following validation”
2. Not of interest for others (generalizability too limited)
“Constellation of internal social factors, adoption factors, staff
training/experience, etc. seemed too unique to make it general enough”
3. No time for writing
Funding ran out; or new projects started; “Too busy implementing CPOE to
publish”
4. Limited scientific quality
“The setup (e.g., amount of interviews) was not robust enough”
5. Political and legal reasons
“Government was unwilling to publicly share negative content of initial
responses”
6. Only meant for internal use
“The evaluation was only meant for the own organization; academic output
is not necessary”
Ammenwerth & De Keizer, J Am Med Inform Assoc. 2007;14:368–371

12. Proposed research 2:
Interrogation of HSDR stakeholders
• Gather perceptions and first-hand experience of relevant
stakeholders
• Selection of study sample
- Who (academics, policy makers? service managers?)
- What discipline to include
• Questionnaire or interview or focus group

13. Empirical evidence 3:
Published survey of systematic
reviews
9%
23%
68%
Assessment of likelihood of publication
bias in Cochrane EPOC reviews (n=99)
Yes Unclear No Li et al. Health Policy
2015; 119:503–510
Yes: explicitly assessed
Unclear: partial information,
e.g. discussed in conclusion
No: not assessed for some
reason or no information

14. Empirical evidence 3:
Published survey of systematic
reviews
• Overview of systematic reviews of interventions for
improving care for patients with diabetes
• Identified 125 reviews, of which 50 higher quality
reviews further assessed
• 22/50 (44%) assessed likelihood of publication
bias
Worswick et al. Syst Rev 2013;2:26

15. Proposed research 3:
Survey of published HSDR
systematic reviews
• Sampling Cochrane EPOC and other published HSDR
systematic reviews
• Reasons for not assessing publication bias
- Too few studies
- Narrative / qualitative synthesis only
- Perceived heterogeneity
- Something else?
• Where the likelihood of publication bias was assessed:
- Methods used
- Findings

16. Empirical evidence 4:
Methodological case studies
Health care utilisation
Costa-Font et al. Health Policy
2013;109: 78– 87
“The winner’s curse”

17. • Identification and selection of cases
- Experimental vs non-experimental
- Intervention effectiveness vs association
• Candidate topics
- QI interventions for care of patients with diabetes
- Computerised decision support systems
- Audit & feedback
- Weekend admissions & mortality
- Volume & outcomes
- Staffing ratio & outcomes
• Methods to explore
- Funnel plot
- Regression (precision, impact factor etc.)
- p-curve?
Proposed research 4:
Case studies of applicability of
common and emerging methods
Head et al. PLoS Biol 2015;13(3): e1002106
p-curve

18. Thank you & acknowledgement
Critique
• Professor Russell Mannion
• Professor John Øvretveit