Dr John Ovretveit's critique on Dr Yen-Fu Chen's presentation on publication bias in service delivery research for the CLAHRC WM Scientific Advisory Group, 10th June 2015, Birmingham, UK

1. Publication Bias
1
John Øvretveit,
Director of Research, Professor of Health Innovation and
Evaluation, Karolinska Institutet, Stockholm, Sweden
6/12/2015

2. Key points
QI interventions - many sites
5-10% average - uncertain attribution = no publish
But high variation between sites
1) Bias to internal validity rather than
external
2) Bias against adaptive implementation
action evaluation
& practitioner partnership research (audit)
3) Bias to intervention research rather than
descriptive explanatory & multi-method26/12/2015

3. RCT of intervention to implement guidelines for
management of urinary tract infection and sore throat
 Trial found average little change, But variation
36/12/2015
Why did
these
change so
much?

4. Process evaluation in parallel to RCT
“A combination of organizational problems
… and lack of time and engagement
…is the most viable explanation for the lack of
effect”
 agreement with guidelines;
 degree of participation in the project;
 taking time to discuss the guidelines and their implementation;
 use of the components of the interventions;
 procedures for telephone consultations;
 communication within each practice.
46/12/2015

5. Under-used “top- and bottom- 5” analysis
 Prospective theory-informed
 Which sites would you expect better performance and why
 Retrospective investigation
 Informant’s theories
 Researchers analysis
 Bias against explanatory and favors
quantitative statistical association
 All Biases = less relevant to
practitioners 5

6. 6/12/2015
6
What do you think?
Surprises?
My examples / experience?