This presentation was given in a Future Health System Consortium organised session at the Global Symposium on Health Systems Research in November 2010. The author is Alex Rowe from the Centers for Disease Control and Prevention.

1. Contact for this presentation:
Alexander K. Rowe, MD, MPH
Malaria Branch, Division of Parasitic Diseases and Malaria, Center for Global Health
Centers for Disease Control and Prevention
Mailstop F22
4770 Buford Highway
Atlanta, GA 30341
United States
Telephone: 1-770-488-3588
Fax: 1-770-488-7761
Email: axr9@cdc.gov
Presentation on the literature review of interventions
to improve health care provider performance
(for the First Global Symposium on Health Systems Research, Session on
Evidence on Improving Health Service Delivery, November 18, 2010)

2. Early results of a systematic review
of strategies to improve health care
provider performance in low- and
middle-income countries
Alexander K. Rowe (Malaria Branch, Centers for Disease
Control and Prevention [CDC])
Samantha Y. Rowe (CDC)
David H. Peters (Johns Hopkins University)
Kathleen A. Holloway (World Health Organization)
John Chalker (Management Sciences for Health)
Dennis Ross-Degnan (Harvard Medical School)

3. Background
• Health care providers (HCPs) play key roles
in improving quality and coverage of health
interventions
• In low- and middle-income countries (LMICs),
however, HCP performance often inadequate
• Most existing reviews* of strategies to
improve performance are outdated or have
important methodological limitations
* Two new reviews on community/lay health workers

4. Objectives
Conduct systematic review of strategies to
improve HCP performance and related health
outcomes in LMICs, produce:
1. Library & database of studies on improving HCP
performance, for policy-makers and researchers
2. Detailed report and summary articles on
effectiveness and costs of strategies
3. Evidence-based policy recommendations
(i.e., guidance on implementing clinical
guidelines in LMICs) and associated
research agenda

5. Methods: inclusion and exclusion criteria
• Source of studies. Published and unpublished
• Timing. For electronic databases, studies by May 2006;
for other methods (e.g., gray literature), no a priori end
of search date (i.e., some studies after 2006)
• Setting. LMICs, as per World Bank

6. Methods: inclusion and exclusion criteria
• Source of studies. Published and unpublished
• Timing. For electronic databases, studies by May 2006;
for other methods (e.g., gray literature), no a priori end
of search date (i.e., some studies after 2006)
• Setting. LMICs, as per World Bank
• Type of health condition. No exclusions; performance
related to any health condition is acceptable
• HCPs. Facility- or community-based health workers,
pharmacists, shopkeepers who sell medicines, and
private sector health workers; only exclusions were
household providers (e.g., patient’s family)

7. • Types of outcomes
– Direct measures of HCP behavior (e.g., tasks related
to diagnosis, treatment, counseling)
– Patient outcomes (e.g., mortality, health care utilization)
– Other (e.g., economic outcomes, HCP knowledge)
Methods: inclusion and exclusion criteria

8. • Types of outcomes
– Direct measures of HCP behavior (e.g., tasks related
to diagnosis, treatment, counseling)
– Patient outcomes (e.g., mortality, health care utilization)
– Other (e.g., economic outcomes, HCP knowledge)
• Sample size. >20 observations per study group and
time point
• Language of the study. No exclusions
Methods: inclusion and exclusion criteria

9. Methods: classification of study design
• “Adequate” study designs for primary analysis:
– Pre-post with comparison (+/- randomization)
– Post-only with randomized comparison group
– Interrupted time series (>3 data points before
and after intervention)

10. Methods: classification of study design
• “Adequate” study designs for primary analysis:
– Pre-post with comparison (+/- randomization)
– Post-only with randomized comparison group
– Interrupted time series (>3 data points before
and after intervention)
• Despite interest in other study designs, due to
very large number of reports, focus on studies
with adequate designs

11. Methods: literature search strategy
• Searched: 15 electronic databases (e.g., MEDLINE,
CINHL, EPOC specialized register, etc.)
• Bibliographies of 510 previous reviews & other articles
• Document inventories and websites of 29 organizations
involved with HCP performance (e.g., MSH, BASICS,
Core Project, DFID, PAHO, QAP, USAID, World Bank)
• INRUD Bibliography and WHO Rational Use of Drugs
database
• Asked colleagues for references & unpublished studies

12. Methods: data abstraction
• Double, independent abstraction with
20-page abstraction form
• Data entered into Access database
• Queries to study investigators
(clarifications, details on contextual factors,
etc.)

13. • Effect size in terms of %-point change
• If outcome is a percentage, effect size =
(%POST – %PRE)intervention
– (%POST – %PRE)control
• Calculate effect size such that positive = success
• If >1 outcome, take median effect size (MES) of
primary outcomes
Methods: analysis of effect size

14. Early, illustrative results
(might not represent final dataset)

15. • >105,000 citations screened
• After removing duplicates: 2,430 reports
identified for abstraction (all study designs)
Results: Literature search

16. Early results: data abstraction
• From August 2007, studies with “adequate”
designs prioritized to be abstracted first
(851 reports on about 425 distinct studies)
• To date, 463 reports double-abstracted on
275 distinct studies
• Studies from >60 LMICs (50% low-income)
• Wide range of health conditions addressed by
HCPs: nutrition; antenatal care; treatment of
ARI, diarrhea, or malaria; family planning, etc.

17. Early results: study designs (N=275)
• “Adequate” designs (223/275, or 81%)
− 145 randomized studies (54%)
• Other designs (52/275, or 19%), e.g., pre-post
study without controls (not primary focus)

18. Early results: strategies tested (N=275)
• 390 intervention groups (i.e., strategies tested)
• 456 comparisons
− 177 intervention vs. “no intervention” control
− 249 intervention A vs. intervention B
• Most strategies had multiple components
(e.g., training + supervision = 2 components)
• Median of 4 components/strategy (range: 1–17)
• Commonly tested strategies include: training,
supervision or feedback, community activities,
new guideline, printed materials, and job aids

19. Early results on effectiveness:
Analysis of 213 comparisons from
172 studies with adequate design
(all comparisons are:
intervention vs. no intervention control)
(might not represent final dataset)

20. 0
5
10
15
20
25
30
35
Magnitude of effect size (percentage-point change)
No.ofMES
-100
to
-109
-0.1
to
-9
0
to
9
10
to
19
-20
to
-29
-30
to
-39
-10
to
-19
20
to
29
40
to
49
50
to
59
30
to
39
60
or
higher
Distribution of 213 MES from 172 studies with
adequate design (all strategies)
Min = -105
Median = 9
Max = 130
IQR: 3–23

21. 0
5
10
15
20
25
30
35
Magnitude of effect size (percentage-point change)
No.ofMES
-100
to
-109
-0.1
to
-9
0
to
9
10
to
19
-20
to
-29
-30
to
-39
-10
to
-19
20
to
29
40
to
49
50
to
59
30
to
39
60
or
higher
Distribution of 213 MES from 172 studies with
adequate design (all strategies)
50% are small
(<10%-points)
or negative
Min = -105
Median = 9
Max = 130
IQR: 3–23

22. 0
5
10
15
20
25
30
35
Magnitude of effect size (percentage-point change)
No.ofMES
-100
to
-109
-0.1
to
-9
0
to
9
10
to
19
-20
to
-29
-30
to
-39
-10
to
-19
20
to
29
40
to
49
50
to
59
Min = -105
Median = 9
Max = 130
IQR: 3–23
30
to
39
60
or
higher
Distribution of 213 MES from 172 studies with
adequate design (all strategies)

23. -110
-60
-10
40
90
140
16 315No. comparisons
Trainingonly
Jobaids/printed
mat.forHCPonly
Training+jobaidsor
printedmat.forHCP
Median MES (%-pts) 10 7 2 13 13 6 10 17
10 17 18 79 14
Train+qualmgmt+
jobaids/printed
materialsforHCP
Community
activities+other
Non-community
activitiesinvolving
commodities+other
Training+quality
management
Qualitymgmt
techniqueonly
Medianeffectsize(%-points)
75th
percentile
median
25th
percentile

24. -110
-60
-10
40
90
140
16 315No. comparisons
Trainingonly
Jobaids/printed
mat.forHCPonly
Training+jobaidsor
printedmat.forHCP
Median MES (%-pts) 10 7 2 13 13 6 10 17
10 17 18 79 14
Train+qualmgmt+
jobaids/printed
materialsforHCP
Community
activities+other
Non-community
activitiesinvolving
commodities+other
Training+quality
management
Qualitymgmt
techniqueonly
Medianeffectsize(%-points)
75th
percentile
median
25th
percentile

25. -110
-60
-10
40
90
140
16 315No. comparisons
Median MES (%-pts) 10 7 2 13 13 6 10 17
10 17 18 79 14
Medianeffectsize(%-points)
75th
percentile
median
25th
percentile
• All median values small/moderate (2–17 %-points)
• Some groups with large variation (some effects large;
potential for positive deviance analysis)

26. -110
-60
-10
40
90
140
Medianeffectsize(%-points)
1 2 3 4 5 >6
No. of components in the strategy
Does MES vary by the number of
components in the strategy?

27. -110
-60
-10
40
90
140
Medianeffectsize(%-points)
1 2 3 4 5 >6
No. of components in the strategy
75th
percentile
median
25th
percentile
Does MES vary by the number of
components in the strategy?

28. • Extreme heterogeneity among studies
– Settings
– Outcomes
– Interventions (e.g., not all “training” is same)
• MES crude summary measure that can
mask variation in effect (improved methods
in development)
• Precision very difficult to assess
Limitations

29. 1. Surprisingly large evidence base on effectiveness
of strategies, with many strategies tested in many
settings
2. Effect sizes vary substantially: half have small
effects, but some have large effects
3. Analyses to identify factors associated with high
effectiveness
Early conclusions (1)

30. 4. Evidence base seems fragmented
– Researchers not building on each others’ work
– Varying methods makes it difficult to put pieces
together
4. Standardizing methods & outcomes would
improve ability to summarize evidence base and
develop effective and practical recommendations;
perhaps coordinated research agenda is needed
5. Final results pending (end of 2012)
Early conclusions (2)

31. Acknowledgments
• Charity Akpala
• Tashana Carty
• Adrijana Corluka
• Didi Cross
• Bhavya Doshi
• Onnalee Gomez
• Meg Griffith
• Karen Herman
• Qing Li
• Connie Liu
• Earl Long
• Eliza McLeod
• Dawn Osterholt
• Gabriel Ponce-de-Léon
• Nancy Pulsipher
• Atiq Rahman
• Nirali Shah
• Banafsheh Siadat
• Sanja Stanojevic
• Savitha Subramanian
• Jeff Willis
• Kindra Willis
• Shannon Wood
• Karen Wosje
• Alicia Wright
• Chunying Xie
• Special thanks to investigators who responded to queries
• Funding: Bill and Melinda Gates Foundation, CDC, World Bank

32. Thanks for your
attention!
“Results! Why, man, I have gotten a lot of results.
I know several thousand things that won’t work.”
Thomas A. Edison (1847–1931)