A presentation from the Global Financing Facility (GFF) in Support of Every Woman Every Child Workshop, held in Nanyuki, Kenya.

1. SHAPING THE LEARNING AGENDA
The way forward
Dinesh Nair and Jake Robbins

2. Program Implementation provides many
Opportunities for Learning
Q u a l i t a t i v e
L e a r n i n g

3. Two Broad Pathways to Learn
with opportunities to intertwine them
• Implementation Research and Delivery Science
• different forms: operations research, mixed method
• helps understanding the context
• Impact Evaluation
• Establish causal link between observed results and evaluated program
• Usually has a credible counterfactual, in the form of a valid control or
comparison group, is critical

4. Implementation Research and
Delivery Science

5. The Five I’s of DDS
5

6. What results will DDS strive to achieve?
Support teams, countries and partners to:
6
Make the best possible investment decisions
Respond to, generate demand for and deliver services
to the best feasible standards,
for the right people,
in the right places,
in the right ways,
at the right time
Achieve the best possible health impact
Plan early to ensure that proven approaches are
institutionalized and sustained

7. India AIDS DDS Example
M
7

8. Operational Data helps track
sub national performance
0
5
10
15
20
25
30
35
40
45
1 2 3 4 1 2 3 4 1 2 3 4 1
2012 2013 2014 2015
%
Pilot
Expansion 1
Expansion 2
Expansion 3
QUALITY OF CARE
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
December
March
June
September
December
March
June
September
December
March
June
September
December
March
2011 2012 2013 2014 2015
AxisTitle
Axis Title
Adamawa
Nasarawa
Ondo
National
Dynamic quality measures
improve outcomes
Recalibration
of quantified
quality
checklist
COVERAGE OF
INSTITUTIONAL DELIVERY
Coverage increases sustained
over phases
Nigeria

9. Nigeria case study on key determinants
• What differentiate the good
and poor performers under
the PBF scheme?
• Health center management
• Contextual factors
• Health systems factors (e.g.,
supervision)
• Interviews, document review,
direct observations
• Best and poorest performers
• Devise appropriate support to
poor performers
Research
question
Areas to
look into
Approaches
Potential
use

10. Detailed look at the operational data revealed the large variations in performance
across Health Centers
Institutional Delivery in Adamawa, normalized by 100,000 population
-
20
40
60
80
100
120
140 Pariya HC
Chigari HC
Dasin Hausa HC
Farang HC
Ribadu HC
Furore MCH HC
Choli HC
Gurin HC
Malabu HC
Karlahi HC
Wuro Bokki HC
Kabilo HC
Saint Mary's Clinic HC
Mayo-Ine HC
• Before PBF, all
health centers
were equally at
very low levels
• After the PBF,
some facilities
achieved 100%
coverage while
others struggle
with limited
improvement

11. Case study on determinants suggests the importance of community engagement and
OIC management
Identified determinants and non-determinants (preliminary)
Non-Determinants
• Level of staffing (best performers
lack staff)
• Remoteness of facilities (best
performers are very rural)
• Technical qualifications of OIC
(many community health workers
manage facilities well)
• Business planning (none use it
effectively yet)
Determinants
• Community engagement (e.g.,
involve and reward community
leaders, daily visits, incentivize
for use of facility)
• OIC’s management capacity
(e.g., full staff involvement,
improve staff environment using
performance bonus, rigorous
performance review)

12. Impact Evaluation

13. Impact evaluation
Impact Evaluation:
• Explores whether there is a causal link between the RBF
intervention and the results.
13
P
r
o
v
i
d
e
r
Report
________
_quantity_
_quality__
_services_
_________
P
u
r
c
h
a
s
e
r
Counter
Verification
Verification
Contract
________
_quantity_
_quality__
_services_
_________
O
u
t
p
u
t
s
O
u
t
c
o
m
e
s
IE IE

14. An Impact Evaluation Answers:
What was the effect of the program on
outcomes?
How much better off are the beneficiaries
because of the program/policy?
How would outcomes change if one changed
the program design?

15. Cameroon RBF Impact evaluation design
• Research questions
1. Does the PBF program increase the coverage and of MCH
services?
2. Is it the enhanced monitoring & evaluation and
supervision or the link between payments and results
that leads to improvements observed in quality or
coverage?
3. What is the contribution of enhanced supervision and
monitoring to improving MCH service coverage and
quality in the absence of increased autonomy or
additional financial resources?
• Public randomization ceremonies in each region
• Baseline June 2012, Endline June 2015
• Results expected by December 2015
• Midline qualitative study embedded in impact evaluation

16. Cameroon RBF IE Midline Qualitative Study
• Focused on two primary themes:
• Experiences in the piloting of PBF at the central, regional and district level:
perspectives of decision-makers and policymakers
• Experiential elements of health service delivery at the operational level:
perspectives of patients and providers
• Research questions:
• What has been the experience of piloting performance-based financing at
various administrative and operational levels of the health system in
Cameroon?
• What has been the experience of health service delivery for health workers,
patients and community members during the first two years of Performance-
based financing?

17. Qualitative study: Sampling Methodology
PBF in Cameroun: Mid-line Qualitative Study Interview Format Total IDIs Total FGDs
Proposed Sampling and # of IDIs and FGDs IDIs FGDs
Central Level
Ministry of Health 2 x 2
Development Partners 2 x 2
Regional Level (4 TOTAL Regions: NW, SW, Littoral, and East)
Regional Health Delegates 1 x 4
District Medical Officers 2* x 8
IE Study Groups (T1, C1, and C2)
District Hospitals and Primary Health Centres (Directors and Administrators) 6** x 72
Community Level
Community Leaders 2*** x 32 x
Community Members, Health Workers, etc. x 4/region*** 16
* 2 DMOs per region
**Stratifed by IE Study Group; 18 total IDIs per region
***Following FGD catchment (2 per village/4 villages total/8 per region) GRAND TOTALS for Midline Qualitative Study
****Stratifed by T1/C2 IE Study Groups; Urban/Rural; Female only IDIs 120
FGs 16
136 interviews TOTAL

18. Qualitative study: Results
• Positive impact on service delivery
• Facilities are cleaner, more organized and better managed, staff motivated
• Increased utilization and quality, price reduction for services
• Better availability of drugs (breaking public monopoly)
• Reduced under the table payments
• Increased collaboration among the various stakeholders
• Regional/district supervision teams and health facilities (in particular private
health facilities)
• Management tools and procedures used in PBF led to enhanced
transparency and accountability in resource management
• Obstacles encountered
• Initial reluctance and adjustment of health facility staff to program
• Obtaining initial buy-in and support of providers and ministry
• Lack of confidence in Ministry of Health to follow through with promises
• Challenges with increasing autonomy of health facilities 18

19. Discussion
• What opportunities exist to make Learning a key part of the GFF
Agenda
• How can “Learning” be mainstreamed through Investment Cases,
implementation plans , projects?
• What technical and other support will be required to make this a
reality?
• How can we incentivize learning and dissemination to be a major
part of the value proposition?