This document discusses the evolution of impact evaluations for family planning programs. It provides historical context on impact evaluations dating back to the 1990s, which primarily used randomized controlled trials and quasi-experimental designs. More recent considerations include theory-based approaches, systems-based approaches, and implementation science to evaluate family planning programs. The document recommends accepting a wide range of evaluation designs that meet but not exceed stakeholder needs.

1. Evolution of Family Planning
Impact Evaluation
New context and methodological
considerations
Janine Barden-O’Fallon
Sian Curtis
Jessica Levy
Ilene Speizer

2. Outline of Presentation
 Background and purpose
 Definition of impact evaluation
 Historical context
 Current considerations
 Discussion and recommendations

3. Addressing the Information “Gap”:Addressing the Information “Gap”:
Increased Focus on Impact EvaluationIncreased Focus on Impact Evaluation
WHEN WILL WE EVER LEARN?
IMPROVING LIVES THROUGH
IMPACT EVALUATION
DFID
Department for
International
Development
Working Paper 38
U.S.
Global Health Initiative

4. Definition of Impact Evaluation
“Impact evaluations measure the change in a
development outcome that is attributable to a defined
intervention; impact evaluations are based on models of
cause and effect and require a credible and rigorously
defined counterfactual to control for factors other than the
intervention that might account for the observed change.
Impact evaluations in which comparisons are made
between beneficiaries that are randomly assigned to
either a treatment or a control group provide the strongest
evidence of a relationship between the intervention under
study and the outcome measured.”
-USAID 2011

5. Ideal Impact Assessment

6. Impact Evaluation in FP:
Historical context
The Evaluation Project (1991-1997)
Randomized experimental designs (RCTs)
Quasi-experimental designs
Multilevel regression models

7. Experimental Designs in FP
 Use of RCTs in FP is relatively rare
 Bauman (1997) identified 16 RCTs of FP programs between
1960-1993
 Mwaikambo et al. (2011) identified 6 RCTs of FP programs
between 1995-2009
 QEDs more common
 Groups are often the level of analysis
 Matlab and Navrongo sites

8. Methodological Constraints to
Experimental Designs
 Pure comparison areas may not exist –
other programs in comparison areas or
cross-over of interventions to comparison
areas
 Non-random placement of programs –
intervention areas and control areas often
not comparable
 Need/ability to control for other factors
beyond the program that might affect
outcomes
 External validity can be an issue when
considering replication in a different context
or replication at scale
Source: Victora et al., 2010.

9. Multilevel Regression Models
+ -
Hierarchical design is well
suited for evaluations
pertaining to social context
and its effect on behavior
Access to reliable data for
all levels is necessary
Multi-equation modeling can
control for endogenous
program placement
Linking data sets is often
necessary but not always
possible
A current example is Gates-funded MLE Project collecting data in
urban Senegal from individual women and at the facilities they attend

10. Impact Evaluation in FP: Current
Considerations
Experimental and multilevel models answer
questions like:
“Did the intervention work?”
Not:
“How could the intervention be improved?” or
“Will the intervention work elsewhere” or
“What are the factors that contributed to the
success of the intervention?”

11. Impact Evaluation in FP: Current
Considerations
 Since 1990s, context for FP evaluation has
evolved:
emphasis on rights and vulnerable populations
emphasis on integrating programs
emphasis on expanding coverage, or ‘scaling up’
emphasis on structural interventions

12. Theory-based Approaches
“Theory based evaluation is an approach to
evaluation (i.e., a conceptual analytical model) and
not a specific method or technique. It is a way of
structuring and undertaking analysis in an
evaluation.” -Centre of Excellence for Evaluation, Treasury
Board of Canada Secretariat, 2012
Plausible program logic that identifies key service components, expected
outcomes, and the hypothesized links between them
Often require mixed-methods and/or links across information sources

13. Theory-based Approaches
 National Platform Approach (Victora et al., 2010)
 Large scale evaluations
 District as unit of design and analysis
 Continuous monitoring
 Multiple sources of data
 Variety of evaluation methodologies
 Political and technical constraints to
implementation = remains largely untested

14. Systems-based Approaches
Systems-based approaches place programs within an
environment (physical, social, political, etc.) and assess
the interaction of the environment on the program itself
or on the anticipated outcomes.
Multilevel
Identify underlying mechanisms
Often require complex modeling

15. Systems-based Approaches
Example:
An organizational network
analysis in Ethiopia
identified gaps and
barriers to HIV/FP client
referral networks
-MEASURE Evaluation, 2012
Client referral network in Kirkos, Ethiopia

16. Implementation Science
 To identify the factors in the implementation
process that lead to success or failure and to
develop and test practical solutions
 To identify if and how interventions should be
modified to achieve sustainable impacts
 To determine the best way to facilitate full-scale
implementation
 Ex: How can the barriers to scaling up a FP promotion
program be overcome so that it reaches all women with unmet
need?

17. Discussion Points
 There is no one-size-fits-all impact evaluation
method
 The evaluation question and programmatic
attributes should drive the choice of evaluation
design and methodology
 All methods have strengths and weaknesses and
any evaluation can be poorly or well executed

18. Recommendations
 Accept a wide range of evaluation designs. See
Realworld Evaluation: Working under budget,
time, data, and political constraints (Baumberger,
Rugh and Mabry, 2012)
 Many options exist for addressing design
limitations, including the use of multiple methods
 The rigor of evaluation designs should meet but
not exceed the needs of stakeholders

20. MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for Health,
and Tulane University. Views expressed in this presentation do not
necessarily reflect the views of USAID or the U.S. Government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.