Presented by: Deanna Olney, PhD October 19, 2016

1. Evaluation of Nutrition-Sensitive
Programs*
Presented by: Deanna Olney, PhD
October 19, 2016
* Olney, Leroy and Ruel. Evaluation of Nutrition Sensitive Interventions. Chapter 44. In: De Pee, Taren and
Bloem (eds.) Nutrition and Health in a Developing World. 3rd Edition (forthcoming).
* Leroy, Olney and Ruel. Evaluation of Nutrition Sensitive Interventions. AOTR

2. Background
• The Lancet Series on Maternal and Child Nutrition calls
for greater investments in large-scale nutrition-
sensitive programs.
• Rigorous evidence of their effectiveness, how they
work and cost-effectiveness is scant attributed to:
• Weaknesses in program design and implementation
• Poor evaluation designs.
• There is a need for rigorous, theory based-evaluations
that can provide evidence of:
• What works to improve nutrition
• How programs work to achieve nutrition impacts
• What other impacts programs have
• Cost and cost-effectiveness of achieving impacts

3. • Complex in design and implementation:
• Multiple goals, inputs, pathways of impacts
• Span across different sectors (e.g. health, agriculture,
education)
• Within each program intervention potential variability in:
• Delivery (quantity and quality)
• Utilization
• Adherence to the program protocol
Key challenges: Program
complexity

4. Key challenges:
Long time frames and impact
pathways
• Long time frames for implementation and evaluation:
• Long pathways from program inputs to impact
• Implement program at the level of quality desired
• Achieve meaningful impacts on some biological outcomes
such as child growth
• Design and carry out a rigorous and comprehensive
evaluation
• … vs often short time frames imposed by donors

5. Key challenges:
Differing priorities, expectations,
incentives, and perceptions
• Priorities of implementers:
• Deliver a high quality program
• Meet targets within the specified budget and time frame
• Priorities of evaluators:
• Rigorously evaluate the program
• Answer key questions related to what impact was achieved
why or why not it was achieved and at what cost
• Misperceptions that can undermine trust:
• Process evaluation viewed as evaluating the performance of
the program implementers vs generating evidence to
improve programming and understand impacts
• Cost studies can be perceived as auditing the program’s
finances

6. Key challenges:
Implementation constraints vs.
evaluation rigor
Treatment 1
Treatment 2
Treatment 3
Control
Regular program
Cluster randomized controlled design
T24
T18
TNFP
Control
Regular program

7. Key challenges:
Assessing benefits beyond
targeted beneficiaries
• Potential “spillover” of benefits beyond the
targeted beneficiaries:
• E.g. siblings, neighbors, etc.
• Adds complexity:
• Capturing all impacts of nutrition-sensitive programs
• Measuring the cost-effectiveness of programs

8. Two essential ways to
address these challenges
• Build a strong partnership and collaboration
between implementers and evaluators
• Design and use a rigorous and comprehensive
evaluation framework:
In-depth understanding of the program, program theory,
program impact pathways, rigorous comprehensive
evaluation design

9. Comprehensive evaluation:
Embrace the complexity of
the program
HKI partners
with local
NGOS
Village Model
Farms (VMF)
established and
VMF owners
trained
Small animal
production
established
Increased
production of
micronutrient-
rich fruits and
vegetables
Women’s
groups
established
Agriculture
inputs (e.g.
seeds, saplings
and poultry
distributed)
Provision of
agriculture
training
Provision of
nutrition, health
and hygiene
education
Improved and
developed
gardens
established
Increased
poultry
production
Increased
household
consumption
Increased
Income
Improved
maternal and
child health and
nutritional
status
Health, hygiene
and nutrition
education
received and
understood by
beneficiaries
Improved child
care and
feeding
practices
Outcomes
Agriculture
training
received and
understood by
beneficiaries
Work with
local authorities
(e.g. village
chief)
Village Health
Support Group
identified and
trained
Improved
women’s
empowerment
Beneficiaries’
agriculture
knowledge
improved
OutputsInputs
Women’s
increased
control over
resources
resource
ImpactsProcesses
Beneficiaries’
health, hygiene
and nutrition
knowledge
improved
Impact
evaluation
What is
the
impact of
the
program?
Process + impact evaluation
How and why
does the
program (not)
have an
impact?
Cost
study
Cost of
the
program?

10. Impact evaluation:
Experimental designs
• Designed to assess program impacts at different levels
• Uses quantitative methods
• Ideally conducted before the program starts and at the end of the
program but can include intermediary time points
• Requires a valid counterfactual and appropriate target groups,
duration and indicators (with adequate sample sizes for each)
• Experimental (or randomized) designs considered gold standard for
impact evaluations.
• Randomization: individual or group (cluster) level.
• *If* done well, one can assume:
• that both groups are comparable;
• that the only difference between the groups is the program;
• that the control group provides a valid counterfactual for the intervention group
exposed to the program.
differences found in the outcomes of interest attributable to
the program.

11. Impact evaluation:
Invalid counterfactuals
68.2
74.8
64.1 64.3
2010 2014
68.2
74.8
64.7
63.0
2010 2014
68.2
74.8
62.3
64.1
2010 2014
68.2
74.8
64.6
65.7
2010 2014
-5.5
-7.1
-5.2
-4.0
-10.0
-5.0
0.0
60.0
65.0
70.0
75.0
80.0
2010-2014 2010-2014 2010-2014 2010-2014
All T24 T18 TNFP
Control Treatment Impact (95% CI)
Impactonstunting(pp)
Wave
68.2
74.8
64.1 64.3
2010 2014
68.2
74.8
64.7
63.0
2010 2014
68.2
74.8
62.3
64.1
2010 2014
68.2
74.8
64.6
65.7
2010 2014
-5.5
-7.1
-5.2
-4.0
-10.0
-5.0
0.0
60.0
65.0
70.0
75.0
80.0
2010-2014 2010-2014 2010-2014 2010-2014
All T24 T18 TNFP
Control Treatment Impact (95% CI)
Impactonstunting(pp)
Wave
Stunting(%)Impact(pp)
0.0
-5.0
Randomized
design
-5.5 pp
68.2
74.8
64.1 64.3
2010 2014
68.2
74.8
64.7
63.0
2010 2014
68.2
74.8
62.3
64.1
2010 2014
68.2
74.8
64.6
65.7
2010 2014
-5.5
-7.1
-5.2
-4.0
-10.0
-5.0
0.0
60.0
65.0
70.0
75.0
80.0
2010-2014 2010-2014 2010-2014 2010-2014
All T24 T18 TNFP
Control Treatment Impact (95% CI)
Impactonstunting(pp) Wave
Before-and-
after
+ 0.2 pp
68.2
74.8
64.1 64.3
2010 2014
68.2
74.8
64.7
63.0
2010 2014
68.2
74.8
62.3
64.1
2010 2014
-5.5
-7.1
-5.2
60.0
65.0
70.0
75.0
80.0
2010-2014 2010-2014 2010-2014
All T24 T18
Control Treatment
Impactonstunting(pp)
Unadjustedstunting(%)
Wave
With-and-
without
- 10.5 pp

12. How and why does the
program (not) have an impact?
• Two ways to assess how impact is achieved :
• Measurement of intermediary measures (outcomes) in impact
study
• Process evaluation
• Can also be used to improve on-going or future programs
• Solid understanding of program theory and program
impact pathways is key

13. Process evaluation
• Examines the primary inputs, processes, outputs and
outcomes along the primary program impact pathways
• Conducted about one year after program implementation
begins and can be repeated
• Use a mix of quantitative and qualitative methods to
examine the different steps along the impact pathways
• Include different key stakeholders (e.g. program
implementers and beneficiaries) and different program
delivery points
• Share the results:
• Workshop with implementers
• Identify what is feasible to improve and how improvements can
be made

14. Cost study
• Designed to estimate the overall cost of the program, the cost
of the main program components and its cost-effectiveness
• Can also be used to estimate savings or costs associated with
adding, changing or dropping program activities or
beneficiaries
• One method that can be used is the Activity Based Costing
Ingredients (ABC-I) approach.
• Using the program impact pathways: provide detailed description of
all program activities.
• Identify the program’s main activities and “ingredients” needed for
each.
• Define the unit cost algorithms, i.e. the different types, quantities
and costs of the “ingredients” necessary for each activity.
• Calculate cost of each program activity and of full program.

15. In summary
• Evidence of what works in nutrition-sensitive programs,
how and at what cost is extremely limited.
• Evaluation of nutrition-sensitive programs is complex but
necessary.
• Strong partnerships between program evaluators and
implementers are needed to carry out rigorous program
evaluations and to ensure that results are used to improve
programming.
• Comprehensive evaluations should include rigorous impact
and process evaluations and when possible cost studies.
• Guidance for future investments requires strong evidence
from rigorous, theory-based comprehensive evaluations of
different nutrition-sensitive program models that bring
together interventions from a variety of sectors.