1) Conducting impact evaluations in fragile settings is challenging but necessary to identify effective interventions and efficiently allocate scarce humanitarian funds. 2) This document outlines the challenges of impact evaluations in fragile settings such as complex environments, time constraints, and ethical concerns, and provides solutions like quasi-experimental designs, mixed methods, and tracking techniques to address issues like invalid counterfactuals and attrition. 3) Rigorous impact evaluations are possible in fragile settings with flexible designs, collaboration between agencies, and a focus on less but more reliable evidence to balance evaluation rigor with humanitarian needs.

1. Economist, FAO
From Challenges to Solutions: A
Comprehensive Guide for Conducting
Impact Evaluations in Fragile Settings
Jeanne Pinay
Jeanne Pinay and Marco d’Errico

2. #2023 AGRODEP CONFERENCE
Motivation
• 2022: 274 million people require humanitarian assistance and protection
• 260 million people (95 percent) living in fragile areas (UN OCHA, 2022)
Fragile countries have attracted
increased attention from actors of
the humanitarian and
development world
• 2021: total international humanitarian assistance = USD 31.3 billion (GHA,
2022)
Enormous amounts of
humanitarian aid have been
injected into crisis countries
Policymakers need to identify
interventions that work best to
better assist people in need, and
more efficiently allocate funds
Rigorous IE in fragile settings is
necessary…
… because rigorous IE in fragile settings is challenging!
… but not well-developed
Why?

3. #2023 AGRODEP CONFERENCE
Contribution and target audience
• Based on the literature (esp. Puri et al., 2017; Bakrania et al., 2021) and new evidence
from FAO case studies
• Contributes to the literature on aid effectiveness and impact evaluation methods
• Target audience: International and local researchers, practitioners, and donors involved
in impact evaluations in fragile settings
Challenges
Implications on data
quality and validity of
impact evaluation
Solutions
Identify the challenges encountered
Determine appropriate solutions to maintain
design, data quality, and credibility of the IE
• Provide guidance to researchers
and local actors working on impact
evaluation in fragile contexts to:

4. #2023 AGRODEP CONFERENCE
Definition of fragile settings and FAO
case studies
• OECD (2022): Fragility is the
combination of exposure to risk and
insufficient coping capacity of the
state, system and/or communities to
manage, absorb or mitigate those risks
• Based on the assessment of the
respective balance between risk and
coping capacities, contexts are
categorized as extremely fragile,
fragile, or ‘rest of the world’

5. #2023 AGRODEP CONFERENCE
Challenges
Practical
• Complex settings or
unpredictability
• Burkina Faso, Mali,
Niger, Nigeria,
Somalia
• Need for speed or time
constraints
• Burkina Faso, Gaza
Strip
• Multiplicity of actors
• High co-variability or
invalid counterfactual
Ethical
• Do no harm
• Gaza Strip
• Randomization
and ethics
• Myanmar
Methodological
• Selection bias
• All FAO case studies
• Information bias
• South Sudan
• Contamination bias
• Somalia
• Attrition
• Somalia, Nigeria, …
Local capacity
• Misidentification of the
treated
• Pakistan
• Misunderstanding on IE
design
• Madagascar
• No HHID
• Gaza Strip
• No experience in tracking
HHs
• Remote trainings and lower
degree of supervision

6. #2023 AGRODEP CONFERENCE
Implications
Invalid
counterfactual
Measurement
error
Attrition

7. #2023 AGRODEP CONFERENCE
Solutions
Invalid
counterfactual
• Quasi-experimental methods
• IV, RDD, DD, Matching
• Combining primary with
secondary data
• Somalia, Madagascar
• Alternative designs for
randomization
• Phase-in randomization
• Factorial design
• Randomization at higher
levels (e.g., village)
Measurement error
• Before fieldwork
• Pilot testing
• Reduce questionnaire
• During fieldwork
• CAPI
• Mixed methods
• Gaza Strip
• After fieldwork
• Correct measurement
error
• Secondary data
• Nigeria
Attrition
• Before fieldwork
• Regular assessment of security
situation
• CATI
• Diversified C group
• Oversampling
• Clear communication
• During fieldwork
• Tracking techniques
• Nigeria
• After fieldwork
• Attrition tests
• Correct attrition (IPW, CiC, bounding)

8. #2023 AGRODEP CONFERENCE
Conclusions
• Rigorous impact evaluation in fragile settings is challenging but possible, and
necessary  cannot wait for the right setting to generate evidence!
• Strike a balance between
• Rigor of the evaluation vs. assistance of people in need
• Rigor of the evaluation vs. costs
• Collaboration among humanitarian agencies, donors and local actors
• Agree upon a common research agenda
• Flexible and adjustable budget
• Combine data collection efforts
Investing substantial public resources in programs
whose effectiveness is unknown is unethical!
(Gertler et al., 2016)
• Program adaptability and flexibility
• Focus time and resources on less
but more reliable evidence

9. #2023 AGRODEP CONFERENCE
Need to identify
innovative and low-
cost techniques
• research designs
• statistical methods
• survey methods
Way forward
Geospatial data
Satellite data
Remote-sensing data

10. #2023 AGRODEP CONFERENCE
References
• Bakrania, S., Balvin, N., Daidone, S. & De Hoop, J. (2021). Impact Evaluation in
Settings of Fragility and Humanitarian Emergency. Office of Research – Innocenti
Discussion Paper DP-2021-02 February 2021
• Global Humanitarian Assistance. (2022) GHA Report 2022.
https://devinit.org/documents/1193/GHA2022_Digital_v8_DknWCsU.pdf
• OECD. (2022). States of Fragility 2022. OECD Publishing.
https://doi.org/10.1787/c7fedf5e-en
• Puri, J., Aladysheva, A., Iversen, V., Ghorpade, Y. and Brück, T. (2017). Can rigorous
impact evaluations improve humanitarian assistance? Journal of Development
Effectiveness, 9:4, 519-542, DOI: 10.1080/19439342.2017.1388267
• UN OCHA. (2022). Global Humanitarian Overview 2022, United Nations Office for
the Coordination of Humanitarian Affairs (UN OCHA), New York,
https://gho.unocha.org/

11. THANK YOU