International Food Policy Research Institute (IFPRI) organized a three days Training Workshop on ‘Monitoring and Evaluation Methods’ on 10-12 March 2014 in New Delhi, India. The workshop is part of an IFAD grant to IFPRI to partner in the Monitoring and Evaluation component of the ongoing projects in the region. The three day workshop is intended to be a collaborative affair between project directors, M & E leaders and M & E experts. As part of the workshop, detailed interaction will take place on the evaluation routines involving sampling, questionnaire development, data collection and management techniques and production of an evaluation report. The workshop is designed to better understand the M & E needs of various projects that are at different stages of implementation. Both the generic issues involved in M & E programs as well as project specific needs will be addressed in the workshop. The objective of the workshop is to come up with a work plan for M & E domains in the IFAD projects and determine the possibilities of collaboration between IFPRI and project leaders.

1. Tuesday, April 01, 2014
Problems of Impact evaluation
Confounding factors and selection biases

2. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 2
Objective of Impact Evaluation
Measure the effect of the program on its beneficiaries (and eventually on its
non-beneficiaries) by answering the counterfactual question:
• How would individuals who participated in a program have fared in the absence
of the program?
• How would those who were not exposed to the program have fared in the
presence of the program?
 Two main problems arise: confounding factors and selection biases.

3. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 3
Comparing averages
• Individual-level measure of impact : what would be the outcome (e.g.
purchase patterns) had he/she not participated to the program (in our
case the treatment?
• Compare the individual with the program, to the same individual without the
program, at the same time ?
Pb: can never observe both, missing data problem.
• Instead: Average impact on given groups of individuals
• Compare mean outcome in group of participants (Treatment group)
to mean outcome in similar group of non-participants (Control group)
• Average Treatment effect on the treated (ATT):

4. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 4
Building a control group
• Compare what is comparable.
• Treatment” and “Control” groups must look the same if there was no
program.
• But: very often, those individuals who benefit from the program initially
differ from those who don’t.
• External selection: programs are explicitly targeted (Particular areas,
Particular individuals).
• Self selection: the decision to participate is voluntary.
 Pb with comparing beneficiaries and non-beneficiaries: the difference can be
attributed to both the impact or the original differences.
• SELECTION BIAS when individuals or groups are selected for
treatment on characteristics that may also affect their outcomes.

5. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 5
Initial
Population
Selection
Treatment Group
(receives procedure X)
Impact = Y Exp – Y Control
Quintile I
(Poorer)
Quintile II Quintile III Quintile IV QuintileV
(Richer)
Program selection does not lead to selection bias
(from Bernard 2006)
Control group
(does not receives procedure X)

6. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 6
Initial
Population
Quintile I
(Poorer)
Quintile II Quintile III Quintile IV QuintileV
(Richer)
Control group
(does not receives procedure X)
Treatment Group
(receives procedure X)
Program selection leads to selection bias
Selection
Impact ≠ Y Exp – Y Control

7. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 7
“Sign” of the selection bias (1)
Program targeted on “worse-off” households
Treatment Control
Observed difference is negative
Actual impact

8. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 8
Treatment Control
Observed difference is very large
Actual impact
“Sign” of the selection bias (2)
Program targeted on “better-off” households

9. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE Page 9
Exercise
1. Detail how confounding factors may be an issue in evaluating the impact
of your project.
2. Suppose that you were to compare households in communities were the
project was implemented to households in the neighboring communities
were the project was not implemented.
- What would be the likely sign of the selection bias?
3. Suppose that you were to compare, within the communities were the
project is implemented, households who have decided to use the project
(e.g. drink water from the tap or build stone bunds in their field), to the
ones who have decided not to use it.
- What would be the likely sign of the selection bias?

10. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE
Step 3: What data to collect -Collect qualitative data and
quantitative data on both treatment and control households
in the baseline
• Qualitative data-key supplement to quantitative IE providing
complementary perspectives on program’s performance.
• Approaches include FGD, expert elicitation, key informant
interviews
• Useful 1. Can use to develop hypotheses as to how and why
the program would work
• 2. Before quantitative IE results are out, qualitative work can
provide quick insights on happenings in the program.
• 3. In the analysis stage, it can provide context and
explanations for the quantitative results
Page 10

11. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE
Mixed methods- Quantitative and qualitative
:
• Possible rationale:
• Triangulation: to cross-check and compare results and offset any
weaknesses in one method by the strengths of another;
• Complementarities: examining overlapping and different facets of a
phenomenon by using several approaches and tools;
• Initiation: discovering paradoxes, identifying contradictions, or obtaining
fresh perspectives that relate to the topic of investigation;
• Development: using quantitative and qualitative methods sequentially, such
that results from the first method inform the use of the second method and
vice versa; and
• Expansion: adding breadth and scope to a project to convey findings and
recommendations to audiences with different capabilities and interests.
Page 11

12. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE
Step 3 linked to Step 2: Focusing on quantitative methods-
Propose to execute double difference methods
• Central feature of the method is use of
longitudinal data to use “difference-in-
differences” or “double difference”.
• Method relies on baseline data collected before
the project implementation and follow-up data
after it starts to develop a “before/after”
comparison.
• Data collected from households receiving the
program and those that do not (“with the
program” / “without the program”).
Page 12

13. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE
Double difference methods: continued
• Why both “before/after” and “with/without” data are necessary
?
• Suppose only collected data from beneficiaries.
• Suppose between the baseline and follow-up, some adverse event occurs.
• —the benefits of the program being more than offset by the damage
from bad event. These effects would show up in the difference over
time in the intervention group, in addition to the effects attributable to
the program.
• More generally, restricting the evaluation to only “before/after”
comparisons makes it impossible to separate program impacts from
the influence of other events that affect beneficiary households.
• To guard against this add a second dimension to evaluation design
that includes data on households “with” and “without” the program.
Page 13

14. INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE
Summary of the method and its application
• The approach- By comparing changes in selected outcome
indicators between treatment group and the comparable
control group, the project impact is estimated quantitatively.
• Approach can also be applied to measure spillover effect from
the treated to the non-treated famers in the treated areas.
• examined by comparing the outcomes between non-treated households
in treatment areas and households in control areas.
• Moreover, impact heterogeneity across population sub-groups can be
investigated.
• The sub-groups can be defined based on caste, gender, agro-
ecological zones etc.
• Such information will be collected in the baseline survey.
Page 14