Growing demand for impact assessment of agricultural research

1. Evaluating Economic Impacts of
Agricultural Research: Examples
and Lessons
George W. Norton
Agricultural
and Applied
Economics
Seminar at the International Center for Tropical Agriculture (CIAT),
Cali, Colombia, June 30, 2015

2. Introduction
 Growing demand for impact assessment of
agricultural research
 Improvements in assessment methods
 Agriculture faces dynamic environment
 Population, income, climate, energy, pests
 Multiple goals and non-priced benefits
 Institutionalized system for research data
management useful for impact assessment

3. Objectives
 Key impact evaluation issues
 Assessment examples
 Lessons

4. Key Impact Evaluation Issues
1. Counterfactual (what would have
happened without the research)
2. Multiple objectives
3. Aggregation
4. Integrating impact assessment with
research data management

5. D
S0
S1
Price
Quantity
0
P0
P1
d
a
b
c
I0
I1
Q0 Q1
R
Bt = P0Q0K(1+.5Ken/(e+n)) =
Where: (1) K = (a-c)/a reflects yield and cost changes, technology adoption,
probability of success, and (2) e and n = supply and demand elasticities
1. Identifying what would have happened
without the research

6. Estimating K is Key
Kt=((E(Y)/ε) - (E(C)/(1+E(Y) )At(1-d)t
Kt = Per unit cost reduction
E(Y) = proportionate yield increase per ha for
adopters
ɛ = the price elasticity of supply
E(C) = the proportionate variable input cost change
per hectare
A = proportion of the area affected by the technology
d = the technology depreciation rate

7. Approaches for estimating K
• For specific technologies, can obtain K from:
• Expert opinions of scientists and others
• Input and yield data from biological field
experiments in budgets combined with
adoption data from surveys
• Farm-level survey data in regressions (e.g.,
using instrumental variables, propensity score
matching, double difference)
• Randomized controlled trials (RCTs);villages
and farmers are randomized with treated
(receives technology) and untreated groups
• Usefulness narrow for research evaluation

8. Retrospective (Ex Post) versus
Prospective (Ex Ante)
 Impact assessment methods can be
similar, but data sources differ
 Analysis often part ex ante, part ex post
 Probabilities and expectations are key in
ex ante impact analysis
 (Probability of research success) X
(Expected cost change per unit) X
(Expected adoption ratet)

9. Example of estimating K (part
ex ante, part ex post)
 Myrick et al (2014): benefits of
biocontrol program for papaya
mealybug in Southern India
 Benefits of more than $500 million on an
investment of $500,000

10. Ex post: CIAT-VT (DIVA) evaluation of
bean varieties in Rwanda and Uganda
 Larochelle et al (2015)
 Yield impacts estimated econometrically (IV)
with plot-level data from 1440 households in
Rwanda and 1908 H.H. in Uganda
 Compared counterfactual and actual income
distributions -- Poverty
would have been 0.4 and
0.1 percent higher in
Rwanda and Uganda in
absence of the improved
bean varieties.

11. Counterfactual for the Value of
CIP Genebank
 Study underway to assess, for
varieties that used material (genetic
resources) from CIP Genebank, what
it would have cost to obtain the
desired traits elsewhere without using
the Genebank.
 Provides lower bound but credible
economic estimate of GB value

12. 2. Managing Multiple objectives
 Productivity/Income
 Poverty
 Environment
 Health/nutrition
 Risk/Resilience
 Gender
Tradeoffs among objectives; effects on
some easier to measure than others

13. Price
Quantity0
S0
S1
D
P0
P1
a
b
cd
I0
I1
Q0 Q1
a) Productivity or Income Impacts
Δ TS = +
Δ CS = +
Δ PS = -

14. Example: Ex post impacts of improved
maize varieties in rural Ethiopia
 Zeng et al (2015) Plot-level yield and cost
changes due to adoption were estimated in an
IV econometric model
 Results were included in an economic surplus
model to identify the counterfactual household
income that would have existed without
improved maize varieties.
 Poverty differences assessed -- Improved
maize varieties have led to a 0.8–1.3
percentage drop in poverty headcount ratio

15. b) Poverty Impacts
 Income gains can be estimated, adoption assessed, and
change in poverty rate calculated using a poverty index
(such as Foster-Greer-Thorbecke) or by calculating
income distributions with and without the intervention.
Assessing changes in poverty indexes or distributions
are complementary with RCTs, IVs, economic surplus
analyses, and other impact assessment methods.

16. Example
 Moyo et al (2007) calculated economic
surplus changes from virus resistant
groundnut varieties, disaggregating
income and poverty rate changes from
FGT poverty index to (a) adopters who
were also groundnut consumers, (b)
adopters who were not, and (c)
consumers who were not groundnut
producers (.5% to 1.5% poverty
reduction)

17. c) Environmental or Sustainable
Intensification Impacts
 Many methods for assessing bio-
physical (RCT, IV) and economic
values (CV, Choice Experiment,
Benefit Transfer)
 Must document research-induced
biophysical changes first
 Soil loss avoided, pesticide risk reduction,
carbon sequestered, etc.
 Then value non-market benefits of
technology or policy change

18. Examples
 Using contingent valuation, Cuyno et al.,
(2001) estimated the value of environmental
benefits from IPM-induced pesticide risk
reduction on onions to be $150,000 per year
in six villages in the Philippines.
 Using a choice experiment, Vaiknoras et al.,
(2015) estimated that farmers would be
willing to pay $10 per hectare in eastern
Uganda for a one-half reduction in soil
erosion per year.

19. d) Nutrition/Health Impacts
 More nutritious food has complex
impact pathways
 For micro-nutrients, can use RCT or IV
analysis to establish change in nutrient
consumption due to the intervention and
calculate disability-adjusted life years
 For macro-nutrients, combine results
from analysis of production and income
changes with demand system to project
consumption (and nutrient) changes

20. Example: Biofortified Cassava
 Nguema et al (2011)
Tj = total number of people in target group j
Mj = mortality rate associated with the deficiency in target group j
Lj = average remaining life expectancy for target group j
Iij = incidence rate of disease i in target group j
Dij = disability weight for disease i in target group j
dij = duration of disease i in target group j (for permanent
diseases dij equals the average remaining life expectancy Lj)
r = discount rate for future life years

21. DALYs lost to Vitamin A deficiency in
Nigeria and DALYs saved by bio-fortified
cassava

22. e) Risk/Resilence Impacts
 Important due to climate change effects on poor
 Benefits from reduction in yield variance
Kostandini et al (2011):
B/Y0= .5R (Y0) (σ2
Y0 - σ2
Y1)
 where B is the money value of reduction in income variation,
 R is coefficient of relative risk aversion
 Y0 is the mean of the income distribution before the technology
 Y1 is the mean after the new technology
 σ2
Y0 is CV squared for income distribution before the new
technology and σ2
Y1 is CV squared for the income distribution
after the new technology.

23. Example
 Kostandini et al., (2009) found the ex
ante benefits of drought-tolerance
research on cereals in eight African
countries to total more than $1 billion
per year with almost half of the
benefits due to yield variance
reduction

24. f) Gender Impacts
 Few quantitative assessments of gender
impacts of agricultural R&D
 Change in gender empowerment index
 Gender-disaggregated adoption analyses

25. 3. Addressing Aggregation
 Project, program, portfolio
 Field, farm, market
 Research Spillovers

26. Impact Matrix to Organize Data
and Methods to Aggregate up
Level for which impact
observed/assessed
Minimum
Data used
Type of
analysis/
model
Indicators Measured/Modeled
Outputs
Human Welfare Outcomes
Environment
Income Poverty
Nutrition/
health
International
National
Region/sub-sector/
ecosystem
Farm/Household/
Enterprise
Plot/Field/…

27. 4. Integrating impact assessment with
research data management
 In-house research impact assessment
capacity is important
 Key data for impact assessment are
often lost over time
 Need an IT system for entering and
storing data on inputs, yields, and other
traits from (1) near final trials, (2)
adoption surveys
 Used for internal and external assessments

28. Example
 Reviewing research data management
system at CIP and possibilities for
improving it for impact assessment
 Met with program leaders to discuss
major topics related to CIP strategic plan
 Identified candidates for assessment,
methods and data needs
 Reviewed current research data collection
by RIU and suggesting possible changes to
make it more useful in the future
 Undertaking impact case studies

29. Lessons
 Many research evaluation methods are
complementary in addressing multiple
objectives
 RCTs are unfortunately less useful for
assessing agricultural research impacts
than for other development interventions
 Tradeoff between cost and credibility of
impact assessment
 Need plan for collecting and managing data
from scientists to facilitate assessment