Improving market access of farmer groups in Uganda: evaluating the role of working capital

Impact evaluation An application to farmers groups in Uganda

Improving market access of farmer groups in
Uganda: evaluating the role of working capital

Ruth Vargas Hill and Eduardo Maruyama

May 9, 2012


Outline

Impact evaluation
Introduction
RCTs

An application to farmers groups in Uganda
Introduction
Implementation
Results
Concluding remarks


Why evaluate?

• Evaluating interventions (policies or programs) helps:
• Understand the actual rather than the anticipated eﬀects of
programs.
• Determine how to design new programs.
• Determine the most cost-eﬀective approach to achieve a
desired goal.


Estimating impact: introduction

• When we conduct impact evaluation we assess how a program
aﬀects the well-being or welfare of individuals, households or
communities:
• Proﬁtability of agricultural production
• Increased income or consumption (or other measures of
welfare) of rural households
• Poverty levels or growth rates at the community level


Impact evaluation versus other M&E tools

• Impact evaluation is different from other M&E tools in that it
focuses on discerning the impact of the program from all
other confounding effects.
• The focus of impact evaluation is providing evidence of the
causal link between an intervention and an outcome.
• This is why impact evaluation is a powerful too, but also what
makes it difficult to implement in practice.


Impact evaluation versus other M&E tools

DIFFICULTY OF
Low High
SHOWING CAUSALITY

Inputs Outputs Outcomes Impacts

Example: A program of providing advice on a new technology to farmers

Visits by
Increased
extensions
Knowledge of Use of the yields, higher
agents,
the new new farm profits,
physical inputs
technology technology improved
(such as
consumption
seeds)


Essential component: counterfactual

• Diﬃculty is determining what would have happened to the
individuals or communities of interest in absence of the
project.
• We are interested in the diﬀerence in an outcome for an
individual with and without the intervention.
• Problem: can only observe people in one state of the world at
one time
• The key component to an impact evaluation is to construct a
suitable comparison group to proxy for the counterfactual.


Before and after comparisons

• Why not collect data on individuals before and after
intervention (the Reflexive)? Difference in income, etc, would
be due to project
• Problem: many things change over time, including the project
• The country is growing and profits are rising. Is this due to the
program or would have occurred in absence of program?
• This is particularly a problem for agricultural interventions:
many factors affect yield (weather, availability of inputs) and
prices in a given year.


Comparison groups

• Instead of using before/after comparisons, we need to use
comparison groups to proxy for the counterfactual
• Two core problems in ﬁnding suitable groups:
• Programs are targeted
• Recipients receive intervention for particular reason
• Participation is voluntary
• Individuals who participate diﬀer in observable and
unobservable ways (selection bias)
• Hence, a comparison of participants and an arbitrary group of
non-participants can lead to misleading or incorrect results


Randomizing to create a true comparison group

• We need a comparison group that is as identical in observable
and unobservable dimensions as possible, to those receiving
the program, and a comparison group that will not receive
spillover beneﬁts.
• Number of techniques:
• Randomized control trials (the gold standard)
• Careful matching techniques: IV, propensity score matching,
regression discontinuity design


Randomized Control Trials (RCTs)

• In RCTs, participation in a policy (or usually eligibility to
participate in a policy) is randomly assigned.
• This is done to ensure that the only diﬀerence between those
in and out of an intervention, is their participation, and as a
result any diﬀerence between participants and
non-participants can be attributed to the program alone.
• Because participation (treatment) is randomized, the
non-treatment outcomes between those that are not treated
and those that are treated is equal.


Households or groups of households


Households or groups of households

C
T C
C
C
T T C
C T C
T
C
T C
T
C T
C C C
C

T T
T C T


Does randomization create a true comparison group?

• We can test that they are equal by collecting data on the two
groups before the intervention and checking that the average
characteristics of the two groups are the same.
• For the treatment and control groups to be statistically equal
you need a large number of each. Cannot have one treated
household and one control household.
• Means that you cannot use this method to answer questions
about country policy changes (e.g. ﬁscal policy changes).
• There are stragegies that can be used to ensure that the
treatment and control groups are equal (e.g. stratiﬁcation).


How do we estimate impact by randomizing?

• Identify the outcome we are interested in (e.g. yields, amount
of output marketed, price received)
• Estimate the average of the outcome in the treatment group.
• Estimate the average of the outcome in the control group.
• Calculate the difference of these averages and test to see if
the two averages are significantly different from each other.
• Average Treatment Effect
• Note: it is just differences in the AVERAGE outcome that are
estimated.


Challenges to estimating impact

• Sometimes the effect of the program is small.
• Or there are many other factors affecting the outcome of
interest that it is hard to see if a difference is statistically
different between two groups.
• We try and control for this in two ways:
• Include a large number of households in treatment and control.
This increases our power to detect a small effect.
• Collect data on characteristics of the household that may
influence the outcome variable at baseline (including the
pre-intervention outcome of interest)


Other challenges to estimating impact

• Are we sure that the intervention had no impact on the
control group? Are there no spillover eﬀects? (E.g. on prices)
• Was there any attrition as a result of the program that means
we miss capturing some of the impact? For example did
people migrate as a result of the program? If so, we will miss
capturing the eﬀect of the program on these people.
• We randomized to avoid selection bias, but some of it still
may remain:
• Did everyone in the treatment group participate as expected?
• Did anyone in the control group participate even if they were
not meant to?


Selection bias

Not in
evaluation

Target
Population

Treatment Participants
group
Evaluation Random No-Shows
Sample Assignment
Non-
Control group Participants
Cross-overs

36


Handling selection bias

• Intent to treat (ITT):
• Average impact of program in practice: treats all noncompliars
as treated, and treats all crossovers as remaining in the control
• Problem: power is reduced by noncompliance and does not
provide an idea of what the average impact of the program on
the treated is.
• Treatment on the treated (ToT):
• Instruments for take-up with assignment: gives an idea of the
average impact of the program for a speciﬁc group


Summary of advantages and disadvantages

• Powerful method to identify causal impact of a policy or
program.
• Careful design is needed to ensure you are able to detect
changes
• Can be expensive: baseline and follow-up, a large number of
participants (especially if it is to be representative)
• Only valid in some circumstances: randomization over a
number of units needs to be possible.
• Provides information on the average outcome.
• Internally valid, repetition and a theory of change needed to
make predictions from results (external validity).


External validity
• Tells us whether something worked in a specific context,
understanding whether an intervention would work again in a
different setting, external validity, is very difficult to know.
• Great benefit of few assumptions, comes with great cost
”narrowness of scope” (Cartwright 2007).
• Any external validity involves some assumptions about the
project working in different conditions.
• Yet replicating a project is almost impossible, replicating
triggers of mechanisms that produce the change is often more
possible.
• Result: we need to repeat impact evaluations and also have a
theory of change to know what will work in the future, i.e. to
really learn.
• But better than an approach that is not internally valid.
Predictions cannot be made from this either.


Other concerns in learning from results

• General equilibrium eﬀects.
• Corruption in implementing a large scale.
• Capacity to implement at a large scale.
• Overlap between new environment and old (example of
medicine)

• Not automatic to go from experiments to learning and policy
advice.


Another approach

• We need a theory of change that guides us in going from
one-off impact evaluation to general lessons. This means
make assumptions and, ideally, use experiments to test and
refine these assumptions.
• When designing impact evaluation for this purpose, it often
looks quite different:
• Theory of change influences the design of the impact
evaluation.
• Often identifying the differential impact of different treatments,
rather than the impact of one treatment against baseline.


Introduction

• Smallholder agriculture in Sub-Saharan Africa is largely
exposed to pervasive market failures, translating into missed
opportunities and sub-optimal economic behavior.
• These failures are often rooted in the importance of
economies of scale in procuring inputs and marketing produce.
• By engaging in markets collectively through a farmers group,
smallholders can overcome economies of scale.
• Despite the renewed interest from governments and donor
agencies in farmers groups as a means to overcome these
market failures, evidence shows that they have so far had
limited success.


Ugandan context

• The majority of Ugandan farmers sell their (unprocessed)
produce at harvest time to itinerant traders at the farm-gate.
• Survey of farmers groups engaged in some form of output
marketing revealed that:
• Farmers get a higher price when they sell collectively.
• Yet few farmers sell through the marketing group of which
they are a member (only 47% make sales through group)
• Farmers are less likely to sell collectively when they are
liquidity constrained and in need of emergency money.
• Groups that oﬀer cash on delivery of produce (rather than
payment some days later) have a higher proportion of
members selling through the group.


Key impact question

• Would providing working capital loans to farmers groups so
that they can provide cash on delivery, improve marketing
outcomes for farmers?

• We cannot infer this from the baseline data: good groups may
be better at collective sales and better able to access ﬁnance
which allows payment on delivery.
• We would like to compare groups of similar quality and see if
working capital loans increase sales amongst those that
received them.


Testing a theory of change
• Farmer groups can offer higher prices but because of the
waiting times involved in receiving payment, farmers find it
costly to sell though the group.
• Farmers are liquidity constrained and often sell coffee to meet
urgent financial needs, so even small delays in payment can be
problematic.
• Waiting for payment involves a high level of trust in the ability
of the group to market and transparency. There is a risk if the
groups cannot be trusted.
• Enabling groups to make payment on delivery through a
working capital loan will reduce the cost of selling through the
group.
• More farmers will sell through the group and receive higher
prices as a result.


The impact of working capital credit
• Randomized provision of working capital credit to farmers
groups that had already been engaged in output marketing:
• Provide selected groups with a fund to make partial cash
payments to farmers upon delivery of produce. Once the group
makes a sale the fund is replenished and farmers are given the
remaining balance. “Cash on Delivery” (CoD)
• Assess the impact of this credit on the proportion of produce
sold through the group and on the price farmers received.
• Understand why this worked?
• Did this work for farmers likely to face liquidity constraints, or
only in groups where trust was already high?
• Implement an intervention on improved transparency to
randomly selected groups to improve trust in some groups. Is
the working capital intervention just as effective in those
groups with the transparency intervention?
• Information on Sales (IoS): SMS system to provide members
with specific information about transactions made by the
group (final sale price, fees deducted, etc.), plus reinforced
training on book-keeping.


Coﬀee/maize group marketing structure

• Farmers groups (“PO”s for producer organizations) are
typically grouped under associations (DCs for district
committees).
• The PO handles bulking and coordination of transport with
members at the village level.
• The DC take care of collection and in some cases value
addition to the next stage of marketing.
• In most cases, a service organization oﬀers support to DCs
and POs through lobbying, access to extension and additional
marketing services.


Implementation
• The study was carried out in 9 DCs marketing coﬀee and
maize, containing 165 POs under them.
• March 2010, Baseline survey:
• A 3-tiered survey which collected detailed information on DCs,
POs, and member households.
• Full roster of members for each PO, and a complete household
survey for at least 2 members of each group.
• November 2010 September 2011, implemented intervention
in randomly selected groups:
• Provided working capital credit to randomly selected POs.
• Provided SMS information on deliveries to randomly selected
POs.
• October 2011, Follow-up survey.
• Collected detailed information on POs and member households.
• Collected administrative data from the DC records to obtain
more reliable delivery data.


Randomization strategy

• We randomized the interventions at the PO level stratifying
the sample by DC, since the sample size is not large enough
at the DC level and the risk of spill-overs is too high at the
household level.
• POs in each DC are randomly assigned into 4 groups: (1)
CoD, (2) IoS, (3) CoD + IoS, and (4) none.
• The fund for the CoD was managed by the DC, and vouchers
were given to treated POs so their members could request
immediate partial payments for output deliveries.
• For the IoS intervention, a DC staﬀ member was selected to
send the messages to key farmers in the treated POs.


Are control and treatment groups equal?

Control CoD IoS Both
(mean)
Members 24.256 2.194 4.558 1.597
(3.106)∗∗∗ (4.365) (4.289) (4.339)
Years since foundation 4.400 1.039 0.460 1.014
(0.526)∗∗∗ (0.739) (0.731) (0.739)
Marketing services 0.825 -0.020 -0.081 -0.093
(0.066)∗∗∗ (0.093) (0.092) (0.093)
Output bulked (kgs.) 854.025 -240.708 -192.862 -325.440
(236.818)∗∗∗ (332.863) (329.018) (332.863)
Female leader 0.250 -0.006 -0.064 -0.030
(0.067)∗∗∗ (0.094) (0.093) (0.094)
Leader’s age 52.200 -4.639 -2.153 0.190
(1.874)∗∗∗ (2.634)∗ (2.604) (2.634)
Leader’s schooling 8.025 0.073 0.208 -0.562
(0.460)∗∗∗ (0.647) (0.639) (0.647)
POs 40 41 43 41


• Implementing the interventions represented a major challenge:

1. The POs in our study are spread over many regions in the
country.
2. Implementation needed to be done by a 3rd party, to avoid
service organizations and DCs contaminating the PO-level
randomization strategy.
3. In order to avoid undesired heterogeneity in implementation,
training, and monitoring of the interventions, a single
implementing agency was favored over several regional
organizations.
4. Training and distribution of vouchers within the PO was
delegated to PO leaders in some DCs.
• Our own monitoring activities as well as the follow-up survey
indicate implementation was problematic.
• Some cross-over and no-shows for CoD intervention
• Overall implementation of IoS intervention.


Empirical strategy

McKenzie (2011) shows that using baseline data on the outcome
variable of interest, allows more power to detect impact.
Therefore, for our analysis we estimate:

Yi,1 = α + γj Di,j + θYi,0 + εi,1
j


Results

Table 2.1: Impact of interventions on produce
deliveries

PO Household
Kgs. P(Delivery) Kgs.
CoD only 747.826 0.186 162.700
(325.294)∗∗ (0.079)∗∗ (88.400)∗
IoS only 355.764 0.089 62.660
(320.386) (0.078) (87.060)
Both -584.566 0.101 122.000
(455.073) (0.077) (86.280)

Observations 165 244 243
R2 0.422 0.269 0.084


Results

Table 2.2: Impact of selling through PO
on transaction features

Price Days
between
sale and
payment
Sold through PO 0.858 -6.540
(instrumented) (0.477)∗ (21.170)
Observations 193 192
R2 0.704 0.210


Concluding remarks

• Despite implementation problems, the CoD intervention has a
signiﬁcant impact on group marketing.
• CoD increases the probability a household will sell through the
group, how much each household will sell, and the total
amount sold by the group.
• By encouraging farmers to sell through the group, CoD has an
eﬀect on increasing the price they receive.
•

Improving market access of farmer groups in Uganda: evaluating the role of working capital

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Improving market access of farmer groups in Uganda: evaluating the role of working capital

Ähnlich wie Improving market access of farmer groups in Uganda: evaluating the role of working capital (20)

Mehr von IFAD International Fund for Agricultural Development

Mehr von IFAD International Fund for Agricultural Development (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Improving market access of farmer groups in Uganda: evaluating the role of working capital