Gustavo Angeles' (UNC) presentation at the 2019 Global Development Conference, October 23-25, 2019.

1. The Transfer Project – Some lessons on making
evaluations useful for policy action
Gustavo Angeles,
University of North Carolina at Chapel Hill (UNC)
On behalf of
the Transfer Project Evaluation Team
2019 Global Development Conference
Bonn, October 23, 2019

2. The Transfer Project: Background & Motivation
 Mid-2000’s, growing momentum for social protection in sub-Saharan
Africa
 Cash transfers (CT) in a few places but skepticism and concerns
 Latin American CT evaluations known but African programs different:
 Unconditional; different supply conditions and institutional context
 Limited administrative capacity
 So, large evidence gap in Africa at a time of increasing demand for
evidence
 Response: In 2009, Transfer Project created as a collaboration
between UNICEF, FAO and UNC to generate the needed evidence.

3. The Transfer Project: Background & Motivation
Key defining features:
 Evaluate ongoing national government CT programs
 Address 3+3 main criticisms of impact evaluations:
 They don’t answer the questions relevant for policy makers
 Results not available at needed time
 Presented in format and language too technical
 Limited learning component beyond impact
 Distant from the policy and program processes
 Limited involvement of local researchers.

4. 4
Transfer Project:
- Conducted evaluations in 10
countries
- This book documents
experience across eight
countries
- Describes evaluation process
and results
- How was evidence used?
- What were key features that
led to research uptake?
- Chapters written jointly by
evaluators and program
managers.

5. https://transfer.cpc.unc.edu/making-research-count-lessons-on-turning-evidence-into-action-from-the-transfer-project/
Read the blog, if you are convinced then
read the book… (350+ pages)

6. Key ‘ingredients’ to research uptake
• Relationship between evaluator and program implementers
• Build trust, same team, same objectives
• Close partnership with govt and managers to understand and
respond to their concerns
• No “arm’s length” evaluation
• Understand the stakeholders’ motivation for the evaluation and
the political economy of the program
• Tailor the evaluation according to the evidence needs of
stakeholders
• This leads you beyond standard IE of “1 program and 1 outcome”:
Added local economy-wide impacts, productive activities; cost;
examine “expected” negative behaviors

7. Key ‘ingredients’ to research uptake
• Managers interested in program design/operations,
policy makers can’t wait 2-3 years: Embed impact evaluation
into broader learning agenda
• Many questions on program design/operations don’t require IE
• Make baseline surveys useful:
• Targeting analysis
• Cash transfer size as share of household consumption
• Simulations of impact
• Qualitative assessments
• Barriers to use of services
• Presence of other services

8. Key ‘ingredients’ to research uptake
• Partnership with national researchers
• Understand local context, sensitivities, research questions
• Readily available to present results when necessary
• Keep it as rigorous as reality permits
• Everybody values that; provides credibility
• But, balance it with political and operational constraints;
RCTs not always feasible. Needs consensus.

9. Key ‘ingredients’ to research uptake
• Use multiple channels of communication:
• 2-day workshops with national & provincial program managers,
Ministry and development partners on programmatic
implications
• Policy briefs
• Overall and specialized reports
• Newsletters
And, Transfer Project Conferences:
• 7th in Arusha in April 2019
• 130 delegates in 2019, up from 39 participants in 2010
• Country teams: government, program managers, local academics,
and development partners
• Exchanges on latest evidence and its use.

10. Ex. 1: Ghana LEAP Cash Transfer Program
• Initial impact evaluation 2010-2012
• Used baseline data to calculate transfer value as share
of consumption of beneficiaries
- Extremely low by international standards
- Simulations showed no/small positive impact
- High administrative cost relative to benefit
transferred
• Ministry used evidence to successfully triple value of
transfer
• Operational aspects of the programs became relevant

11. Transfer as Share of Participant Consumption:
LEAP level is very low by international standards
[Actual slide shown at baseline workshop, 2011]
7
15
17
20
22
25
27
29
0
5
10
15
20
25
30
35
Transfer Share

12. Endline evaluation showed limited impacts—a key
assumption in the log frame did not hold
0
1
2
3
4
5
6
Apr-10
May-10
Jun-10
Jul-10
Aug-10
Sep-10
Oct-10
Nov-10
Dec-10
Jan-11
Feb-11
Mar-11
Apr-11
May-11
Jun-11
Jul-11
Aug-11
Sep-11
Oct-11
Nov-11
Dec-11
Jan-12
Feb-12
Mar-12
Apr-12
#ofpayments
Ghana LEAP
Transfers were lumpy and irregular rather than predictable

13. Ex. 2: Zambia cash grants IEs
• Explicit request from Ministry for IE to examine
effects on dependency, hand-outs, laziness
• Strong economic/productive angle to IE design
• No work decline; increases in production,
• Strategic, repeated dissemination of these results
• Other ministries, Cabinet, Ministry of Finance
• Social protection week
• Research team estimated a ‘multiplier’ effect of 1.5
• Powerful, succinct rebuttal to dependency claims

15. Zambia: Cash transfer scale-up and evaluation timeline
GoZ budget contribution went from $5m to $35m in 2014 and $45m in 2015
0
50000
100000
150000
200000
250000
2003 2005 2007 2009 2011 2013 2015 2017
Households Reached by Cash Transfers in Zambia
CGP MCP
follow-ups
CGP MCP
baselines
Monze
baseline
Monze
follow-up
CGP MCP
follow-ups
GoZ take over
Election

16. 16

17. Ex. 3: Malawi Social Cash Transfer Programme (SCTP)
• Initial IE 2006-07 Mchinji pilot
• Led to expansion to 8 districts
• Stagnation 2008-2012
• GoM interest in Public Works (MASAF) and FISP
• Ministry/UNICEF kept citing Mchinji results, developed a
National Social Protection Policy (NSSP), ‘kept pedaling the
bicycle’
• SCTP take-off from 2012+
• Donors supported NSSP, more $ commitment from GoM
• New IE 2013-2016 as part of expansion

18. Malawi - Uptake of evidence from 2013-16 IE
• Baseline simulations on transfer value directly led to
increase in transfer size in 2015
• Strong positive impacts across the board
• Demonstrated Ministry’s ability to implement program,
seriousness and professionalism in using evidence
• Maintained donor enthusiasm and commitment
• Strong positive impacts coincided with ‘negative’ impact
results from MASAF public works
• World Bank (finally) took note of SCTP, diverted funds from
MASAF to SCTP
• SCTP scaled up to all districts now
• Limited effects on nutrition and morbidity
• Need to link beneficiaries with other programs

19. Over 95% of eligible households have received their
transfers*
95 95 95 95
99 99 97 97 97 95 95 97
0
10
20
30
40
50
60
70
80
90
100
I II III IV V VI VII VIII IX X XI XII
PercentageofeligibleHHreceivingtransfer
*: According to the transfers payment data.
Transfer
s
Midline Endlin
e

20. Malawi evolution of real value of transfers
• Amounts increased in May 2015 by about 55%
• However, about 60% inflation between Aug 2013 and May 2015
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
MalawiKwachas
Transfer: 4+ persons
Nominal: 2,400 MWK
3,700 MWK
Real
-37%
-54%
Baseline (Aug 2013) Midline (Dec 2014) Endline
(Nov 2015)

21. Summary:
Program is operating well, providing regular and predicable
transfers, but must protect real value of transfer.
Payday in
Salima: Photo
credit, Jacob de Hoop

22. Examples illustrate different paths to evidence
uptake
• Ghana: Actual IE results were null, but highlighted
key operational issues that led to change
• Zambia: Focus on addressing critics, strategic and
concerted dissemination efforts
• Malawi: Patience, long-run perspective; influencing
other development partners

23. Reflection on whether evidence has been used
to improve programs
• Yes, but it takes special effort; Transfer Project model
identifies some key ingredients:
• Implementers must trust evaluator, same team
• One-off IE not the best strategy – need broad learning agenda
• Understand the political debate
• Strategic packaging and dissemination
• National research counterparts to present evidence at key
moments
• Patience…need confluence of other factors, but must have
evidence available at the right moment
All this requires time, commitment, skills, patience and
funding.

24.  Transfer Project website: https://transfer.cpc.unc.edu/
 Facebook: https://www.facebook.com/TransferProject
 Twitter: @TransferProjct
For more information