The document discusses methods for integrating impact evaluations of social protection programs with administrative data and household surveys. It provides three examples: 1) Brazil's Bolsa Familia program matched survey and administrative records to assess program impacts on education with similar results from both data sources. 2) Yemen's social welfare program used matching to divide beneficiaries and evaluate impacts on expenditures, income, and child labor. 3) Spatial matching of administrative and survey data in multiple cross-section and panel surveys can identify treatment and comparison groups to evaluate social programs.

1. Impact Evaluation
of Social Protection
Programs:
household surveys
and administrative
data
www.ipc-undp.org
Rio de Janeiro, 04/10/2014

2. 2
Introduction
The basic question of an Impact Evaluation of a
Social Protection Program: “Have the
Program made any difference in the life of the
participants?”
As easy it is to ask as hard it is to answer.

3. 3
Introdu ction
1. It implies the reformulation as: What would
have happened to the participants’ life it they
were not in the program?
2. It implies a causal relationship between the
programs and the outcomes
3. The question is usually asked after the
program had been implemented
4. It requires a very sound methodological
design
5. It is a lengthy usually 2 years;
6. It is costly (USD 200 thousand to over 2
million Grosh et al)

4. 4
Introduction
The integration of the impact evaluation survey with
administrative data and the national household
surveys enhances the reliability of the results and
lower the costs of the evaluation.
Three examples:
1. The impact evaluation of the Bolsa Familia (AIBF in
Brazil)- phase one.
2. The National Social Protection Monitoring (NSPMS
in Yemen)
3. The spatial matching of administrative records
with national household surveys)

5. 5
AIBF Phase 1 (Brazil)
Year 2005
Sample Size: 15, 426 households 65,000 individuals
Design:
•In the 41 largest municipalities the Primary Sampling
Unit (PSU) was the Census Tract
•The other municipalities were aggregated to have a
minimum size of 50,000, they constitute the PSU and
the census tract was the Secondary Sampling Unit
(SSU)
•Census tracts were screened to identify families
beneficiaries, families non beneficiaries but registered
in the registry system (CadUnico) and those who were
neither beneficiaries or registered, composed as 3:6:1

6. 6
AIBF Phase 1 (Brazil)
The individual and household records were
linked to the CadUnico records with the
objective of:
•Correct the information of the NIS of the
survey
•Assess the quality of self reported groups of
study
•Compare the effect on education on the
survey and the matched groups
•Compare the results on education of
Propensity Score Matching and Regression-
Discontinuity (Sharp) methods

7. 7
AIBF Phase 1 (Brazil)
The matching of the AIBF survey and
Administrative Data (CadUnico) combined
the Probabilistic and Deterministic
methods of record linkage
Information from CadUnico used were:
Full Name, Municipality of Residence,
Birth Date and Sex

8. 8
AIBF Phase 1 (Brazil)
A sketchy list of activities:
1. Cleaning and standardization of Data Sources
2. Linkage by deterministic method
3. Classification of the record: Matched and Non Matched.
(73.8% of NIS and 35% of families were matched)
4. For the Non Matched : Blocks of Beneficiaries and Non
Beneficiaries in the Survey and Soundex of the first name +
soundex of the last name + municipality + sex
5. Classification Total matching /Partial Matching
6. Manual Revision
7. Final Classification 73.5% of individuals 30% of families
8. Reallocation of the Treatment and Comparison Groups
9. Education: attendance, dropouts, progression, working,
non progression by PSM and RD Sharp

9. 9
AIBF Phase 1 (Brazil)
The overall results of effects on education did not
differ greatly in the survey and in the matched
databases by the PSM method, both indicating
the same signals in the differences on drop outs,
progressions, repetitions and out-schools
children.
The RD-Sharp the utilized the income declared in
the CadUnico revealed more conservative results
than the PSM.
Source
Racchumi Romero, J. R. Utilizando o
Relacionamento de Bases de Dados para
Avaliação de Políticas Públicas: uma aplicação
para o Programa Bolsa Família. Tese de
doutorado, Cedeplar, UFMG, 2008

10. 10
NSPMS-Yemen
•A Longitudinal Quarterly Panel Survey
to monitor socio-economic indicators and
assess the targeting and possible
impacts of the Social Welfare Fund (SWF)
•Year October 2012- September 2013
•Sample Size: 6,397 balanced size out of
7,152 households in the first round.

11. 11
NSPMS-Yemen
Design:
•Census Enumeration Areas (EA) are the PSU
stratified by Governorates. 30 EA in each
Governorate.
•In each EA, 12 households were selected
using a stratified simple random sampling
procedure. Households were selected from the
three groups identified in the listing of the
household in the selected enumeration areas,
namely, beneficiary of Social Welfare Funds
(SWF), potential beneficiary of SWF
(registered, but not receiving it yet), and non-beneficiary
and non-registered.
•Each household was interviewed four times

12. 12
NSPMS-Yemen
The SFW
•The SWF expansion was due to the incorporation of
new beneficiaries into the programme. New
beneficiaries were identified in the 2008 Comprehensive
Social Survey (CSS) and selected through a proxy
means testing (PMT), but were only systematically
incorporated into the programme from October 2012
onwards.
•New beneficiaries correspond to about 33 per cent of
the total number of beneficiary households.
•Some new beneficiaries received their first payment in
the first quarter of 2011, after that, payments were
suspended and only resumed in the last quarter of 2012.
A lump sum payment varying from YER 30,000 to YER
60,000 corresponding to the 5 quarters in arrears was
paid to them

13. 13
NSPMS-Yemen
The matching with administrative date were
necessary to:
•To assess the distribution of the different
categories of SWF beneficiaries.
•To divide the sample of beneficiaries into old
beneficiary (pre-2008) and new beneficiary (post-
2008) since they were selected differently. Only the
latter was chosen based on a PMT. It was necessary
to use the administrative data because almost 50%
of the sample replied they did not know when they
had started receiving the SWF benefit. Thus, one
could not classify the two groups (new and old
beneficiary) based on the survey information.

14. 14
NSPMS-Yemen
Matching of the survey info and SWF administrative
information was based on “number of SWF card”,
name of beneficiary, and when name did not match
on other characteristics of the main beneficiary.
734 SWF beneficiaries in the NSPMS sample that
were not matched with the SWF administrative
database).
New parameters: the total amount of SWF transfers
received during round 1 (October-December 2012)
and the self-reported year of accreditation into the
programme.
This procedure yield similar estimates of new
beneficiaries in both admin data and NSPMS sample:
33%, out of 1,5 million total beneficiaries.

15. 15
NSPMS-Yemen
Some results
•The propensity score estimates confirmed that new
SWF beneficiaries were more likely to be poor (as
identified by the PMT) and have higher predicted
probabilities of SWF receipt than the comparison
group members.
•As for expenditures on food, we find that all of the
estimated effects are positive, and most are also
statistically significant, particularly, for old SWF
beneficiaries.
•As for household income and agricultural
production we find that income from work and from
agricultural production are both significantly
reduced among the old SWF beneficiary households.

16. 16
NSPMS-Yemen
• New SWF beneficiaries are more likely to make
investments in agricultural inputs and, they are
also significantly more likely to possess livestock
than non-beneficiaries.
• Higher rates of child labour and unpaid family
work for female SWF new beneficiaries ages 6-11
(compared to non beneficiaries) while school
• Reductions in the probability that both male and
female children of younger (6-11) and older (12-
14) ages were absent from school
• Higher rates of unpaid family work for males 6-11
and 12-14 years (also new beneficiaries).
Source: Veras, Fabio et al. National Social Protection
Monitorin Survey. IPC-IG/UNDP UNICEF-Yemen,
2014

17. 17
NSPMS-Yemen
• New SWF beneficiaries are more likely to make
investments in agricultural inputs and, they are
also significantly more likely to possess livestock
than non-beneficiaries.
• Higher rates of child labour and unpaid family
work for female SWF new beneficiaries ages 6-11
(compared to non beneficiaries) while school
• Reductions in the probability that both male and
female children of younger (6-11) and older (12-
14) ages were absent from school
• Higher rates of unpaid family work for males 6-11
and 12-14 years (also new beneficiaries).
Source: Veras, Fabio et al. National Social Protection
Monitorin Survey. IPC-IG/UNDP UNICEF-Yemen,
2014

18. 18
Spatial Matching of Administrative
Records with National Household Survey
Multiple Cross Section Surveys
In most of those surveys, the Primary
Sampling Units (PSU) of the multiple cross
section surveys are constant throughout the
years.
The spatial matching of the (future)
beneficiaries with the PSU, using the address
of the future beneficiaries. Matching procedure
will define the comparison group in the same
PSU.
Additional questions on participation in the
Social Programme should be added to the
National Survey.

19. 19
Spatial Matching of Administrative
Records with National Household Survey
National Panel Surveys:
The matching of (future) beneficiaries
with the Panel subjects. The beneficiaries
will constitute a sub-sample of the of the
panel subjects.
Matching procedures, among the Panel
subjects, such as PSM will define the
comparison group at the baseline survey
(for the evaluation purpose). The
frequency of the interviews will follow the
same schedule of the larger project.

20. 20
Thank you!
Obrigado pela atenção
Diana Sawyer
Diana.sawyer@ipc-undp.org