Presented by Jenifer Chapman at the October 2013 American Evaluation Association Annual Conference.

1. A Toolkit for
Evaluating the
Impact of HIV/AIDS
Programming on
Children in Africa
Jenifer Chapman, PhD
MEASURE Evaluation/Futures
Group

2. Stepping back for a minute
 HIV/AIDS Programming = President’s Plan for
AIDS RELIEF (PEFPAR)
 Interagency effort
 Introduced by President Bush in 2003
 Largest Commitment in history by any nation
to combat a single disease
 Reauthorized in 2008 for $48 billion

3. PEPFAR I and II
 First phase of PEPFAR (2003-2008) focus on
emergency response—program reviews, simple
evaluations, operations research
 Second phase of PEPFAR (2009-2013) focus on
sustainability, better attribution, program
outcomes and impact

4. PEPFAR Mission & Goal (OVC)
 Mission: To mitigate the social, emotional and
economic impacts of HIV/AIDS on children and to
reduce their risk and vulnerability while
increasing their resilience
 A goal: To care for 5 million orphans and
vulnerable children
 10% funding earmark for OVC

5. What is an “OVC”?
 A child infected or affected by HIV
 And what does that really mean?
 PEPFAR 1 vs. PEPFAR 2
And how do we “care” for them?

6. Guidance!
 6+1 areas of
programming
 Health, Education,
PSS, Protection,
Food & Nutrition,
Shelter & Care, HES

7. PEPFAR’s
approach to
OVC
programming

8. OVC Programs, under PEPFAR, aim to:
 Improve the wellbeing of children and families
 Strengthen families as primary caregivers of children
 Support the capacity of communities to create
protective, caring environments
 Build the capacity of social service systems to protect
the most vulnerable

9. Implementation
 Community-based programs
 Focus: case management (social work)
 Home-visiting by community based volunteers
(beneficiaries are targeted)
 Community-wide interventions (HES, Kids Clubs)
 6+1 domains of programming (health, PSS, etc.)
 Referral networks

10. Evaluating PEPFAR’s contribution
 Reviews of the OVC portfolio found that despite
great investment, we know little about impact and
“what works”

11. Not so long ago…
 Few evaluations conducted at program level
 Outcomes monitoring at case management level
(good, but led to problems in evaluation practice)
 Studies carried out were challenged:
 Tool misuse, low data quality, inadequate
measurement and poor choice of constructs, no
comparability of measures, unethical practices
 Poor availability of information for decision making
The State of Evaluation Practice was:

12. A proposed solution
Standardized questionnaires for use in a survey of
children ages 0-17 years and their adult caregivers

13. The purpose
 Standardize population-level child and caregiver wellbeing data beyond what is available from routine
surveys
 Produce actionable data to inform programs and
enable mid-course corrections
 Enable comparative assessments of child and
caregiver well-being and household economic status
across a diverse set of interventions and regions

14. But what do we measure?
 Developing country
context

16. Focus on PEPFAR OVC programs
 Indicators need to reflect
& be amenable to change
by PEPFAR program
intervention
 HH interventions led by
home visitors
 Community interventions
 Low direct funding per
target, focus on linkages
 Often inadequate
services in vicinity

17. Who are these tools for?
 Local and international
research institutions
and other implementing
organizations with
evaluation agenda
 USAID Forward –
supporting local
researchers

18. So, where to start…
 Two step, participatory process:
 Build consensus around core impact indicators for
PEPFAR-funded OVC programs
 Develop OVC program evaluation (survey) tools

19. Distilling the core indicators
 Our starting point: 6+1 domains of PEPFAR OVC
programming
 Step 1: Extensive literature search
 Step 2: Gaps (HES, PSS) filled through targeted
research
 Result: >600 child/HH wellbeing
questions/indicators
 Step 3: Analysis against 8 criteria
 Result: shorter list of questions for discussion
19

20. Inclusion criteria
1. Measures impact/outcomes
2. Amenable to change from program interventions
3. Relevant across a wide range of interventions
4.
5.
6.
7.
8.
20
Contributes to a holistic vision of child wellbeing
Verifiable through another source
Easy to implement
Relevant across different regions / countries
Relevant or easily adapted across age and sex

21. Finalizing the core indicators
 External working group: solicited review from 49
stakeholders
 Finalized core set of 12 child and 3 household
measures
21

22. Developing the tools
 Tools drafted with
strong stakeholder input
 Draft tools piloted in
Zambia and Nigeria
 Cognitive interviews to
test key concepts (e.g.
social support)
 Household pre-test of
full tools, procedures

23. Structure and content
1. Caregiver questionnaire (including questions on
household)
2. Child questionnaire (ages 0-9 years),
administered to caregiver
3. Child questionnaire (ages 10-17), administered
to child with parental consent & child assent

26. Caregiver questionnaire
Sections
Core questions
Section 1: Household
schedule
•
•
Household schedule* (10)
Changes in household
composition (4)
Section 2: Background
Information on Caregiver
and Household
•
•
•
•
Demographic information* (7)
Work* (3)
Access to money (3)
Shelter (1)
•
Section 3: Food Security
•
Household food security (6)
•
Section 4: Caregiver Wellbeing and Attitudes
•
•
•
General health (2)
Caregiver support (4)
Parental self-efficacy (1)
•
Basic HIV/AIDS knowledge* (7)
HIV testing* (3)
Attitudes to condom educ (1)
•
Section 5: HIV/AIDS Testing, •
Knowledge, Attitudes
•
•
Section 6: Access to HIV
Prevention, Care & Support
*DHS, bold=core indicator
•
Household access to services
(1)
Optional modules
•
•
Household Economic Status
(forthcoming)
Progress out of Poverty
Index or similar (country
specific)
Dietary Diversity (1)
Perceptions and experience
of child discipline, violent
discipline (forthcoming)
Gender roles and
decisionmaking power* (9)
HIV/AIDS attitudes* (4)

27. Child questionnaire (ages 0-9)
Sections
Core questions
Optional modules
Section 1: Child Health and
Welfare
•
•
•
•
•
•
•
•
•
Confirm demographics (5)
General health & disability (4)
Birth certificate (2)
Vaccinations (11)
Fever (<5 years)* (1)
Diarrhea (<5 years)* (1)
Experience of neglect (2)
Slept under mosquito net* (1)
HIV testing experience* (2)
•
•
•
Section 2: Education and
Work
•
Section 3: Food
Consumption
Section 4: Access to HIV
Prevention, Care & Support
•
School attendance*,
progression/repeats, drop-outs,
missed school days (5+ years) (9)
Work for wages (2)
Early childhood stimulation (2)
Food consumption (2+ years) (8)
•
Child access to services (1)
Section 5: Anthropometric
Measures (of Children)
•
Weight*, Height*, MUAC
•
•
*DHS, bold=core indicator
•
Fever: extended* (4)
Diarrhea: extended* (3)
Health for children
living with HIV/AIDS
(forthcoming)
Dietary diversity (1)

28. Child questionnaire (ages 10-17)
Sections
Core questions
Section 1: Background
Information on Child
Section 2: Diary
Section 3: Education
•
•
•
•
Section 4: Chores & Work
•
•
•
•
•
•
•
Confirm demographics* (5)
Identity of caregiver (1)
Daily log (6)
School attendance*,
progression/repeats, dropouts (9)
Chores (3)
Work (7)
Food consumption (8)
Alcohol consumption (3)
Birth certificate (2)
General health & disability (3)
General support (4)
•
•
Basic HIV/AIDS knowledge* (7) •
HIV testing * (3)
•
Section 8: Sexual Experience
Section 9: Access to HIV
•
Prevention, Care & Support
Section 10: Anthropometric
•
Measures: Weight and Height
•
Section 5: Food & Alcohol
Consumption
Section 6: Health, Support &
Protection
Section 7: HIV Testing,
Knowledge, and Attitudes
Child access to services (1)
Weight, Height, MUAC
Optional modules
•
Dietary diversity (1)
•
Health for children living with
HIV/AIDS (forthcoming)
Perceptions/experience of
violence (forthcoming)
Child development knowledge (6)
HIV/AIDS attitudes and beliefs (4)
•
Sexual behavior (13-17 yrs) (5)

29. Lessons from piloting
 Access to money
 Social support (and PSS generally)
 The importance of validating translations and
pre-testing
 Addressing child headed households

30. You said a toolkit?
 Tools & Manual
 Template protocol with
consent/assent forms
 Methodological guidance
 Data analysis guide
 Data collector training
manual and materials

31. Now….
 Evaluation and data use are high on the agenda
 We have standardized tools/measures and
guidance for evaluating OVC
programs/interventions with global buy-in
 Tools misuse is being corrected
 More information available for decision making
The State of Evaluation Practice is:

32. Where can I find out more?
Go to our website:
http://www.cpc.unc.edu/measure/
our-work/ovc
Keep in touch on ChildStatusNet:
http://childstatus.net/
Email:
Jenifer Chapman:
jchapman@futuresgroup.com
Janet Shriberg: jshriberg@usaid.gov

33. The research presented here has been supported by the
President’s Emergency Plan for AIDS Relief (PEPFAR)
through the United States Agency for International
Development (USAID) under the terms of MEASURE
Evaluation cooperative agreement GHA-A-00-08-0000300. Views expressed are not necessarily those of
PEPFAR, USAID or the United States government.
MEASURE Evaluation is implemented by the Carolina
Population Center at the University of North Carolina at
Chapel Hill in partnership with Futures Group, ICF
International, John Snow, Inc., Management Sciences for
Health, and Tulane University.

34. Questions?