This presentation summarizes a multi-agency impact evaluation of malaria interventions in mainland Tanzania from 1999 to present. It provides an overview of the evaluation, including the institutions and stakeholders involved, key indicators and data sources being used, and the evaluation design and timeline. Challenges discussed include linking intervention coverage trends to morbidity and mortality trends over time given changing policies and limited early data. The presentation concludes with lessons learned around the need for country ownership, complementary district-level analyses, and additional data sources.

1. GHI Presentation, March 22nd , 2012, 1:00pm-2:00pm
USAID/Washington DC
Multi-Agency Impact Evaluation
Challenges in Evaluating the Impact of the Scale-up
of Malaria Interventions: The Tanzania Impact
Evaluation
Yazoume Ye, MEASURE Evaluation/ICF International

2. Content of the Presentation
 Overview of the Multiagency Impact Evaluation
 Mainland Tanzania impact evaluation
o Scope
o Process and methodological approach
GHI Presentation, March 22nd , 2012, USAID/Washington DC

3. Overview of the Multiagency Impact
Evaluation

4. Rationale
 Substantial increase in
funding for malaria
control in the recent
decade
 Progressing scaled up of
key interventions
 What is the effect of the
scale-up of key
interventions on malaria
burden? Source: Donor report & Gvt of Tanzania budget
GHI Presentation, March 22nd , 2012, USAID/Washington DC

5. Institutions Involved
 PMI
o USAID - Washington, DC and Country RA
o CDC - Atlanta, GA and Country RA
 ICF International
o MEASURE Evaluation
o MEASURE DHS
 National Malaria Control Programs
 National Statistic Bureaus
 In country Research Institution
GHI Presentation, March 22nd , 2012, USAID/Washington DC

6. Evaluation Questions
 What impact have malaria control interventions had on
malaria-related morbidity and mortality?
 Can we demonstrate and quantify plausible association
between intervention and impact?
GHI Presentation, March 22nd , 2012, USAID/Washington DC

7. Key Indicators
Program-based Data Population-based Data
Inputs Processes Outputs Outcomes Impact
• Strategies • Human • Service • Intervention • Disease
• Policies resources delivery coverage burden
• Guidelines • Training • Knowledge, • Use of • Mortality
• Financing • Commodities skills, practice intervention
Focus
GHI Presentation, March 22nd , 2012, USAID/Washington DC

8. What is the Story?
Linking Morbidity and Mortality Change to Interventions
Malaria intervention coverage
Mortality trend scenario 1
Mortality trend scenario 2
1999 2010
GHI Presentation, March 22nd , 2012, USAID/Washington DC

9. Evaluation Scope
 Focus on impact evaluation, NOT program
effectiveness and efficiency
 Not restricted to PMI-funded malaria control
 Focus on period of malaria control scale-up
and change in outcomes (1999-present)
GHI Presentation, March 22nd , 2012, USAID/Washington DC

10. Achieving Collective Ownership
 Prior consultation:
o Common ownership of SINGLE impact evaluation report
o Workshop involved NMCP, RBM-MERG, WHO, GFATM, and
PMI
 Stakeholders review preliminary findings
 TAG: Comprised of RBM members and other international
experts review first draft
 Disseminate final report nationally and internationally
GHI Presentation, March 22nd , 2012, USAID/Washington DC

11. Evaluation Design
 Describe trends in intervention coverage
 Describe trends in morbidity and mortality
 Linking these trends
o “Plausibility” approach
o Temporal, spatial, age-pattern, “dose-response” associations
o Use models (LisT) to estimate number of deaths averted
 Multivariate models when possible with available data
GHI Presentation, March 22nd , 2012, USAID/Washington DC

12. Plausibility Framework
Indicators
• ITN Ownership
• Parasitemia All cause under-five
• ITN use
• Anaemia (<8g/dL) mortality (5q0)
• IPTp
• Treatment
Decreased
Increase in effective Decreased malaria-
intervention coverage morbidity associated
mortality
Contextual
• Health intervention • Socioeconomic
factors
• Climatic factor • Health care • Education
• Rainfall utilization • Fertility risk
• Temperature • ANC, EPI, Vit A, • Housing condition
PMTCT • Nutrition
GHI Presentation, March 22nd , 2012, USAID/Washington DC

13. Data Sources
 DHS, MIS, MICS surveys
 Other national/large scale surveys
 HMIS data
 HDSS (if any in the country) - case study
 Studies and reports
GHI Presentation, March 22nd , 2012, USAID/Washington DC

14. Evaluation Timeline
Angola GF PMI Nov Jun
Benin GF PMI Feb Feb
Ethiopia GF PMI Feb Feb
Ghana GF PMI Feb Feb
Kenya GF PMI
Liberia GF PMI
Madagascar GF PMI
GF PMI Nov Jun
Malawi
Mali GF PMI
Mozambique GF PMI Feb Feb
Rwanda GF PMI Feb Dec
Senegal GF PMI Feb Dec
Tanzania GF PMI Feb
Uganda GF PMI Feb Feb
Zambia GF PMI
Zanzibar Feb Dec
2003/04 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

15. Mainland Tanzania Impact Evaluation
The Tanzania Impact Evaluation Team
Alex Mwita Fabrizio Molteni Monica Olewe Rita Njau
Angelica Rugarabamu Frank Chaky Paul Smithson Rose Lusinde
Achuyt Bhattarai Fred Arnold Peter McElroy Salim Abdulla
Christine Hershey Honorathy Masanja Renata Mandike Yazoume Ye
Erin Eckert Lia Florey Rene Salgado

16. Evaluation Team
 National Malaria Control Program -Tanzania
 National Bureau of Statistics
 Ifakara Health Institute
 PMI
o USAID -Washington, DC and Tanzania
o CDC – Atlanta GA and Tanzania
 ICF International
o MEASURE Evaluation
o MEASURE DHS
GHI Presentation, March 22nd , 2012, USAID/Washington DC

17. Data Sources
 “DHS” surveys: 1992, 1996, 1999, 2004/5, 2007/8, 2009/10
 Other national/large scale surveys
o HBS 2001, 2007; NMCP 2006, 2008; others
 Facility-based data: Rapid impact assessment 2001-2007
 Un/Published studies and report
GHI Presentation, March 22nd , 2012, USAID/Washington DC

18. Outcome Indicators
Intervention Indicator Trends
ITN “Proportion of Households with at Least One ITN” 1999-2009/10
“Proportion of children under five reporting sleeping 1999-2009/10
under an ITN the night prior to interview”
IPTp “Proportion of women with a live birth in the 2 years 2004/5-2009/10
prior to survey who received ≥2 doses of SP during
her most recent pregnancy, ≥1 via ANC”
Assess the effect of SP stock-out (program data)
Treatment “Proportion of under-fives with self-reported fever 1999-2009/10
in the previous two weeks treated with first line anti-
malarial same/next day after fever onset”
GHI Presentation, March 22nd , 2012, USAID/Washington DC

19. Impact Indicators
Health outcome Indicator Trends
Morbidity Parasitemia prevalence: Proportion of children 2007/08 only +
aged 6-59 months with malaria infection NMCP surveys
Anemia prevalence: Proportion of children age 2004/5, 2007/8,
6-59 months with a hemoglobin measurement 2009/10
of <8 g/dL
Fever prevalence: Proportion of children aged 1999, 2004/05,
6-59 month with reported fever 2007/8, 2009/10
Mortality All-cause under five mortality rate (5q0) 1990-2009/10
GHI Presentation, March 22nd , 2012, USAID/Washington DC

20. Contextual Factors
 Fertility risk, female education, literacy, access to media
 Housing condition
 Nutrition status
 Health care utilization
 Health interventions: ANC, Vaccination, Vit A, PMTCT
 Co-morbidities (ARI, diarrhea)
 Climatic factors-Rainfall pattern from 1990
 Change in GDP
GHI Presentation, March 22nd , 2012, USAID/Washington DC

21. Plausibility-Morbidity and Mortality
 Timing of the change
 Space, dose-response, age pattern
 Correspondence malaria morbidity and mortality
change
 Other factors
 LiST deaths averted (magnitude expected)
GHI Presentation, March 22nd , 2012, USAID/Washington DC

22. Additional Analysis
 District level analysis- HMIS :
o Factors associated with malaria case incidence
o Factors associated malaria deaths
 Analysis of cross-sectional datasets to assess the
association of ITN exposure (and other control
measures) with malaria health outcomes; and
 Survival analysis - ITN exposure and child survival
GHI Presentation, March 22nd , 2012, USAID/Washington DC

23. Challenges - Technical
 Timing of collection of intervention data compared with
measurement of outcomes
 Changing drug policies make it difficult to link trends in
treatment with trends in morbidity/mortality
 Early surveys did not contain standard questions necessary
for calculating some of these indicators (i.e ITN use)
 Data were not always available for the required period
 Plausibility versus causality
GHI Presentation, March 22nd , 2012, USAID/Washington DC

24. Challenges - organizational
 Country ownership - Involving country- NMCP
 Getting all partners to agree one approach
 Review process:
o Consider time to get feedback from the TAG
o Having the report approved by all partners involved takes time
GHI Presentation, March 22nd , 2012, USAID/Washington DC

25. Lessons learned
 Country ownership, but need external oversight to move the
process
 Plausibility approach yes, but further district level analysis
is needed (where possible with available data) to strengthen
the results
 National level estimates from Survey are good, but HMIS
and HDSS data could complement and help tell the story
GHI Presentation, March 22nd , 2012, USAID/Washington DC

26. MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Carolina Population Center at the University of North Carolina
at Chapel Hill, in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for Health,
and Tulane University. Views expressed in this presentation do not
necessarily reflect the views of USAID or the U.S. Government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.
GHI Presentation, March 22nd , 2012, USAID/Washington DC

27. Thank You!
GHI Presentation, March 22nd , 2012, USAID/Washington DC