IHME presents on the Salud Mesoamerica Initiative process evaluation.

1. Salud Mesoamerica Process Evaluation
Evidence on Culture Change in Health Systems
Charbel El Bcheraoui, PhD, MSc
Assistant Professor,
Institute for Health Metrics and Evaluation,
Seattle, WA, USA
Ali Mokdad, PhD
Director, Middle Eastern Initiatives
Professor, Global Health
Institute for Health Metrics and Evaluation,
Seattle, WA, USA
6 October 2016

2. Outline
• Rationale
• Methods
• Preliminary Findings
• Next Steps
2

3. SMI Initial Results - Chiapas
• Tremendous improvement in
supplies from baseline to 18-
month, to 24-month
measurements
• Performance targets were
not met at first follow-up in
Mexico, but were achieved
when given extra time
Mexico indicator for basic child care
Baseline (%)
18-Month
(%)
24-Month
(%)
7.3 40.7 100
Equipment 27.3 64.4 100
Pediatric scale/salter scale 70.9 81.7 100
Child scale/salter scale 52.7 98.3 100
Height rod 69.1 100 100
Pediatric stethoscope 38.9 50 100
Stethoscope 60 100 100
Pediatric blood pressure apparatus 22.2 50 100
Digital/mercury thermometer 97.3 95.6 100
Growth & development card/National
vaccination
card/National health card (0-9 years old) 97.3 100 100
Vaccines1
26.9 29.6 100
Pentavalent (DPT + HepB + Hib) 73.1 59.3 100
MMR 80.8 88.9 100
Rotavirus 73.1 81.5 100
Pneumococcal conjugate 34.6 40.7 100
BCG 69.2 48.1 100
Pharmacy inputs 40.4 88.3 100
Oral rehydration salt/serum 75 100 100
Ferrous sulfate drops/micronutrients 57.7 90 100
Albendazole/mebendazole 76.9 100 100
Antibiotics 76.6 98 100
Ringer's lactate/Hartmann's solution/
saline solution 31.3 100 100
1Only applicable to facilities that store vaccines
3

4. Need for process evaluation
• Unexpected findings to explain
• Crucial design topics not explored
• Inability to assess certain topics quantitatively
• Basically, the “How” and “Why” questions
4

5. Process Evaluation Questions
• SMI influential components
o Use of information
• SMI contribution in the performance of health systems
o Technical assistance
• SMI contribution to the visualization and prioritization of the poor
o Policy dialogue model
• SMI vs. other financing or intervention models
o SMI Design
• Effects of specific interventions and possibility of scale-up
o Sustainability
5

6. Outline
• Rationale
• Methods
• Preliminary Findings
• Next Steps
6

7. Sample Summary
120
Key
Informants
290
Discussion
Participants
410
Total
Participants
7

8. Outline
• Rationale
• Methods
• Preliminary Findings
• Next Steps
8

9. Use of information
9

10. SMI-generated Information is used for different
purposes
• Having a baseline to acknowledge the health status of the population
• Making decisions regarding SMI implementation
• Continuous monitoring of improvements and delays and taking corrective actions
• Evaluating the effectiveness and real impact of the initiative
• Providing accurate data to measure quality of services and countries data
• Dissemination of the information as feedback to stakeholders
• Sharing evidence through scientific articles
• Improving a culture of using information to take decisions
• Using the information for internal learning purposes
SSA members believe that the state and IDB are the main users of SMI data and the data
are not being used by the federal level which has its own indicators and sources of data.
10

11. External evaluation valued by all stakeholders
• In general, there is not trust on accuracy and timeliness of the health system information
data and an independent evaluation is necessary.
• Donor organizations have been a huge consumer of the external evaluative surveys
• IHME provides easy access to original data through solid household and health facilities
surveys
• The most important decisions were based on external evaluation baseline data. The
federal level found some facts about the vaccination coverage and detected weak points
of the cold chain.
• The results of external evaluation were used to set political priorities and helped in
negotiation with the Senate to modify the national health law in terms of vaccination.
11

12. Examples of use of
information
Jurisdictional
warehouses storage
conditions for
medications and
inputs; a revision
after decades
A research in Chiapas
on vaccine potency
(assessment of dried
blood spot): effect on
cold chain and
immunization policies
Time-bound
assessment of
maternal deaths in
Chiapas; installation
of a specific center
(not limited to SMI)
Data from Nicaragua:
challenge of service
coverage gap vs.
quality of care and its
impact on second
operation
Sharing experience at
regional level;
Putting SMI as an
integral part of the
MoH programs in
Guatemala
Local solutions: Some
remote villages went
together to send
someone to the
capital to sort out
commodities
12

13. Improved practices due to knowledge shared
• SMI has facilitated knowledge sharing at all levels: between
countries, states, jurisdictions
• Quality improvement methods that are applied within SMI areas are
being transferred to other areas
• The capacity to draw lessons from and benchmark against other
countries is valued; Chiapas respondents mention reading
newsletters about other countries’ implementation and using
lessons learned
• Regional learning events have been memorable and impactful for
participants; shared ideas such as maternal waiting homes were
shared by respondents from multiple audiences
13

14. Improvements in Health Systems
14

15. Increased role for IDB from fiduciary to technical
• The coordination unit’s role changed from an intermediary between
the bank and the donors, to a pivotal one in organizing and
structuring the technical assistance to support countries in their
operations
• They are perceived as flexible and adaptive to the local context,
where their in-depth analyses of problems has been useful.
• Federal ministry of health respondents, whose attentions are split
between Mexico’s 31 states, valued the local coordination unit
• Chiapas representatives see the technical assistance from IDB as a
great help.
15

16. Improvements were introduced at an appropriate pace
• “You could ask me why I haven’t told you that earlier. You know, it’s
because it was too much. We have been building the whole- since
the start of the initiative, we have been progressively introducing
different tools, according to our opinion, where maturity is good
enough to do so. Yeah? Even though we have that risk management
tool and plan during the design and updated every six months, we
knew it was already too much. And we have to pick our battles.
Every time.”
16

17. MSH is perceived as an essential factor in
strengthening the health system
• MSH provides technical assistant to the Ministries of Health through
SMI
• The focus of their role has evolved from inputs to service delivery
• Seen as successful in improving hospital processes and bringing
alignment between norms and implemented processes
• Collaborative entities who exchange ideas and solve problems,
rather than imposing orders
• Despite the strengths of MSH, the sociopolitical environment of
Chiapas is noted as a barrier to their success
17

18. Health facility management
• SMI has contributed to standardization, the introduction of new
policies, and the increased use and enforcement of protocols at
health facilities
o increased efficiency of internal emergency alert systems (specifically,
when an obstetric emergency is detected)
o Increased organization while applying existing protocols
o implementation of zero rejection policy (for all pregnant women)
o increased use of clinical practice guidelines
o Standardization of staff protocol and procedures from paperwork to
patient intake
18

19. Health facility management
• Consensus that communication has increased and collaboration has
improved at many levels due to SMI
o Increased communication between the staff, the directors, and Mesored
o More contact with the municipal government
o Increase of evidence-based decision making
─ Decision making has become more inclusive and a team based process
• Health facilities note patient-oriented changes in facility management
o Active follow up with patients that miss appointments
o Cultural adapted service provision
o Community outreach
19

20. Logistics and medical products
• General agreement that the increased availability of supplies and
equipment has led to improved operations and processes
o Appropriate equipment directly affected quality of service provision
o New equipment allowed staff to refine skills when dealing with
complicated pregnancies
o Additional materials led to updated procedures
• SMI focus on supplies has increased responsiveness towards health
facility needs/requests
o The majority of informants note more timely responses from ISECH/
Jurisdiction for material requests
20

21. Human resources
• Increased quality and frequency of trainings is well received and
seen as having a direct impact on health system strengthening
o Increased focus on attention protocols and use of flow charts
o Increased knowledge of topics such as family planning, child care, and
postpartum care is perceived to have led to better quality service and a
reduction of deaths/complications.
o Staff self-assessments are now used to track staff progress
• Better defined roles of staff and use of attendance lists have
increased frequency and quality of communication and distribution
of activities
• Increased involvement of jurisdiction to provide human capital
21

22. Human resources
• Though improvements seen, lack of human resources remains a key
issue in Chiapas and is frequently noted as a barrier to improved
health services in the region
o Lack of personnel and specialists continue to prevent some
facilities from effectively reaching their indicator goals and
increasing patient wait times significantly
o Current staff is being stretched across many tasks
o Understaffing, high volume/frequent turnover, and lack of
protocol knowledge has led to patient dissatisfaction
22

23. Information Systems
• Significant improvements and increased use of registries
• Improved documentation of processes, reorganization of services
and human resources
• Implementation of vaccination program with increased record
keeping
• Record keeping trainings led to better materials management and
reduction in stock-outs
• Increased use of management tools, such as a patient database, led
to better informed and more precise decision making in regards to
corrections, improvements, and healthcare risk assessments
23

24. SMI Policy Dialogue Model
24

25. SMI changed the conversation around policies
• SMI has created new discussions around policies in Chiapas, although
some believe they have been in a limited sector within ISECH
• Conversation has changed to a result-based conversation
• Most of the affected policies are cited at the operation level; in the mid-
level, existing policies are being put to operation
• Opinions of SSA staff were generally different with other groups
o They do not feel a policy change and think that SMI has aligned to the
health policies of the Federal Ministry of Health. At the state level,
however, the care quality has been improved.
25

26. Policy changes in Chiapas
• The general approach has been changed from an immediate response
approach to planning with a preventative approach
• Policies which have been developed, amended, and implemented or
have been reinforced due to SMI, include:
o Installation of logistic unit for formal follow-up to supply medical units
o Quality management and continuous quality improvement brought to the
level of the general management
o Managerial information system
o Monitoring or accountability system
o Approval of the CONE strategy
26

27. Faster policies process in Chiapas
• SMI made all the approval processes of policies reportedly faster
o In addition, it created new needs such as quality improvement
─ This was not a big need until SMI.
• IDB developed a methodology for the State to estimate their inputs to
program the acquisitions for the next year
o ISECH applied the methodology to maternal and child health and everything
in the health centers, hospital level, and even the blood banks
• SSA interviewees believe what SMI processes is similar to the previous
processes; there were some changes at the Chiapas State level, but not
at the federal level.
27

28. Disagreement among respondents whether SMI
contributed to the prioritization of the poor
• Some of the interventions (Zinc supplement for management of
diarrhea) are not pro-poor by themselves, but mainly affect the poorest
populations
• Even though the country and state know where the poorest regions are,
they do not know what their needs are
o SMI helped identify these needs
• The main difficulty or weakness cited from SSA is that to have a real
impact, SMI cannot only happen only in one state or 30 municipalities
28

29. SMI Design
29

30. Original timeline shorter than needed or expected
Times for measurements, negotiations, and preparation for following
operations was not totally accounted for. In result, more time was
given for countries to change was they do business
30

31. SMI goals aligned with National Health Plans
• SMI goals fit under MDGs and hence under national health plans to
reduce maternal and child mortality
o National health plans include everything
o Timelines misaligned as SMI continued beyond 2015
─ SDG include maternal and child mortality as well
• In Mexico, while the initiative is aligned with country priorities,
stakeholders at the State level found that the design did not account
for possible social and political difficulties like the ones faced during
first and second operations
o Nevertheless, this risk is accounted for in the risk management plans
and accepted as one that can’t be mitigated
31

32. The RBF model held countries accountable
• It is the defining factor that makes this initiative different from
others and it creates an additional stimulus to achieve results
• It provides a learning platform and the fact that there will be
consequences holds actors accountable
• It forces the mobilization of decision-makers around this financing
• The measurement component promotes continuous monitoring of
pre-established and well-defined goals
32

33. The regional model promoted competition
• “Though not a lot of money from SMI, being measured and fear of
failing is a big motivator for the country and it pushes everybody to
start mobilizing. We mobilize everybody in terms of this objective
and this goal. It's like we are taking an exam and we want to pass"
• “The fact that it is RBF on a regional scale adds a lot of pressure to
achieve results and to not look bad”
• Specifically for Chiapas, competition will be stronger for the second
operation with focus on service delivery
33

34. Caveats during design stages improved from first to
second operation
• Stakeholders engaged down to the State level at the appropriate
stages, but:
o Engagement of local level players was weaker in first, compared to
second operation
o Needs assessment relied on barriers studies and workshops attended by
subject matter experts from participating countries
─ More frequent assessments, including this process evaluation
─ Human resources capacity to implement were reportedly assessed but
informants are in disagreement whether the assessment was accurate
» High staff turnover cited as an issue not taken into account
34

35. Not all stakeholders had equal power in decision
making at al stages
• Deferment of decision-making from SSA to ISECH is seen as the decision to
not decide, to not take part
• Ultimately it was up to ISECH and IDB to make decisions about the
operation in Chiapas
o The country team submitted the operational design for review by a central
IDB review board and other experts.
o The IDB produced a revised version of the proposal including their
comments and recommendations, and submitted it to the Donor Committee
• Few respondents from initial stages of SMI felt that it was not an equal
partnership between IDB and Chiapas to design the operation, but that IDB
was often imposing from above and trying to force policy changes
o Current stakeholders from Chiapas see an equal relationship with IDB
35

36. Sustainability
36

37. Sustainability of changes is possible
• Constant changes in governance impacts public employees at all
levels, all the time.
o This risk is seen as a barrier to sustainability, as priorities are re-
evaluated and knowledge is lost with each shift
o Hard work on technical assistance side (PCU)
─ Active engagement of newly appointed leaders and key stakeholders
• However, policy changes due to SMI are a lasting legacy of the
initiative
• Potentially sustainable changes include accountability,
communication, improved culture of service provision, and close
monitoring and data use at the local level
• Mexico proved willing to fund programs between 1st and 2nd
operation
37

38. Root cause analysis: sustainability of changes due to
SMI
38
Shortage in
human
resource
quantity and
specialists
Improved
quality in
human
resources
Stronger
facilities in
most SMI
areas
Use of
evidence in
decision
making
Between-
country
competition
environment
Continued
monitoring
and
evaluation
SMI as a
regional
model
SMI as a
results-based
aid model
SMI as a policy
dialogue
model
Availability
of supplies
Better health
services
Improved
processes
Technical
assistance Availability
of evidence
New and modified
policies
Need to meet
indicators
Start of a reform
of health
systems
Birth of a
culture of
accountability
Increased demand
from SMI and non-
SMI areas
Longer implementation and
planning time allowing for
changing habits in health
system
Times for measurements,
negotiations, and preparation
for following operations not
well accounted for
Application of newly-
introduced
approaches from SMI
to non-SMI areas
Positive environment for
sustainability of changes
introduced by SMI
Root cause
Obstacle
Context
Consequence
Success

39. Exit plans following SMI are underway in Mexico
• At the local level, the plan is to continue shifting responsibility over
communication, monitoring, and information processes from the
SMI project coordinating unit and MSH to the country
• Federal respondent feels that Mexico will continue to invest in SMI
programs and areas once SMI has ended
• As suggested by a key informant: “Economists have a saying that
“Policy without budget is poetry.” And I think in a way that co-
financing is an attempt to try to put – ministries all have policies
about getting services into the most disadvantaged areas. But in a
way, this actually forces them to put skin in the game and put money
there.”
39

40. Country perspectives on SMI
• SMI is seen as extremely aligned with country priorities to reduce
maternal and child mortality and aim for the MDGs/SDGs
• The initiative empowers countries to revamp existing programs and
revisit adherence to existing health norms and policies
• Many norms, standards, and policies have been directly affected by
SMI, including for the use of micronutrients for children in
Honduras, the treatment of adolescents in Costa Rica, and the
reorganization of service platform levels (the EONC strategy)
• SMI provides a platform for learning and generating new evidence
for health strategies
40

41. In Summary
• Greatest strengths
o SMI is well perceived by countries
─ Making a huge difference in health systems in the area
─ Many lessons for outside donors and global health in general
o Culture of accountability born in many of participating countries
o Beyond knowledge: wisdom in decision making
o Sustainability through policies, scale-up, and spillover
o Drivers of success: regionality, RBF, use of information, and technical
assistance
• Weaknesses and room for improvement
o Local politics: social, institutional, systemic
41

42. Next Steps
• Convergence of findings between quantitative and qualitative data
• Qualitative methods were valuable in complementing the
quantitative ones
o Explained many of the “Hows” and “Whys”
o Also uncovered new fields for investigation
o Will benefit the next rounds of surveys
• Need for process evaluations in remaining countries to account for
their particularities
• Major implication on IHME
o Pragmatic mixed-methods evaluations
42

43. Extra Slides
43

44. Chiapas Performance Improvement Plan
44

45. Multilevel contextual factors slowed Chiapas from
meeting targets in first round
• Change in government caused delays in implementation
o Government shifts caused immense delays in the operation and delays
in access to funds
─ Similar situation for the second operation
o Short period of time between start of implementation and
measurement
• Political instability/political upheaval
o Difficulty to operate with such high turnover
• Less than optimal support from federal to state level
• An information problem/ weakness in the supply chain and the
delays in both procurement and distribution
• Evaluation challenges / all-or-nothing indicators
45

46. Lessons learned from first operation
• Better communication was needed
o Health care staff need to know what the indicators are and what is expected
of them
o The initiative involves working as a team and not in silos or isolation
o Need to continuously strive and to plan well so that there's no rush at the
last minute
• Stronger support from the federal level was needed
• Community involvement and accountability and the role of social factors
are key to success
o The culture of the indigenous people must be integrated into the initiative
• Donors feel that transparency around results is crucial and that the
targets may have been too high
46

47. Changes during the improvement plan
• Positive aspects:
o Communication improved across all levels in the country – federal, state,
jurisdiction, all the way to health care providers
o Banners were placed everywhere and information better known
o IDB provided technical assistance for internal monitoring
o Changes to supply logistics
47

48. Changes during the improvement
• Negative aspects:
o State level acknowledged that this was not a sustainable process – especially
in regard to supply and stocks being prioritized
o May have detracted focus away from other health areas and jurisdictions
during this push for success
o Disagreement amongst donors - some still feel that letting Chiapas and
Guatemala move forward despite not having the requisite equipment in time
is setting the second operation up to fail
48

49. Delays and bottlenecks in the availability of funds and
financial flows (KIIs, C)
There was an almost general agreement between different groups of interviewees
(except some of the SSA members) about the presence of delays and bottlenecks in the
availability of funds and financial flows.
Donors
• Donor organizations: Delays due to internal decision making and audit processes at donor organizations;
Spain government failure to provide committed fund
Donor-
IDB
• Donor-IDB: The necessity of including a highly bureaucratic organization such as IDB between the donors and
the Chiapas government;
IDB
• IDB: The necessity of changing Mexico’s 2009-2010 budget law to make the contribution of CSF possible
(donations from philanthropic organizations to other countries)
Federal
MoF
• Federal level, Ministry of Finance: Lack of trust in government to handle the resources; Delay in audit and
approval process from the Federal government to the State level;
State
MoF
• State level, Ministry of Finance: Social unrest and financial deficit in the State; Non-straight forward and
uneasy access to Federal funds
ISECH
• ISECH: Complexity of the process of allocation of international funds to ISECH through the Ministry of Finance
JD
• Jurisdictions: Complexity of the process in the local bank such as opening a specific account and signature
verification
49

50. Data Analysis
50
1. Transcribe data
2. Group transcribed data by topic
a. Data can be used to answer more than one topic and in these cases are
grouped in both sections
3. Highlight essential portion of each comment
4. Remove portions of comment not relevant to topic at hand
5. Keep code-like sentences for the answers provided
6. Write narrative according these codes with a focus on
a. Agreement of different audiences on responses
b. Level of shared knowledge on same topics or themes by different audiences
c. Agreement between responses from KIIs and document review
7. Compare and interpret information from different groups and
stakeholders

51. Design: Sequential Explanatory/Exploratory
QUAN qual
QUAN
Data
Collection
QUAN
Data
Analysis
qual
Data
Collection
qual
Data
Analysis
Interpretation of
entire analysis
QUAL quan
QUAL
Data
Collection
QUAL
Data
Analysis
quan
Data
Collection
quan
Data
Analysis
Interpretation of
entire analysis
51

52. Outline
• Rationale
• Methods
• Preliminary Findings
• Next Steps
52

53. Document Review (DR)
• Operational plans
• Proposal documents from Mexico, including meeting minutes,
approval letters, and requests for clarification
• Budget documents and expenditure reports for Mexico
• SMI cost-benefit analysis for indicator and target setting
• Mexico National Health Norms
• Master plans
• SMI-Chiapas official reports and memos
• Available meeting notes related to SMI in Chiapas and mission
reports
53

54. Key Informants (KIs)
Study Audience* KIs
SMI Donor Representatives 11
IDB/SMI Coordinating Unit + Management Sciences for Health 12
Regional MOH Representatives (outside of Mexico) 9
SSA + ISECH, including Jurisdiction Leaders 21
TOTAL 53
*Key informants include both individuals who are currently involved in the initiative and
individuals who were previously involved, but are no longer in the same position.
54

55. Key Informants (KIs)
Study Audience* KIs
Health care providers - SMI 44
Health care providers - non- SMI 12
Midwives 11
TOTAL 67
*All health care providers interviews were in their position for at least two years
55

56. Focus Group Discussions (FGDs)*
FGD audience Total FGDs Total Participants
SMI (234) Health committees 4 35
Women with children under 5 17 110
Women without children 7 44
Men 8 45
Non- SMI (56) Health committees 2 10
Women with children under 5 4 26
Women without children 2 11
Men 2 9
TOTAL 46 290
*Data from FGDs not included in the analysis included in this presentation
56