LQAS in Liberia: Piloting a Performance Monitoring System in a Post-Conflict Environment
•
3 likes•985 views
Piloting a Performance Monitoring System in a Post-Conflict Environment, Stephanie Watson-Grant, Anna McCrerey
![[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to LQAS in Liberia: Piloting a Performance Monitoring System in a Post-Conflict Environment
Similar to LQAS in Liberia: Piloting a Performance Monitoring System in a Post-Conflict Environment (20)
More from MEASURE Evaluation
More from MEASURE Evaluation (20)
Recently uploaded
Recently uploaded (20)
LQAS in Liberia: Piloting a Performance Monitoring System in a Post-Conflict Environment
- 1. LQAS In Liberia: Piloting a Performance Monitoring System in a Post-Conflict Environment Anna McCrerey USAID/Liberia Health Officer Stephanie Watson-Grant MEASURE Evaluation Country Portfolio Manager 13 July 2011
- 5. Current GOL/MOHSW Monitoring and Evaluation System Starting Point MOHSW Progress Key Challenges Remaining Fragmented vertical reporting structures Initiated roll-out of an integrated system with standardized indicators and data collection tools System is sporadically utilized and unreliable Not comprehensive; irregular reporting Initiated use of PBCs and annual reviews with incentives for proper reporting and data use Data culture ’ -- data is not systematically used for decision-making Lack of M&E human resource capacity at all levels Recruitment/retention for several upper management positions with requisite experience Shortage of mid-career professionals
- 8. Piloting LQAS to determine feasibility as an appropriate Health Outcome Monitoring System
- 12. Ex: Lofa County Donor Implementing Partner Communities USAID IRC Bahn, Beadatuo, Beeplay, Boyee, Camp One Township, Central, Gelakpoah Viilage, Glarlay New Town, Graie Township, Guerkpahnah, Lorplay, Paye, Tappita City, Tengbein, Yekepa, Yreah, Zeanpea, Zorn Tiah USAID EQUIP Banlah, Beatuo, Boakai Village, Buutuo, Devongbin, Garyeesonnoh, Gbayblin, Gblarlay, Glann Town, Kparblee, Kpowin, Lepula, Rlekporlay, Saclapea, Sarlay, Tuudin, Yarsonnoh, Yenkpalah, Zuotuo USAID EQUIP Beo-Yoolar, Bonla, Duo Boe, Duo Gorton, Ganta, Gbonnie, Kormapea Village, Kpayelepula, Lugbye, Maoh, Nyoanplay 1, Slangonplay, Tahnplay, Yekepa, Yekepa-Barpa, Youbah, Younlay, Zorlepula Irish Aid IRC Behplay, Ganta, Geanplay, Guowee, Karnplay/Larpea, Kissayplay, Loe Lay, Sanniquelle, Sehyi-Geh, Sergeant Old Town, Sopeay, Zolowee ------- MOHSW Biapa, Cocopa Camp 2, Cocopa Camp 6, Gampa, Ganta, Nyansin Old, Nyoanplay, Saclepea
- 20. Challenge Weak implementation of decentralization policy Solution Design dissemination activities that re-enforce and support decentralization including 4 county dissemination meetings with 72 participants mostly from County Health Teams
- 21. Sample of Results for County Level Estimates
- 22. Example 1
- 27. Sample of Results at Lot Level
- 32. Looking Forward
- 36. THANK YOU!
Editor's Notes
- 14 year period of civil conflict GDP fell by 90% and debt swelled to $4.6 billion
- 14 year period of civil conflict Looking forward, Liberia must consolidate socioeconomic and political development foundations to minimize risk of regress. At the end of the conflict in 2003, Liberia was left with a Gross Domestic Product, which was only 50% of the 1988 level and is only now rising to pre-war levels of approximately $1 billion. The Government of Liberia (GOL) strives to become a middle-income country by 2030, which will require tremendous growth and accelerated progress. Recognizing that good health is essential to stability and economic growth, the Ministry of Health and Social Welfare (MOHSW) is working to improve access, availability, and quality of basic health services for all Liberians.
- Infant Mortality Rate (IMR) decreased from 144/1,000 to 71/1,000 Under 5 Mortality Rate (U5M) decreased from 220/1,000 to 111/1,000 Important Details on Infant Mortality: Approximately 18% of <5 deaths are neonatal Most occur in the first 6 days if life. No neonatal-specific indicators in HMIS. Most associated with causes of maternal mortality. Important Details about still births Approximately 2,000 stillbirths per year in Liberia. Most occur just before birth and are associated with complications of labor and delivery or infections, e.g., STIs and malaria. Most could be averted. *other sources have been quoted as >40% stunting rate; we deferred to LDHS 2007 for this statistic Malaria prevalence has decreased by half in between 2007-09 Declines in IMR and U5M attributed to restoration of basic service provision and improvements in the health sector, especially in the area of malaria prevention and treatment. Even with the decrease, however, these mortality rates are still high. And despite the overall decline in mortality, high rates of key morbidities such as diarrhea, acute respiratory illnesses (ARI), malaria, and undernutrition are still concerning
- In the 7 years since war ended, services have been restored at over 80% of public facilities. The government is currently contracting out services at 75% of these 303 health clinics, health centers and hospitals. 14 year period of civil conflict In 2007 - 300 facilities were NGO-managed with uncertain funding The health system was highly fragmented (information systems, salaries, drug procurement, etc.) Human resources for health crisis – only 51 practicing Liberian doctors and shortage of > 7,000 health workers.
- Advantages - LQAS is a relatively rapid and inexpensive approach to data collection that provides a viable alternative to traditional surveys. The method allows for smaller sample sizes than standard probability surveys, and the lower associated costs allow for more frequent sampling . LQAS data can also be used in conjunction with other sources of health information , such as service statistics, to obtain information on coverage and quality of care at the population level . Also, the value of the exercise is its usefulness in local level decision making.
- LQAS is currently the ONLY source of county level estimates in Liberia (both DHS and MIS only drill down to regional level, grouping several counties together) This health outcome performance monitoring system complements service statistics from the HMIS and is usable at the local level by County Health Teams (CHTs). Limitations: Does not produce point estimates for SAs Not intended to measure incremental change over time for the county-based catchment areas: Wide confidence intervals around point estimates Improvements will not register as statistically significant unless they are large Not useful for very low or high-prevalence indicators (<20% or >90%) at the SA-level
- In LQAS application in Liberia, first step was to identify the catchment areas. A catchment area was defined as a county, and four counties were selected as LQAS pilot sites. Division of the entire program area, or catchment area, into meaningful sub-divisions, or “lots”, usually defined as program supervisory areas. At a stakeholder meeting in January, it was decided that the LQAS-based outcome monitoring survey would cover five program areas: water, sanitation and hygiene -+- immunization and child health -+- nutrition -+- malaria -+- maternal health -+- family planning Criteria were used to select indicators, such as programmatic relevance and viability of inclusion, given the structure of LQAS As a result, about 26 indicators were selected, and the following sub-groups were included in the study: women of reproductive age (15-49 years) and children aged 0-59 months -+- children aged 0-5 months -+- children aged 0-23 months -+- children aged 12-23 months -+- children aged 6-23 months -+- children aged 0-59 months that had in the previous 2 weeks: a diarrheal episode -+- a febrile episode -+- cough and difficulty breathing For each LQAS, all indicators included in the study have the same sample size. Each indicator has a denominator of 19 within a single supervisory area, which are enough observations to provide the acceptable/unacceptable determination for that indicator A simple random sample of 19 sample points was chosen within each supervisory area. A detailed protocol was developed to select the interview starting point. The LQAS protocol relies on interview teams to then randomize in the field, thus training and field supervision are key to the successful implementation of LQAS. At the household identified as the starting point, interviewers determined the number of sub-groups able to be completed at the household. The interviewer then proceeded to the household directly to the left (or to the nearest household) to continue completing sub-sections of the questionnaire. This approach is called “parallel sampling”, and is unique to LQAS. This method ensures that each indicator has the same number in the denominator regardless of household composition found at the identified starting point.
- In Liberia, lots were developed according to the source of funding and implementing partner; for example, all facility catchment areas supported by USAID/IRC in Nimba County were considered to be within the same lot. Explain collaborative process of determining lots – consulting with USAID, RBHS, Africare, MOHSW Explain that facilities were used as base for determining catchments, but that it was a community-based data collection methodology Highlight that due to size, Africare was split to two lots…..reminding audience of the need for 5 lot minimum -- Africare assisted in determined how to split their catchment facilities/communities into three lots
- Highlight how MOHSW has facilities here, as well as Irish Aid. Also, point out that USAID supports two different implementing partners in this county and LQAS provides opportunity to compare performance. Would not be possible with county level HMIS data that gets rolled into one figure.
- Discuss how USAID/Liberia had preference to support MOHSW and its staff to do ALL parts of LQAS entirely, but how Asst. Min Wesseh pushed back explaining they didn’t have sufficient staff; MOHSW decision/recommendation to use private sub-partner…..because MOHSW had buy-in, sub-partner was well received and good relationship with MOHSW Discuss where there is still significant room for improvement here. There’s another opportunity to discuss this further along in the presentation (i.e. next steps with capacity building), but can allude the fact that this year was about assessment and laying foundation Mention that even local PRIVATE sub-partner, still required capacity building to underscore challenges facing Liberia
- 32 participants at training in LQAS field survey work - 7 MoHSW staff from national and county offices, including: M&E Officers from Bomi, Bong, Lofa, and Nimba Counties, all of whom went through training and assisted in field survey work - 3 survey research staff from Subah Belleh and Associates (SBA) - 22 survey candidates recruited by SBA, who have varying levels of survey work experience Basic statistical concepts of LQAS -+- Techniques for random selection of community -+- Techniques for random selection of interviewees -+- Conducting parallel sampling of interviewees -+- Survey instruments, protocols, data recording -+- Oversight of survey teams by designated team leader After the training, 16 Survey Interviewers were selected for the 4 County Survey Teams (4 per Team); 4 Survey Team Leaders were also selected (1 per Team)
- Interviewers reviewed their own work Team Supervisors reviewed all questionnaires Roving MEASURE Evaluation 5 th car visits ALL teams – REAL TIME Survey Teams Before leaving a sample point, Survey Interviewers were expected to review the questionnaire forms they had filled out, checking for accuracy and completeness Quality Assurance Team Weekly visits to teams in the field for review of questionnaires allowed for prompt identification of items that needed to be corrected or redone Coordination of visits with each of 4 survey teams facilitated by ongoing cell phone contact, although telecommunications coverage was variable
- Interviewers reviewed their own work Team Supervisors reviewed all questionnaires Roving MEASURE Evaluation 5 th car visits ALL teams – REAL TIME Survey Teams Before leaving a sample point, Survey Interviewers were expected to review the questionnaire forms they had filled out, checking for accuracy and completeness Team Leader sometimes assisted in conducting interviews, but primary duty was to serve as first ‘external’ reviewer Quality Assurance Team Weekly visits to teams in the field for review of questionnaires allowed for prompt identification of items that needed to be corrected or redone Coordination of visits with each of 4 survey teams facilitated by ongoing cell phone contact, although telecommunications coverage was variable
- Interviewers reviewed their own work Team Supervisors reviewed all questionnaires Roving MEASURE Evaluation 5 th car visits ALL teams – REAL TIME Survey Teams Before leaving a sample point, Survey Interviewers were expected to review the questionnaire forms they had filled out, checking for accuracy and completeness Team Leader sometimes assisted in conducting interviews, but primary duty was to serve as first ‘external’ reviewer Survey Interviewers’ performance improved through Quality Assurance Team’s second ‘external’ review of questionnaires and its observation of field survey practices, conducted weekly Quality Assurance Team Weekly visits to teams in the field for review of questionnaires allowed for prompt identification of items that needed to be corrected or redone Coordination of visits with each of 4 survey teams facilitated by ongoing cell phone contact, although telecommunications coverage was variable
- Survey Interviewers’ performance improved through Quality Assurance Team’s second ‘external’ review of questionnaires and its observation of field survey practices, conducted weekly
- Return of completed questionnaires to Monrovia at end of each week allowed for more efficient data processing 08 days development of indicators and lots for LQAS 07 days training in LQAS for Survey Team members 03 days training in data entry and analysis 05 days photocopying and pre-labeling of questionnaires 18 days 4 simultaneous LQAS undertaken concurrently with quality assurance of survey work and data entry 02 days additional time for completing data entry 07 days double entry of questionnaire responses for additional quality assurance
- Example with performance-based indicator where lots had varying results
- Example where all lots met performance-based indicator
- Example where there was no performance-based indicator, so each lot received a different target based on county averages
- After carefully reviewing health statuses, available resources and other donor activities, the USAID/Liberia health team decided to focus the bulk of its programming in three of these six counties: Bong, Lofa, and Nimba. USAID/Liberia believes that the progress and achievements within these three counties will have the greatest effect on overall national health outcomes. With a 1/3 of Liberia’s population residing in Bong, Nimba, and Lofa, there is the greatest potential for influencing national statistics. Additionally, these are counties with potential for strong County Health Team (CHT) leadership, and they are home to existing and planned training institutions. Because of their location in relation to the development corridor, there are also important opportunities for private-public partnerships. In short, these are the three counties, which hold the potential and promise to become model counties of prevention and health care practices, laying the groundwork for replication in other counties in the future.
- After carefully reviewing health statuses, available resources and other donor activities, the USAID/Liberia health team decided to focus the bulk of its programming in three of these six counties: Bong, Lofa, and Nimba. USAID/Liberia believes that the progress and achievements within these three counties will have the greatest effect on overall national health outcomes. With a 1/3 of Liberia’s population residing in Bong, Nimba, and Lofa, there is the greatest potential for influencing national statistics. Additionally, these are counties with potential for strong County Health Team (CHT) leadership, and they are home to existing and planned training institutions. Because of their location in relation to the development corridor, there are also important opportunities for private-public partnerships. In short, these are the three counties, which hold the potential and promise to become model counties of prevention and health care practices, laying the groundwork for replication in other counties in the future.
- Supply Chain Systems, add ’t details: 1. Lack of government leadership to mobilize and coordination donor funds to implement Supply Chain Master Plan…. is primarily due to lack of Government Ownership 2. Chronic stock outs due to lack of coordination/integration, particularly in areas of forecasting, procurement, distribution, and information systems Health Financing, add ’t details: Financing the health system in Liberia continues to be a challenge, and donor support finances the bulk of all primary and secondary service delivery. Approximately 6.7% of the overall GOL budget is allocated to MOHSW; however, the overall health sector expenses far exceed this $25 million – which covers roughly 25% of the overall health costs. The donor community covers approximately 40%, and private out-of-pocket expenditures account for almost 35% of total health expenditures.
- At a stakeholder meeting in January, it was decided that the LQAS-based outcome monitoring survey would cover five program areas: water, sanitation and hygiene -+- immunization and child health -+- nutrition -+- malaria -+- maternal health -+- family planning Criteria were used to select indicators, such as programmatic relevance and viability of inclusion, given the structure of LQAS As a result, about 30 indicators were selected, and the following sub-groups were included in the study: women of reproductive age (15-49 years) and children aged 0-59 months -+- children aged 0-5 months -+- children aged 0-23 months -+- children aged 12-23 months -+- children aged 6-23 months -+- children aged 0-59 months that had in the previous 2 weeks: a diarrheal episode -+- a febrile episode -+- cough and difficulty breathing