Presentation that captures self-assessments of two teams of Ethiopian health officers (most of whom have M&E responsibilities): those from SNNP Regional Health Bureau and those from the Sidama Zonal Health Department.
Sustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Self-Assessment of Organizational Capacity in Monitoring & Evaluation
1. SELF-ASSESSMENT OF ORGANIZATIONAL
CAPACITY IN MONITORING & EVALUATION
by
SNNP REGIONAL HEALTH BUREAU
SIDAMA ZONAL HEALTH DEPARTMENT
PRESENTATION of FINDINGS
05 December 2012
2. Proposed Time Schedule for Discussion
10 Minutes Introductions
PowerPoint Presentations
20 Minutes Terms & Concepts on:
* Capacity Building
* Organizational Development
* SCORE ME org. self-assessment tool of M&E capacity
30 Minutes SNNP Regional Health Bureau
Presentation of Findings from use of SCORE ME
30 Minutes Sidama Zonal Health Department
Presentation of Findings from use of SCORE ME
30 Minutes Questions, Comments, Next Steps
3. DEFINITIONS OF CAPACITY and CAPACITY BUILDING
CAPACITY: Set of abilities and resources to support a level
of desired performance – whether by an individual or an
organization of any size
CAPACITY BUILDING: Processes that improve the ability
of a person, group, organization, or system to meet
objectives and/or to perform better
4. CAPACITY BUILDING: SOMETHING TO THINK ABOUT
• Common approach is to provide training, funding, material
resources, and technical assistance
• Results would be more effective with consideration of
other factors influencing performance of an individual or
organization
behavioral
cultural
systemic
5. MEASURE EVALUATION’S FOCUS in CAPACITY BUILDING
Support the ability of its partners to independently and
sustainably carry out activities in monitoring and evaluation
(M&E) and routine health information systems (RHIS)
Phase I (1998 – 2003) Phase III (2009 – 2014)
• Design of new tools ** Knowledge management
• Data analysis ** Organization development
Phase II (2004 – 2009)
• Routine health information
systems strengthening
• Data demand & use
• Data quality
• M&E technical training
6. FOCAL AREAS for CAPACITY BUILDING in M&E
Operational Functions:
• Management and development of human resources in RHIS
and M&E
• Material and financial resources, and systems to manage
them
• Technical RHIS and M&E skills
• Communication and facilitation of use of information
7. FOCAL AREAS for CAPACITY BUILDING in M&E
Leadership and Relationship Building:
• Promotion of culture favoring information use
• Strategic leadership
• Management of stakeholder relations
8. MEASURE EVALUATION’s CONCEPT of
ORGANIZATION DEVELOPMENT (OD) to SUPPORT M&E
OD Goal: Improve organization’s effectiveness in
managing its own efforts to achieving objectives
Strategy: Motivate individuals and groups within an
organization to seek and sustain best possible
performance
9. MEASURE EVALUATION’s CONCEPT of
ORGANIZATION DEVELOPMENT (OD) to SUPPORT M&E
Method — Assist Organization to:
• Identify individual and institutional needs for obtaining
desired results and levels of achievement
• Identify its current strengths and challenges
• Increase capacity of those at management and leadership
positions to support and champion performance
improvement and positive responsiveness to change
• Improve M&E performance of organization to regularly
assess current strategies, structures, processes,
achievements, results
10. MEASURE EVALUATION’s CONCEPT of
ORGANIZATION DEVELOPMENT (OD) to SUPPORT M&E
Needs of Organization:
• Clarity of organizational vision
• Comprehensive strategic planning
• Effective organizational structure
• Strong governance structure
• Strong capacity among leaders for promoting teamwork and
managing change
• Strong accountability systems (human and financial resource
management, inventory control)
11. MEASURE EVALUATION’s CONCEPT of
ORGANIZATION DEVELOPMENT (OD) to SUPPORT M&E
Results — Organization is able to Self-Assess and Improve its
Accountability Systems for:
1. Planning and executing operational functions and achieving pre-set
objectives
2. Actively undertaking problem solving
3. Committing to its own organizational development as ongoing process
4. Learning and sharing knowledge and perceptions with organizations in
and across sectors
5. Assessing and accessing means for remaining operationally sustainable
and relevant to future needs and demands
12. MEASURE EVALUATION’s CONCEPT of
ORGANIZATION DEVELOPMENT (OD) to SUPPORT M&E
Outcome — Improved organizational management and
performance in:
• Use of M&E methods and tools
• Collection and analysis of routine and strategic information
• Use of routine information for program M&E
• Use of strategic information for organizational planning
• Evidence-based decision making by organization’s leadership
and management
• Sustainability of all the above performance areas
14. Organization’s Functional Capacities
for Self-Assessment in M&E Performance
• Strategic Planning
• Budgeting and Managing Finances
• Implementation Planning
• Documenting & Reporting of Work Activities & Results for Monitoring
Oversight
• Documenting & Reporting of Work Activities & Results for Evaluation
Assessment
• Managing Facilities and Physical Assets
• Managing and Developing Staff
• Promoting Continuous Quality Improvement
15. Features of SCORE ME: Tool for
Self-Assessment in M&E Performance
Developed as a self-administered tool for an M&E
Department/Unit of any agency (government/non-
governmental) to appraise its own level of functional
capacity in eight areas
Designed so that its use on a periodic basis would
promote M&E skills and competencies
Self-assessment in each functional area determined
by number of elements by which agency M&E
performance meets conditions (with reference made if
supporting evidence is available or not)
16. Features of SCORE ME (cont’d)
Agency can customize the use and wording of M&E
performance elements to better suit its purpose for
self-assessment in M&E capacity
Agency can select those functional areas of greater
interest on which to first focus its capacity building
efforts
Agency can develop plans for M&E capacity building
that are based on a formalized pre-assessment of its
M&E performance
17. Introduction of SCORE ME to
Health Sectors in SNNP Region
• Review of content and use of SCORE ME instrument
• Organizational and individual self-assessments of M&E
capacity, with on-site technical assistance, as required
• Formulation of improvement plan for organizational
capacity in M&E, as based on self-assessment findings
18. SNNP REGIONAL HEALTH BUREAU FUNCTIONAL
PROCESSES
• CORE Processes
– Disease Prevention Services and Good Health Practices Promotion
– Curative Health Services
– Regulatory Oversight
– Public Health Emergency Response Services
– HIV/AIDS Preventive Health Services and Disease Control
• SUPPORTING Processes
– Health Program Planning and Health Program M&E
– Disease Diagnoses through Laboratory Services
– Medical Research
– Human Resources Management
– Financial Administration
19. Memorable Quotes
"Measurement is the first step that leads to control and eventually to
improvement. If you can't measure something, you can't understand
it. If you can't understand it, you can't control it. If you can't control it,
you can't improve it." — H. James Harrington
"What gets measured gets done, what gets measured and fed back
gets done well, what gets rewarded gets repeated." — John E. Jones
"Without a standard there is no logical basis for making a decision or
taking action." — Joseph M. Juran
"Nothing changes until you change; everything changes once you
change." — posted at Sidama Zonal Health Department
21. Organization’s Functional Capacities
for Self-Assessment in M&E Performance
A Strategic Planning
B Budgeting and Managing Finances
C Implementation Planning
D Documenting/Reporting Work Activities/Results for Monitoring Oversight
E Documenting/Reporting Work Activities/Results for Evaluation Assessment
F Managing Facilities and Physical Assets
G Managing and Developing Staff
H Promoting Continuous Quality Improvement
SNNP Regional Health Bureau
22. D Documenting/Reporting Work
Activities/Results for Monitoring Oversight
Elements of Capacity Condition
Building/Organizational Met? Evidence Available?
Development Performance in If “Yes”, Specify
M&E by Organization/Unit Check
Unit has written policies /
procedures and docu-
mented evidence for
monitoring on a quarterly
or more frequent basis:
We have: written business process re-engineering
documents; facility standard documents; expenditure reports;
1. Work inputs woreda-based plans; balanced score card; standard reporting
format; standard check-list; routine performance review;
database generated summary data; program-based reporting
Information, Education, Communication/ Behavior Change
Communication design, distribution, dissemination; integrated
2. Work activities
supportive supervision feedback; quarterly HMIS review
feedback to zones
SNNP Regional Health Bureau
23. D Documenting/Reporting Work
Activities/Results for Monitoring Oversight
Elements of Capacity Condition
Building / Organizational Met? Evidence Available?
Development Performance in If “Yes”, Specify
M&E by Organization / Unit Check
Review guidelines; process maturity assessment; standard
integrated supportive supervision; standard guidelines for
3. Work processes reviewing, licensing, and inspecting health facilities; different
service delivery guidelines; laboratory external quality
assurance
Quarterly feedback on program-specific activities; monthly,
quarterly, semi-annual, and annual reports (physical and
4. Work outputs financial); payroll reports; attendance sheets; various models
on in-and-out assets; epidemic monitoring charts; expanded
program for immunizations monitoring charts
SNNP Regional Health Bureau
24. D Documenting/Reporting for Monitoring Oversight
Unit has written
Condition
policies/procedures and
Met? Evidence Available?
documented evidence for
If “Yes”, Specify
responsiveness to
Check
ensuring:
5. Collection of all
required work data
6. Timely collection and
transmission of all
required work data
See the abovelisted justifications. Performance
review team, guidelines, reporting time frame,
7. High quality of all
LQAS needs further improvement
required work data
8. Analysis of all
X
required work data
SNNP Regional Health Bureau
25. D Documenting/Reporting for Monitoring Oversight
Unit has written Condition
policies/procedures and Met? Evidence Available?
documented evidence for If “Yes”, Specify
responsiveness to ensuring: Check
Board minutes book for hospitals; guidelines and manuals
for HIV/AIDS; prevention board and council Management
9. Unit’s leadership/Board
standards for woreda health offices, zonal health
Department, and Regional Health Bureau
Management standards and management minutes book for:
10. Unit’s management hospitals, zonal health department, district health offices and
health centers
Procedures for fund allocations, fund liquidation; project
11. Unit’s funders and management manual; MOA, standard reporting, formal
supporters (financial and physical); standard checklist for project
appraisal
12. Unit’s target population Community-facility forum guidelines; human development
army guidelines
representatives
SNNP Regional Health Bureau
26. D Documenting/Reporting for Monitoring Oversight
Unit has written
Condition
policies/procedures and
Met? Evidence Available?
documented evidence for
If “Yes”, Specify
regular review and use of
Check
work data findings by:
Management standards
13. Unit’s leadership/Board Board minutes book
Management standards
14. Unit’s management Management minutes book
SNNP Regional Health Bureau
27. D Documenting/Reporting for Monitoring Oversight
Scoring from Self-Assessment Findings
Total Number
of 13 / 4 or
5 or 6 7 or 8
9 or 10 or 12 or
Conditions
Met 14 fewer 11 more
1 2 3 4 5
SNNP Regional Health Bureau
28. D Documenting/Reporting for Monitoring Oversight
Corrective Action Plan based on Self-Assessment Findings
MEANS for
Ele- DELIVERABLE(S) RESPONSIBLE PROJECTED
CURRENT ACTION(S) TO MONITORING
ment that can be PARTY for COMPLETN
SITUATION TAKE PROGRESS to
# DOCUMENTED DELIVERABLE(S) DATE
DELIVERABLE(S)
No
procedure
or Availability of
Develop Planning, M&E
guidelines Procedures, document at each By end of
8 for analysis
procedures or
guidelines level (of health care
Process Owner
May 2013
guidelines and partners
of all administration)
required
data
SNNP Regional Health Bureau
29. E Documenting/Reporting Work
Activities/Results for Evaluation Assessment
Unit has written policies/procedures
and documented evidence for having Condition
evaluated for work effectiveness and Met? Evidence Available?
efficiency the projections that had If “Yes”, Specify
been set in Annual Budgets and Check
Implementation Plans:
Assignment of personnel based on BPR
1. Budgeting and deploying inputs document; allocated funds (budgeting)
Process maturity evaluation of evaluation for-
2. Undertaking work processes mat; BPR & BSC implementation manuals;
planning, monitoring, and evaluation processes
3. Completing planned activities X --
4. Generating projected outputs X --
5. Meeting pre-set objectives X --
6. Achieving desired outcomes Operationalization of health research process
7. Contributing to desired impact --
SNNP Regional Health Bureau
30. E Documenting/Reporting for Evaluation Assessment
Condition
Unit has documented evidence for
Met? Evidence Available?
allocation of budget and
If “Yes”, Specify
other resources for:
Check
8. Collection of all required work Resource mobilization for HMIS; woreda-based
planning for research
data and information
9. Quality assessments of work
X --
data
10. Quality assessments of work
X --
performance
11. Analyses of work data X --
SNNP Regional Health Bureau
31. E Documenting/Reporting for Evaluation Assessment
Condition
Unit has documented evidence for
Met? Evidence Available?
allocation of budget and
If “Yes”, Specify
other resources for:
Check
12. Reporting on work data to Monthly, quarterly, semi-annual and annual
reports
concerned stakeholders
13. Using the findings from
reports for the improvement Annual performance used for evidence-based
planning and written feedback
of planning and
implementation
Availability of research staff to do research and
14.Conducting research on ways technology transfer sub-process and
to promote work disseminate research results; rapid
effectiveness and efficiency assessment conducted on data utilization for
decision-making
SNNP Regional Health Bureau
32. E Documenting/Reporting for Evaluation Assessment
Scoring from Self-Assessment Findings
Total Number
of 8/ 4 or
5 or 6 7 or 8
9 or 10 or 12 or
Conditions
Met 14 fewer 11 more
1 2 3 4 5
SNNP Regional Health Bureau
33. E Documenting/Reporting for Evaluation Assessment
Corrective Action Plan based on Self-Assessment Findings
MEANS for
Ele- DELIVERABLE(S) RESPONSIBLE PROJECTED
CURRENT ACTION(S) TO MONITORING
ment that can be PARTY for COMPLETN
SITUATION TAKE PROGRESS to
# DOCUMENTED DELIVERABLE(S) DATE
DELIVERABLE(S)
Amount of
budget mobilized
Financial reports
Inadequate Amount of
allocation for Revise Response letters
budget from partners SNNP RHB
quality resource
allocated
allocations Reports on
assessments, MEASURE June
1 performance Resource quality Evaluation 2013
Develop assessments
reviews, and mapping
resource and performance Partners
analysis of document
mapping reviews
developed
work data conducted
Reports on work
data analysis
SNNP Regional Health Bureau
34. E Documenting/Reporting for Evaluation Assessment
Corrective Action Plan based on Self-Assessment Findings
MEANS for
Ele- DELIVERABLE(S) RESPONSIBLE PROJECTED
CURRENT ACTION(S) TO MONITORING
ment that can be PARTY for COMPLETN
SITUATION TAKE PROGRESS to
# DOCUMENTED DELIVERABLE(S) DATE
DELIVERABLE(S)
Terms of reference
Skill gap in Capacity End of
developing Number of trained Selection of
building training January
evaluation M&E Officers and trainees
for basic MEASURE 2013
other focal
frame work for concepts of Evaluation (training)
2 persons Attendance lists
evaluation evaluation for
activities, out- M&E Officers SNNP RHB May 2013
M&E framework Training documents
puts, & pre-set and other focal M&E
developed
persons framework
objectives Pre- & post-test
evaluations
SNNP Regional Health Bureau
37. Organization’s Functional Capacities
for Self-Assessment in M&E Performance
A Strategic Planning
B Budgeting and Managing Finances
C Implementation Planning
D Documenting/Reporting Work Activities/Results for Monitoring Oversight
E Documenting/Reporting Work Activities/Results for Evaluation Assessment
F Managing Facilities and Physical Assets
G Managing and Developing Staff
H Promoting Continuous Quality Improvement
Sidama Zonal Health Department
38. D Documenting/Reporting Work
Activities/Results for Monitoring Oversight
Condition
Elements of Capacity Building /
Met? Evidence Available?
Organizational Development Performance
If “Yes”, Specify
in M&E by Organization / Unit
Check
Unit has written policies/procedures and
documented evidence for monitoring
on a quarterly or more frequent basis:
Strategic plans
1. Work inputs Woreda-based plans
Standard guidelines for different
2. Work activities programs (TB, HIV, RH, TB/HIV
collaborative activities)
Performance reports
3. Work processes Business process re-engineering
Balanced score card
Standard checklists
4. Work outputs
E-HMIS Summary reports
Sidama Zonal Health Department
39. D Documenting/Reporting for Monitoring Oversight
Condition
Unit has written policies/procedures and
Met? Evidence Available?
documented evidence for
If “Yes”, Specify
responsiveness to ensuring:
Check
5. Collection of all required work
data
6. Timely collection and HMIS tools and guidelines
transmission of all required work Standard reporting schedules
HMIS minutes book
data
Performance Review Team
minutes book
7. High quality of all required work Analysis of routine performance
X
data reports from respective woredas,
as well as within the Organization
(unit)
8. Analysis of all required work data
Sidama Zonal Health Department
40. D Documenting/Reporting for Monitoring Oversight
Condition
Unit has written policies/procedures and
Met? Evidence Available?
documented evidence for
If “Yes”, Specify
responsiveness to ensuring:
Check
9. Unit’s leadership/Board
Management minutes book
Management standards
10. Unit’s management
Regular monthly/quarterly
11. Unit’s funders and supporters reporting, including review
meetings with concerned bodies
12. Unit’s target population Contractual agreements with
partners
representatives
Standard checklists
13. Unit’s leadership/Board
Management minutes book
Management standards
14. Unit’s management
Sidama Zonal Health Department
41. D Documenting/Reporting for Monitoring Oversight
Scoring from Self-Assessment Findings
Total
Number of 13 / 4 or
5 or 6 7 or 8
9 or 10 or 12 or
Conditions
Met 14 fewer 11 more
1 2 3 4 5
Sidama Zonal Health Department
42. D Documenting/Reporting for Monitoring Oversight
Corrective Action Plan based on Self-Assessment Findings
MEANS for
Ele- DELIVERABLE(S) RESPONSIBLE PROJECTED
CURRENT MONITORING
ment ACTION(S) TO TAKE that can be PARTY for COMPLETN
SITUATION PROGRESS to
# DOCUMENTED DELIVERABLE(S) DATE
DELIVERABLE(S)
Criteria based
Capacity
selection of
building on Number of
trainees and
data manage- staff trained MEASURE Jan 2013
trainers
ment on evaluation Evaluation (capacity
Training
Appropriate framework SNNP RHB building)
manuals
Low technology for Number of Sidama ZHD Jan 2013
Pre- and post-
data manage- offline HMIS All partners (techno-
7 quality ment (software data
tests
working in logy)
data Available
data analysis management health April
technology for
at zonal as centers programs in 2013
offline data
well as woreda Established Sidama (integra-
management
levels) One Report Zone tion)
Established
Data System
One Report
integration
System
Sidama Zonal Health Department
43. E Documenting/Reporting Work
Activities/Results for Evaluation Assessment
Unit has written policies/procedures and
Condition
documented evidence for having evaluated
Met? Evidence Available?
for work effectiveness and efficiency
If “Yes”, Specify
the projections that had been set in Annual
Check
Budgets and Implementation Plans:
1. Budgeting and deploying inputs X --
2. Undertaking work processes Performance Report
3. Completing planned activities X --
4. Generating projected outputs X --
5. Meeting pre-set objectives X --
6. Achieving desired outcomes X --
7. Contributing to desired impact X --
Sidama Zonal Health Department
44. E Documenting/Reporting for Evaluation Assessment
Condition
Unit has documented evidence for
Met? Evidence Available?
allocation of budget and
If “Yes”, Specify
other resources for:
Check
8. Collection of all required work
Performance reports
data and information
9. Quality assessments of work
X --
data
10. Quality assessments of work
X --
performance
Analysis of data from performance
reports
11. Analyses of work data Analysis of data from project
evaluations
Sidama Zonal Health Department
45. E Documenting/Reporting for Evaluation Assessment
Condition
Unit has documented evidence for
Met? Evidence Available?
allocation of budget and
If “Yes”, Specify
other resources for:
Check
Dissemination to community
representatives
12. Reporting on work data to
Dissemination to partners
concerned stakeholders Dissemination/communication within
the Organization
13. Using the findings from Reviewing performance reports by
management teams
reports for the improvement
Reviewing performance reports by
of planning and core-process/supporting process
implementation teams
14.Conducting research on ways
to promote work X --
effectiveness and efficiency
Sidama Zonal Health Department
46. E Documenting/Reporting for Evaluation Assessment
Scoring from Self-Assessment Findings
Total Number
of 5/ 4 or
5 or 6 7 or 8
9 or 10 or 12 or
Conditions
Met 14 fewer 11 more
1 2 3 4 5
Sidama Zonal Health Department
47. E Documenting/Reporting for Evaluation Assessment
Corrective Action Plan based on Self-Assessment Findings
MEANS for
Ele- DELIVERABLE(S) RESPONSIBLE PROJECTED
CURRENT ACTION(S) TO MONITORING
ment that can be PARTY for COMPLETN
SITUATION TAKE PROGRESS to
# DOCUMENTED DELIVERABLE(S) DATE
DELIVERABLE(S)
Feedback MEASURE
following Evaluation
budget SNNP RHB
Resource Resource Amount of approval Sidama ZHD
Jan
1 (financial) (financial) budget Amount of All partners
2013
scarcity scarcity allocated budget really working in
spent for health
evaluation programs in
purposes Sidama Zone
Sidama Zonal Health Department
48. E Documenting/Reporting for Evaluation Assessment
Corrective Action Plan based on Self-Assessment Findings
MEANS for
Ele- DELIVERABLE(S) RESPONSIBLE PROJECTED
CURRENT MONITORING
ment ACTION(S) TO TAKE that can be PARTY for COMPLETN
SITUATION PROGRESS to
# DOCUMENTED DELIVERABLE(S) DATE
DELIVERABLE(S)
Criteria based
MEASURE
selection of
Institutionalizing Evaluation
Documented trainees and
an evaluation SNNP RHB
Failure culture within the
evaluation trainers
Sidama
to plan plans Training
Organization ZHD
Number of manuals Jan
3 for Capacity All partners
staff trained Pre- and post- 2013
evalu- building in working in
on tests
planning and health
ations evaluation Cross-
implementation programs in
frameworks checking the
of evaluation Sidama
plan
Zone
documented
Sidama Zonal Health Department
52. MEASURE Evaluation is funded by the U.S. Agency for
International Development (USAID) and implemented by the
Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF International, John Snow, Inc., Management Sciences for
Health, and Tulane University. Views expressed in this
presentation do not necessarily reflect the views of USAID or the
U.S. government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.