1. Monitoring and Evaluation
Basic Definitions, Institutional
and
Attributes to Establish an M&E System
8 Aug 2012
By Tariq Zaman
1

2. What is M&E?
M&E is about collecting,
storing, analyzing and
finally transforming
data into strategic
information
“Do not COLLECT data
unless somebody is
going to USE IT”
2

3. What Is M&E?
Monitoring
A continuing function that uses systematic collection of data
on specified indicators to provide management with
indications of the extent of progress and achievement of
objectives and progress in the use of allocated funds
Evaluation
The process of determining the worth or significance of a
program to determine the relevance of objectives, the efficacy
of design and implementation, the efficiency or resource use,
and the sustainability of results
M&E are synergistic…monitoring is a necessary, but not
sufficient, input into evaluation
3

4. What is Program Monitoring,
Evaluation?
Monitoring is the Evaluation is the use of
social research
routine process of
methods to
data collection and systematically
measurement of investigate an
progress toward achievement of a
program objectives program’s results
4

5. Why M&E?
S
Data & M y
information & s
t
E e
m
Strategic Information
Evidence informed decision making
Reduced morbidity and mortality
5

6. Importance of M&E
 Provides information:
a) on program progress and effectiveness
b) for policy-making and advocacy
c) to plan future resource needs
 Improves program management and decision-making
(managing by results)
 Allows accountability to stakeholders, including donors
 Ensures most effective and efficient use of resources
6

7. Definition of Monitoring
• Monitoring is the systematic and continuous
collection and analysis of information about
the progress of a project over time
• It is a tool for identifying strengths and
weaknesses in a project and for providing
management with sufficient information to
make the right decisions
7

8. Differences
 Monitoring  Evaluation

Short-term view
 Long-term view

Immediate analysis of
 Through analysis of
achievement of
ongoing activities
specific objectives
 Influences control of  Influences future
ongoing activities planning
 Periodical and regular  A detailed report with
 Carried out by staff or suggestions
donors  Carried out by external
or internal (donors)
8

9. DIFFERENCE BETWEEN
MONITORING AND EVALUATION
 Monitoring means tracking the key
elements of programme performance on a
regular basis (inputs, activities, results)
 In contrast, evaluation is the episodic
assessment of the change in targeted
results that can be attributed to the
programme/project intervention, or the
analysis of inputs and activities to
determine their contribution to results. 9

10. The Purpose of M&E
Program
Improvement
Data Sharing
with Partners
Reporting/
Accountability
10

11. What is the purpose….?
 Improve program implementation

Data on program progress and implementation
 Improve program management and decision making
 Inform future programming
 Inform stakeholders
 Accountability (donors, beneficiaries)
 Advocacy
11

12. Purpose of Monitoring
 Produces timely, accurate and adequate
information about the adherence of a project with its
plan
 Provides data so that plans can be adjusted and
resources managed in answer to project needs
and opportunities
 Records information in sufficient detail to illustrate
accountability and provide for future evaluations
 Appropriate monitoring generates the minimum
data necessary for analysis and uses the simplest
effective data collection methods
12

13. Purpose of Monitoring
 Determining whether the inputs in the project are
well utilized
 Ensuring all activities are carried out properly by the
right people and in time
 Identifying problems facing the community or project
and finding solutions
 Determining whether the way the project was
planned is the most appropriate way of solving the
problem at hand
13

14. Who needs, uses M&E
Information?
Managers
To Improve program
implementation…
 Donors
To Inform and improve  Governments
future programs  Technocrats
 Donors
Inform stakeholders  Governments
 Communities
 Beneficiaries
14

15. Who conducts M&E….?
Program implementer
Stakeholders
Beneficiary
Remember ..
M&E Technical skills
Participatory process
15

16. M&E Results – RBM Approach
SMART Results
pecific: Results must describe a specific future condition
S
easurable: Results, whether quantitative or qualitative, must have
M measurable indicators, making it possible to assess whether they were
achieved or not
A chievable: Results must be within the capacity to achieve
elevant: Results must make a contribution to selected priorities of the
R national development framework
imebound: Results are never open-ended - there is an expected date of
T accomplishment
16

17. BENEFITS OF MONITORING &
EVALUATION
Monitoring and evaluation (M&E) helps programme
implementers to:
Determine the extent to which the programme
/project is on track and to make any needed
corrections accordingly
Make informed decisions regarding operations
management and service delivery;
Ensure the most effective and efficient use of
resources;
Evaluate the extent to which the programme /project
is having or has had the desired impact. 17

18. Monitoring System
 It is a system for collecting and using information
about the progress of a project
 It helps to take appropriate decisions
 Provides a communication system, in which
information flows in different directions between all the
people involved
18

19. Essential Components of a
Monitoring System
 Selection of indicators for each activity
 Collection of data concerning the indicators
 Analysis of data
 Presenting information in an appropriate way
 Using this information to improve the work
19

20. Monitoring Tools and Templates
 Monitoring and Evaluation Framework
 LFA, RBM, and other formats of donors
 Internal/External reviews reports
 Progress reports and reviews
 Work plan and reporting matrix
 Budget utilization and audit reports, etc
 Surveys (structured and un structured interviews)
 Data – primary and secondary (office record,
progress reports)
20

21. Baseline Data
 A collection of data about the characteristics, for
example of a population or an area before a
program or project is set up. The data can be compared
with a study of the same characteristics carried out later in
order to see what has changed.
21

22. Baselines, Targets and
Performance
Commitment
Performance
Current
level of
achieveme
nt
Baseline Target Achievement
22

23. Logical Framework Analysis/Approach
 Logical Framework Analysis/Approach (LFA) is an
analytical process for structuring and systematizing
the analysis of a project or program
23

24. LFA MATRIX
Narrative Verifiable Means of Assumptions
Summary Indicators Verification
Goals
Purpose
Outcome/Output
Activities
24

25. The Logical approach
Goal ofLong-term, widespread improvement

the Results Chain
(Long in society
Term
“Big picture”(country longer term
Impacts)
strategy)
Strategy
 Effects or behavior changes
Outcomes
Results Planning
resulting from a strategic program
Outputs  Products and services that need to
be delivered to achieve the expected
Programming
outcomes
Activities  What actually was done with the
available resources to produce the
intended outputs
Inputs  Critical resources (expertise,
equipment, supplies) needed to 25
implement the planned activities

26. Kinds of Monitoring
 Process Monitoring
 Effect Monitoring
 Monitoring Significant Change
28

27. Process Monitoring
 Process monitoring includes information on the use
of resources, progress of activities, and the way they
are carried out, which is known as process
monitoring
29

28. Process Monitoring
 Process monitoring is collecting information on the
use of inputs, the progress of activities, and the
way these are carried out
 Process monitoring looks at why and how things
have happened; it looks at relevance,
effectiveness and the efficiency of processes
 It involves stakeholders and beneficiaries in
planning, in deciding what is to be monitored, and
in developing and recording monitoring processes
 Process monitoring requires documentation of
how the process was carried out
30

29. Effect Monitoring
 Effect monitoring is collecting information on
progress towards achieving objectives, and on
what the effects are in relation to these objectives
 Effect monitoring is a form of continuous self-
evaluation.
 If it is done well, formal evaluations will be needed
less often, and if a formal evaluation is carried out,
the program staff will already be familiar with their
work in relation to their objectives.
 They will be able to participate more fully in the
evaluation, and find it less threatening.
 All monitoring systems should include both
process and effect monitoring. 31

30. Monitoring Significant Change
 The "significant change"; method of
monitoring is not new, but it is not widely
known
 The method has been used by Australian
Overseas Volunteers to assess their
contribution in development agencies,
during their overseas appointment
Contd.....
32

31. Monitoring Significant Change
 The first step to take is for the staff of the
implementing organization to identify what
areas, or domains, of change they want to
monitor using the significant change method.
 The primary focus should be on two types of
change: changes in the lives of individuals,
and changes in the organization
 The basis of the significant change method is
a simple question: "Describe what you think
was the most significant change that you
contributed to your project"
33

32. The Chain Of Results: Causal Sequence For An
Intervention To Achieve Desired Objective
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
Quality of life
. Management . Services
Conditions:
.Infrastructure . Training
. Counseling
- Access
- Quality
. Behaviors
. Practices
. Human
. Human . Economic
. Logistic . Awareness . Decision
. Finance . Civic
management . Knowledge . Utilization
. Equipment . Environment
. Operation research . Attitude of services
. Technology MDGs:
. Capacities
. Policy
. Conference . Competency . Poverty
. Time
. Facilitation . Opinion . Morbidity
. Volunteers
. etc. . Aspiration . Mortality
. Partners
. Motivation . HIV prevalence
. Education
. Employment
. Gender equality
Measure process
UNCT 26th February 2009 Measure impact 34

33. Inputs
 The human, financial and technical
resources deployed, their effectiveness,
cost effectiveness and opportunities can
be assessed
35

34. Output Monitoring
 Output monitoring looks at the immediate results of
a development programe or project
OR
 The immediate results the project achieves.It is
sometimes called as “deliverables”
36

35. Outcome Monitoring
 Outcome monitoring is the regular reporting of
program results in ways that stakeholders can use to
understand and judge these results
37

36. Impact Assessment
 Significant lasting changes in people’s
lives, brought about by a given action
or a series of actions
38

37. Roadmap: From Input to Impact
• Child mortality reduced,
poverty reduced, higher
Goal income levels, improved
Results (Impacts) gender equality, sustainable
agriculture
• Increased market access,
Outcomes increased short term income
• # producers in the system, #
Outputs new products available, #
Implementation
producers supported, #
consumers aware
• Awareness raising campaigns,
Activities development of standards,
producer support, audits
• Funds, staff & resources etc
Inputs
39

38. Multi-tiered Holistic M&E
Rebuilding Lives and Communities
Input Output Outcome Impact
Monitoring Monitoring Monitoring Assessment
CMTs SSTs
• PC-I •TMRs of •Measurement of •Captures
the services changes in
• Planning and provided by ERRA peoples lives
Designing physical
reconstruction
• Tendering works
•Commencement
•Carried out at •Carried out at
CMTs facilities and SSTs the Household
•Carried out at beneficiary level Level
the Projects’
Site

39. Monitoring and Evaluations
Framework
41

40. M&E Questions
 Monitoring questions

What is being done?

By whom?

Target population?
 When?

How much?

How often?

Additional outputs?
 Resources used? (Staff,
funds, materials, etc.)
42

41. M&E Questions
 Evaluation Questions?

Is the content of the
intervention or the activity
being delivered as
planned?

Does the content of the
intervention or the activity
reflect the requisite
standards?
 Have the intervention
achieved the expected
results?
43

42. What do we need to answer these questions…?
INDICATORS …to take measurements.
44

43. Indicators: Definition
 Markers that help to measure change by
showing progress towards meeting
objectives
 Observable, measurable, and agreed upon
as valid markers of a less well-defined
concept or objective
 Indicators differ from objectives in that they
address specific criteria that will be used to
judge the success of the project or program.
See comment for examples
45

44. What is an indicator?
 The United Nations World Food Programme's Office of
Evaluation describes an as a indicator quantitative or
qualitative factor or variable that provides a simple and
reliable means to measure achievement or to reflect the
changes connected with an intervention. Indicators are
compared over time in order to assess change. In the
logical framework approach, an operation is broken
down into design elements (inputs, activities, outputs,
outcomes and impacts) and separate indicators are used
to measure performance.
46

45. General characteristics
The desired properties of indicators, also known as
variables, will depend on the approach adopted and the
nature of the programme or project being evaluated. All
indicators have specific characteristics:
 Numeric = the values are numbers
 Nominal = the values have names (e.g. male and
female)
 Continuous = the values are infinite or very large
 Ordinal/categorical = the values have a known order
(e.g. low to high)
47

46. Characteristics of Good Indicator
 Indicators will vary from one project to
another, according to the work and its
context, but in general they are often
expected to be:
 SMART (specific, measurable, attainable,
relevant and time-bound)
 SPICED (subjective, participatory,
interpreted, cross-checked, empowering
and diverse)
48

47. Type and Level of Each Indicator
 Type

Input/Process (Monitoring)
 Outcome / Impact (Evaluation)
 Level

Global level
 Country level
 Program level
49

48. What Is a Good Indicator?
 Valid: Measures the effect it is supposed to measure
 Reliable: Gives same result if measured in the same way
 Precise: Is operationally defined so people are clear about what
they are measuring
 Timely: Can be measured at an interval that is appropriate to the
level of change expected
 Comparable: Can be compared across different target groups or
project approaches
50

49. KPI = Key Performance Indicator
55

50. Key Performance Indicators
 Key performance indicator (KPI) is a measure that is
employed to refer to a concept when no direct
measure is available
57

51. WFP Emergency Operation M&E Framework
RESULTS
Input Activities Output Outcomes Impact
(Resources) ( Interventions, Targeted Increased Increased
Services) women receiving household food consumption
full family ration supply especially W,
Ch & V Ind.
X kg maize, Distribution of # of family % of target house Average # of
X kg oil Family ration to ration recipients Hold with adequate Meals per day
X Kg other Women disaggregated Food supply by gender
by gender and age
Program-based Data Population-based Survey
Measure process Measure impact 58

52. Evaluation
 “Evaluation is a management tool which measures
the change or results a project intervention
 An assessment at one point in time of the effects of
a piece of work and the extent to which stated
OBJECTIVES have been achieved.”
60

53. Purpose of Evaluation
 Worth or significance of a development activity, policy or
programme
 The relevance of objectives of projects
 The relevance and effectiveness of programme/project design
and implementation
 The efficiency of resource use
 The sustainability of results
 Incorporates lessons learned into the decision making process
of both partner and donor organizations
 Provides recommendations for policy formulation
 Presents longer term implications in terms of sustainability of
the proposed intervention
61

54. Evaluation should not be done for
 To justify a decision which has already been made
for other reasons, for example, the decision to stop
funding a place of work
 To assign blame for a problem which has arisen
62

55. Types of Questions
Addressed by Evaluation
 About how the programme could be improved
 About how the aims and objectives should be
modified or revised
 About the work can be monitored and evaluated
 About how the work could be made more cost
effective
63

56. Role of Evaluator
 The evaluator is a CONTROLLER in an attempt to hold
implementing agencies responsible for their decisions
and actions
 The evaluator is a MEDIATOR between divergent
knowledge interests
 The evaluator is a FACILITATOR in support of weak
groups attempts to increase their decisions and influence
their own lives 64

57. Advantages and Disadvantages of Internal
and External Evaluators
External Internal
Can a fresh look at the programme Knows the programme too well
Not personally involved so it is easier to be Finds it hardest to be objective
objective
Is not part of the normal power structure Is part of the power and authority structure
Gains nothing from the programme, but May be motivated by hopes of personal gain
may gain prestige from the evaluation
Trained in evaluation methods. May have May not be specially trained in evaluation
experience in other evaluators. Regarded methods. Does not have more (or only a
as an expert by the programme little more) training than others in the
programme
An outsider who may not understand the Is familiar with and understands the
programme or the people involved. May programme and can interpret personal
take a long time to read background behavior and attitudes
information
May cause anxiety as programme staff and Known to the programme, so poses no
participants be not sure of his or her threat of anxiety or disruption. Final
motives recommendations may appear 65
less
threatening

58. Methodology of evaluations
 Combination of qualitative(desk reviews,
key informants interviews, focus group
discussions, observations) and
quantitative(household surveys, health
facility surveys or other special surveys)
methods need to be used and spelled out
in this section.
66

59. Theory-based evaluation
 Similar to Log Frame but more detailed in
understanding programme logic
 Seeks to identify causal or determining factors
seen as important for success and then what
should be monitored
 Ultimately leads to the determination of critical
success factors (CSFs)
 Evaluation of CSFs used to inform likelihood
of programme success
67

60. Cost benefit evaluation
 Cost-benefit and cost-effectiveness analyses
are tools for assessing whether or not the
costs of an activity can be justified by the
outcomes and impacts
 Cost-benefit analysis measures both inputs
and outputs in monetary terms
 Cost-effectiveness analysis estimates inputs
in monetary terms and outcomes in non-
monetary quantitative terms 69

61. Impact Assessment
 The systematic identification of the effects
–positive or negative, intended or not –
caused by a program or project
 Impact evaluations can range fromlarge
scale sample surveys to small-scale rapid
assessment
71

62. Impact Domains
 Impact at Individual and House hold levels

Physical assets

Financial assets

Human assets

Income

Food security
 Impact at Community Level

Physical assets

Natural resource base

Social capital
 Higher-Level Impact

On Institutions

On Policies and Regulations
82

63. M&E SYSTEM FOR QUALITY
RESULTS
 A robust M&E systems supports:

Compliance and mid-course correction

Organizational effectiveness

Efficient resource management and

Timely accomplishment of quality results
83

64. Monitoring and Ealuation Framework
Core Principles
 Joint and Harmonized. Provides a robust M&E Umbrella for all
stakeholders
 Results Focused. Focuses on results and impacts as well as
Budgets
 Lesson Learning. Presents information for continuous lesson
learning and programme planning as well as accountability
 People Focused. Involves all stakeholders and includes direct
feedback from beneficiaries
 Transparency and Communication. Demonstration independence
and communicates information to appropriate people
84

65. Strategic Planning for M&E: Setting
Realistic Expectations
All Most Some Few
Number
of
Projects
Input/ Output Process Outcome Impact
Monitoring Evaluation Monitoring / Monitoring /
Evaluation Evaluation
Levels of Monitoring & Evaluation Effort
93
93