International Food Policy Research Institute (IFPRI) organized a three days Training Workshop on ‘Monitoring and Evaluation Methods’ on 10-12 March 2014 in New Delhi, India. The workshop is part of an IFAD grant to IFPRI to partner in the Monitoring and Evaluation component of the ongoing projects in the region. The three day workshop is intended to be a collaborative affair between project directors, M & E leaders and M & E experts. As part of the workshop, detailed interaction will take place on the evaluation routines involving sampling, questionnaire development, data collection and management techniques and production of an evaluation report. The workshop is designed to better understand the M & E needs of various projects that are at different stages of implementation. Both the generic issues involved in M & E programs as well as project specific needs will be addressed in the workshop. The objective of the workshop is to come up with a work plan for M & E domains in the IFAD projects and determine the possibilities of collaboration between IFPRI and project leaders.

1. Results and Impact
Management System (RIMS)
Devesh Roy and IFPRI/IFAD team (March 10, 2014)
Delhi

2. 2
Management Information System (MIS)
• System -provides information at any point in the life of the project –
both past and present, or up-to date information for taking project
initiatives, and for corrective actions.
• Information in the MIS comprise both M&E information, and
information regarding the context of the project area-critical for
effective operation of the project.

3. 3
Basic Components of the MIS (Other elements used to the extent
employed in M&E)
• Basic Statistics
• M&E Information based on Component wise Results Indicators
• Financial Information System
• Information on Production and Price
• Information on Public and Private Service Providers
• Information on Markets
• Information on Training Institutes and Research Stations
• Information on Innovations and Technologies
• Information on Wholesalers, Exporters and Transporters
• Process Documentation, etc…

4. Core philosophy and rationale of M&E
• Shift focus from activities and outputs to higher level results
(outcomes and impact)

5. 5
Broad steps for setting up the M&E system
• M&E stakeholders workshop for review and revision of the Logframe
• Preparation of the M&E Plan
• Train staff on RIMS and Baseline Survey
• Conduct RIMS and Baseline Survey
• Prepare the Results Based AWPB (Annual work plan and Budgeting)
• Prepare and test Input-Output monitoring forms
• Ensure that the M&E system is manually up and running & capable of generating reports
• Establish data base for other information areas for the entire MIS
• Computerize the system and test the system
• Enter data regularly, generate reports and use it for learning and decision-making

6. Key Steps in Setting Up an M&E System and Preparing
the M&E Plan
1. Establish the Purpose and Scope;
2. Identify performance questions, stakeholders’ information
needs, and indicators;
3. Define information collection tools, methodologies,
frequencies, and responsibilities;
4. Planning for data processing and analysis;
5. Planning for quality communication, information sharing,
and reporting arrangements;
6. Planning for critical reflection and learning events;

7. Key Elements of an M&E Plan (1)
• Purpose & Scope
• Project overview and objectives
• Rationale for M&E system design
• How the system will support project management and its
partners
• How the system will meet reporting requirements
• Discussion of extent of participation of stakeholders
• Approach
• Overview of how stakeholders will be involved (project
team members at various level, implementing partners,
beneficiaries, external resources…)
• What information gathering and analyzing methods will be
used

8. Key Elements of an M&E Plan (2)
• Performance questions, indicators, and related
information needs
• Define key questions to be answered to understand
whether the project is performing as planned and moving
towards desired objectives;
• Definition of target outcomes, indicators and related
information/data needs
• Baseline data requirement
• Assessment of stakeholders' information needs

9. Key Elements of an M&E Plan (4)
• Information Dissemination and Utilization
• Reporting (format, frequency, responsibilities)
• Communication strategy
• Events for performance review and critical reflections to inform
decision making
• How will the project follow up on M&E information and corrective
action
• Arrangements to support and provide information to external reviews
and evaluations (supervision missions, mid-term review/evaluation,
interim evaluation, completion assessment)
• M&E Work Plan and timing of Activities
• Monitoring of resources, activities an implementation for effective
operations
• Evaluation of outcomes and impact for guiding the project strategy

10. Key Elements of an M&E Plan (5)
• Establishment of Necessary Conditions and Capacities
• M&E organization
• Human resource needs: staff, incentives, external assistance required
• Capacity building plan for M&E: training, exposure, piloting
• Resource needs: equipment, vehicles, TA….
• M&E detailed budget
• Appendices
• Original and revised log frames
• List of proposed indicators
• Outline formats for data collection
• Outline formats for reporting
• Baseline survey questionnaire
• M&E work plan

11. 11
Monitoring & Evaluation ?
• Monitoring -continuous observation, reflection, and correction of
activities. Done frequently –at short time intervals
• Evaluation a similar exercise, where observations on CHANGE as a
result of project intervention (intended, unintended, positive and
negative) is compared over time (baseline, mid-term and end-of-
project), here lessons and best practices are also drawn to improve
present as well as future project design and management.

12. Monitoring and evaluation: Why?
Support project management with information needed for
decision making (management tool);
Provide information on project progress, results, and impacts;
Timely trend detection can prevent project failures and facilitate
necessary adjustment to project activities and strategy;
Incorporating “lessons learned” will facilitate improving
implementation and enhance chances of achieving desired
impact (learning tool);
Accountability for donor funds.

13. 13
M&E Tools in IFAD Projects
• Logical Framework
• Results and Impact Management System
• Results Based Annual Work Plan and Budget
• Annual Outcome Surveys
• Multidimensional Poverty Assessment Tool (MPAT)

14. RIMS: Aims and Objectives
Motivation/objective: Establish a comprehensive system for measuring and
reporting on the results and impact of IFAD supported country programmes
• Provides framework for systematic reporting by the projects to IFAD and by
IFAD to its governing bodies.
• Includes a menu of indicators used to measure and report on the
performance of IFAD projects – at activity, output and impact level.
• The menu provides a standardized list of indicators that could be applied
across all projects, and therefore can be aggregated.
• Standardized questionnaire- issue of flexibility in design and analysis??

15. Reporting in the context of RIMS
• More attention to impact management
• Core indicators, mandatory for all IFAD projects
• Selection of programme specific indicators from the RIMS universe
• Data on indicators reported to IFAD annually, periodically reviewed with IFAD HQ,
ICO or supervision teams
• Impact surveys required at baseline and completion

16. Variables measured through RIMS impact surveys
 The “anchor” indicators:
• Household asset index
• Child malnutrition
• Female/ male literacy
• Access to safe water
• Access to improved sanitation
 Others:
• Food insecurity (intensity and spread of hungry
seasons)

17. 17
RIMS Impact Indicators
MDG Indicators
Goal-1: Eradicate extreme poverty and
hunger
•Households with improvement in household
assets ownership index*
•Reduction in the prevalence of child
malnutrition*
Goal 2: Achieve universal primary
education
Number of Children completing
primary/secondary school (male/female)
Goal 3: Promote gender equality and
empower women
•Ratio of literate females to males
•Literacy rate (by gender)
Goal 6: Combat HIV/AIDS, malaria and
other diseases
Reduction in the incidence of infectious disease
(HIV/AIDS, malaria, tuberculosis)
Goal 7: Ensure environmental
sustainability
•People with access to improved sanitation
•People with sustainable access to an improved
source of water (drinking water)

18. From results to impact
1st LEVEL
RESULTS
[OUTPUTS]
2nd LEVEL
RESULTS
[OUTCOMES]
IMPACT
Measures financial and
physical progress
indicators
— outputs
Assesses effectiveness
and sustainability and/or
modified behaviour
— outcomes
Measures combined
effect of 1st and 2nd level
indicators
— goal
RIMS Reporting Impact Surveys

19. Executing RIMS from start-up to project completion
• Identify and agree on a list of indicators, including targets.
• Ensure that selected RIMS indicators are part of the M&E system.
• Monitor the indicators and report to IFAD annually.
• An annual record is to be submitted to IFAD with annual Progress reports
and reviewed by IFAD supervision missions.
• Conduct Baseline assessment at start-up and impact assessment at mid-
term and completion.
• RIMS Annual Outcome Surveys

20. Launching and implementing RIMS
• Project Logframe -reviewed to reflect RIMS indicators.
• IFAD and PMU to agree on a number of indicators for the 1st and
2nd level (along with appraisal targets and AWPB estimates)
• An annual record to be submitted to IFAD with annual Progress
report and reviewed by IFAD supervision missions.

21. Executing RIMS: survey software
Available in English, French, Spanish, Arabic
Reports:
All standard RIMS analyses embedded in software;
Reports produced in tables and graphs;
Child malnutrition measured against new WHO standards;
Relative wealth reports conducted;
All reports and all data can be exported to different
formats;
Follow up surveys can be compared
Manual available online

22. 22
Objective of M & E
To support managing for project results by focusing on improvement of project
performance by
• Tracking progress of project outcome (intermediate impacts) and Impact
based on data collected at baseline, mid-term and at the end-of-project;
• Quarterly, half-yearly and annual monitoring of achievement of project
outputs through the conversion of input to output against AWPB targets; and
• Reflecting, learning and taking actions to improve performance to achieve
results.

23. Key processes in the M&E system

24. Background on RIMS: Some salient points
• Where suitable, household-centered indicators introduced to better measure project outreach.
• Rating-based approach for reporting 2nd level results. The ratings supported by a flexible mix of
evidence using qualitative and/or quantitative methods including RIMS 2nd level indicators.
• Projects to choose the most suitable method for measuring 2nd level results based on local context
and characteristics, including that of countries’ existing M&E systems.
• RIMS overall rationale, the guiding principles, features and concepts.
• how RIMS indicators should be selected.
• guidance on each first-level indicator including definitions and operational aspects related to
reporting. methodological issues that are common to several indicators.
• Guidance in formulating assessments of second level results (and includes RIMS second-level
indicators where appropriate).
• the reporting process, including sample report forms.

25. Measurement method
• New measurement approach for 2nd level results introduced.
• Achievement of 2nd level results measured by a rating scale.
A score of 1 corresponds to highly unsatisfactory
effectiveness or very weak sustainability. By contrast, a score
of 6 corresponds to highly satisfactory effectiveness or very
strong sustainability.
• Question to ask about the tradeoff between effectiveness
and sustainability (Union or intersection of the two markers)

26. Measurement issues
• Question- effectiveness and sustainability- union or intersection of
the two
• How does a moderate ranking (say 4) on sustainability or like it
follow? Rankings are ordinal or cardinal?
• Political economy internalized in sustainability?
• How to ensure objectivity in both effectiveness and sustainability?

27. Effectiveness and sustainability

28. Overview of RIMS First- and Second-Level Results
• Natural resources (land and water)
• Agricultural technologies and production
• Rural financial services
• Markets
• Rural enterprise development and employment
• Policy and community programming
• Social infrastructure
• Total outreach

30. Use of RIMS level 1 and 2
• Comparing targets of 1st level results with actual achievement may lead to
an analysis of factors that positively or negatively affected project
implementation.
• The analysis of 2nd level results provides more in-depth information on the
changes occurred at the level of beneficiaries, households, communities or
institutions.
• E.g. After a training program on livestock veterinary practices, the small number of
trainees adopting the recommended techniques may highlight problems with the
training method like language problems or inaccessible locations.
• Taking into consideration these findings, project management may decide on training
in a different form or that another training service provider will be recruited.

31. RIMS indicator levels
• 1st level: Project outputs
Reported annually
• 2nd level: outcomes
Reported from mid-term onwards
• 3rd level: impact
At baseline and completion:
- Asset ownership
- Child nutrition
- Water, sanitation, housing, etc.

32. Definitions: First-Level Results (Outputs)
• Measure financial and physical progress, mostly quantitative
(numbers and percentages) and are generally planned and
implemented on annual basis.
• Measures of results at either the activity or output level of the
logframe.

33. Definitions: Second-Level Results
(outputs/outcomes)
• Measure improved functionality and/or behavioural change. They are
more qualitative and normally take longer to realise. Should they be
or there is scope or reason to alter this???
• This level corresponds to either the output or objective level of the
Project Logframe.

34. Definitions: Third-Level Results (impact)
• Measure the combined effect of the 1st and 2nd level results.
• quantitative and measured at specific point during project life.
• may refer to the goal level of the project logframe and in some case
to the objective level.

35. Frequently Asked Questions
• Does the list of first-level indicators replace the previous one?
• Yes. Compared to earlier few indicators have been added and the presentation has been reorganized. For
example, “people trained” has been replaced by people trained in natural resources management, people
trained in infrastructure management) or the training recipient (staff of service providers)
• What if RIMS first-level indicators do not cover all activities and outputs that are relevant for my project?
• The list of 1st level indicators cannot be comprehensive of all activities and outputs of all projects. Choose
only first-level indicators that are relevant to the project characteristics.
• Are there any compulsory first-level indicators?
• Yes. At least one indicator that shows the number of individuals and households that during the period
under review have received project services should be reported.
• Do all indicators under a given category need to be reported?
• No, the indicators have been categorized for ease of reference. For a given component, relevant indicators
may be found in any of the categories used in the Handbook.
• Should first level indicators continue to be reported on the RIMS form if they have been dropped?
• No, but the data collected may prove useful for the project M&E.
• What if there are no planned or appraisal target figures?
• If targets are not available, NA should be reported. However, planning for results is necessary for project
accountability and is an important element of the M&E system. The Logframe may need to be revised so
that indicators and appraisal targets are established.

36. 36
The Logframe Matrix
• Used to communicate key information about
project objectives, outcomes, outputs and
activities in a systematic and logical way.
• The basic Logframe matrix contains 4 columns
and 4 rows.
• Describes external factors that influence the
project’s success: assumptions and risks

37. 37
The Logical Framework Matrix
Objectives &
activities
Purpose/
(Outcome)
Goal
(Impact)
Outputs
Activities
Means
Indicators
Means of
verification
Assumptions
Cost

38. Key Elements of an M&E Plan (3)
• Data/Information Sources, Gathering Methods, and
Responsibilities
• How, who, when of data collection
• Data/Information Management Information System
(MIS)
• How, who, when of data storage, synthesis, and
aggregation
• Data/Information Processing and Analysis
• How, who, when of data analysis

39. FAQs: Continued
• Why is achievement of second-level results reported with a rating?
• Ratings introduced to allow projects to report achievement of 2nd level results by using project-specific indicators and any other
evidence available. Assessments allow information from other sources, in particular national systems. Ratings also align RIMS
with other evaluative instruments such as Project Completion Reports and formal evaluations.
• How about the risk that rating may be too subjective?
• Rating should be a realistic and candid assessment by as wide a group of stakeholders and may be validated during supervision
missions and project reviews.
• Who should rate the 2nd level results?
• Unless IFAD and project agree on a different arrangement, the rating of 2nd level results should be done by the project.
• When should second-level results be reported?
• 2nd level results should be reported after the Mid-Term Review. The rating should be updated annually based on new evidence.
2nd level results can be reported earlier if evidence is available.
• What are the implications of a low rating for a second-level result?
• A low rating does not imply that the project management is not committed to achieve the development objectives. Lower-than-
expected results should be used to identify corrective actions
• What if the M&E system does not include indicators for assessing RIMS second-level results?
• Outcome-level indicators need to be included in the project M&E system. Consider reviewing the project Logframe in the next
annual planning phase so that appropriate indicators can be included

40. Qualitative tools
1. Key Performance Questions
2. Knowledge, Attitude, Practice (KAP) Surveys
3. Most Significant Change (MSC)
4. Focus Group Discussions
5. Key Informant Interviews

41. FAQs: continued
• How is the supporting evidence for second level results submitted?
• When the RIMS form is submitted (normally an Excel file), the supporting evidence used
for assessing 2nd level results should be included in an additional worksheet or separate
text document(s).
• Clear indications of the methodology and source should be provided.
• The supporting evidence should be in summary form.
• Where studies or surveys have been carried, it is not necessary to submit the entire
study/survey but clearly reference these as source material.
• The evidence may include RIMS second level indicators which should be reported as
evidence under the relevant second level result.
• When should the RIMS form be sent to IFAD?
• The deadline for submission of the RIMS form is 31 March.
• To what period do RIMS results refer?
• RIMS results refer to the period of the Annual Work Plan and Budget (e.g. January to
December or July to June).

42. Challenges (from IFAD perspective)
 Data quality problems:
 Problems of double-counting of outputs
 Data not produced timely (e.g. before
supervision missions)
 Few credible data on outcomes and impact

43. Challenges and risks
 RIMS surveys methodology not really adequate to
measure project impact:
 No control/comparison group
 Sample households are not necessarily project
beneficiaries
 Food insecurity and child malnutrition measures
highly sensible to annual variations in food
production
 Three malnutrition variables difficult to
interpret

44. RIMS First and Second level reporting
• First section of first level results
• First level results are reported annually
• AWPB target and actual result-km of roads target to be
constructed/rehabilitated and actual km of roads
completed/rehabilitated – Percentage of target met
• Second section of first level results
• Cumulative value of each indicator and comparison with the target
results reported in the Logframe or in the Appraisal documents.

45. Illustration

46. Reporting of RIMS second level results
• Recall - Second-level results is reported on the basis of a rating scale from 1 to 6. A
score of 1 corresponds to highly unsatisfactory effectiveness or very weak
sustainability. A rating of 6 corresponds to highly satisfactory effectiveness or very
strong sustainability.
• The rating should be reported under the column of “actual” for the period under
review. No other value or number should be shown in the RIMS form for second-
level results. The section on cumulative and appraisal values does not apply to
RIMS second-level results.
• When the RIMS form is submitted (normally an Excel workbook), supporting
evidence used for the assessments should be included in an additional worksheet
or separate text document(s).
• Clear indications of the methodology and source should be provided. The
supporting evidence should be in summary form.
• RIMS second-level results should be provided at the time of Mid-term Review
(MTR). The ratings of RIMS second-level results should be updated every year on
the basis of the new data and evidence available.

47. Checklist for sending the RIMS form
• Clearly outlined in the RIMS manual - reminder about gold medal, silver medal and bronze
medal gaps
• Are first-level indicators reported in the appropriate units (number, hectares, kilometers,
etc.) as specified in the indicator charts?
• Where relevant, are sex-disaggregated figures reported or other stratification?
• For projects after PY2, are cumulative figures consistent with the data in the previous
years? Have figures for previous years been attached or entered on the spreadsheet?
• If revisions have been undertaken in the Logframe, are targets updated?
• Are the ratings for second-level results shown? Are ratings updated on the basis of the
new evidence and data available?
• Are papers with evidence and supporting indicators of second-level results attached?

49. Challenges in reporting
• Your views!!!
• Is the format demanding – what project specific changes you would
recommend
• Is there an agency problem in reporting?