During this session we will:
*Review importance of monitoring and evaluation
*Share overview of grant model evaluation plan
*Review methodologies used in previous evaluations
*Share plans for future evaluation methodologies
2. CADRE TRAINING | 2
PRESENTERS
Philip Silvers
•Cadre Chairman
2014-15
•Past RI Director and
board liaison to the
Trustees
•44-year career as a
program evaluation
and survey research
professional
•10 years professor of
graduate-level
Research Methods
James Robinson
•Director, Strategy
Innovation and
Evaluation
•Former Division
Manager
Humanitarian Grants
and Future Vision
Pilot
•RI-USAID Steering
Committee
Korhan Atilla
•Cadre Technical
Coordinator 2008-
11
•Civil Engineer
•Conducted
sustainability check
site visits
•District Governor
Elect 2014-15, D-
2430
3. CADRE TRAINING | 3
Cadre’s Efforts to Support Sustainability
Phil Silvers
4. CADRE TRAINING | 4
The New Grants Model Is Working Fairly Well
– TRF Manuals Are Helpful
– Clubs and Districts are working together
– The Cadre Is an Important Resource
• Technical assistance
• Stewardship (accountability)
Proposal Writers and Sponsors are struggling
– Online application system is a challenge
– Effective needs assessments are not being done
– Sustainability—still an elusive concept
– Monitoring and Evaluation (M&E) Who will do it?
– Average time to grant approval: 3 months
YEAR ONE – WHAT DID WE LEARN?
5. CADRE TRAINING | 5
Core Effects:
Sustainable
Change/
Lasting
Results
Process
Results
(‘Doing Good’)
Training
Cultural
Experiences
Adventure
Travel
Friendships
Humanitarian
Supplies
Equipment
Externalities
(The ‘Sizzle’)
Challenges
Rotary
“Credits”
6. CADRE TRAINING | 6
• Real sustainability begins with the Community
Needs Assessment—by talking to the grass-roots people
before you come up with the solution.
– Need to comprehend their view of the world—their
fears, their challenges, their dreams for their kids
• Needs Assessment, Sustainability, and M&E are not
independent concepts—they must be integrated into one
piece by one team.
• Depending on what you are trying to accomplish, the
following slide gives you some options on measuring
Sustainability (or ‘red flags’ if you omit them)
BASIC PRINCIPLES OF SUSTAINABILITY
7. CADRE TRAINING | 7
Security—
•People
•Premises
•Equipment
Security—
•People
•Premises
•Equipment
Dimensions
of
Sustainability
Ecological
Human Capital Organizational
Systems
Behavioral Change Mechanical
Systems
New Technologies
(Tools) Cultural Ethos
Legacy
DIMENSIONS OF SUSTAINABILITY
Economic New Policy/
Legislation
Cultural Ethos/
Legacy
8. CADRE TRAINING | 8
Security refers to the protection against harm to
people, to premises, and to equipment. Other than
a good needs assessment, this is the most
fundamental issue in Sustainability.
Are the participants secure from coercion, threats to their reputation,
privacy, and physical/mental well-being?
Are the premises secure from unwanted intruders?
Are the equipment and materials safe from theft and vandalism?
9. CADRE TRAINING | 9
Economic refers to the financial viability of the
project over the long haul.
Are revenue streams planned to support the program after the grant
funding ends?
Operating costs?
Equipment maintenance and replacement?
Who will be responsible for managing the revenue streams?
Are contingency plans in place if the planned funding streams fail to
deliver?
Economic
10. CADRE TRAINING | 10
Cultural ethos refers to what is acceptable or not
acceptable behavior within society, e.g., smoking
in common areas, young girls’ access to
education, women’s role in government, getting
tested for HIV/AIDS, tribal enmities,
xenophobia.
Have project planners done a ‘force-field’ analysis of the factors
inhibiting and fostering change?
What rewards, incentives, modeling, or strategies might help shift
prevailing attitudes?
What various measures will reveal whether a change has happened?
Cultural Ethos/
Legacy
11. CADRE TRAINING | 11
SUSTAINABILITY
How Else Can You Say It?
Game changer
Systemic change
Lasting results
A permanent difference
Self-perpetuating
Continuity
Can continue without outside help
Can exist on its own indefinitely
The ‘buck’ does not stop here!
13. CADRE TRAINING | 13
EVALUATION PLAN OVERVIEW
M & E Activities
13-
14
14-
15
15-
16
16-
17
17-
18
18-
19
19-
20
20-
21
21-
22
Areas of Focus X X X
Strategic Partners X X X
Grant Model X X X
Sustainability Assessments X X X
Multi-year evaluation
Site visits for 5-10% of Global Grants
Study common project types in each area of focus
Replicate sustainability check methodology
15. CADRE TRAINING | 15
TERMINOLOGY
Monitoring typically refers to the routine and ongoing
process of collecting data as agreed upon before project
implementation.
Evaluation generally involves a post-project assessment
of the degree to which the project goals were met and the
impact the project activities had on beneficiaries.
A Measure is a quantitative or qualitative description of a
project’s activities or impact.
Slide 15
19. CADRE TRAINING | 19
Slide 19
Is monitoring and
evaluation data required
to close a global grant?
Yes. Sponsors must provide the minimum standard (at
least one measure per AoF) for M & E data in the final
report. This information is required to close the grant.
If Sponsors promised more M & E data than they can
provide at the time of reporting, can close the grant if
the minimum standard has been met.
How should M & E funds
budgeted in a global
grant be processed as
part of grant closure?
Sponsors can utilize up to 10% of the grant budget for
M & E.
Provide accounting of expenditures at time of final
report. If applicable, sponsors may provide a plan to
expend remaining funds post closure and report M & E
data into online system
GRANT REPORTING REQUIREMENTS
20. CADRE TRAINING | 20
CHARACTERISTICS OF GOOD M & E PLANS
1. Less is More
Rotarians should be modest in the number of measures
they will track…if it’s in the application, we’ll expect to
see it in the report.
Advise Rotarians to limit the number of areas of focus
they select in the application as this will multiply the
number of measures required.
Slide 20
21. CADRE TRAINING | 21
2. Measures match core project components
Rotarians do NOT need to measure every project
component. Instead, they should select the most
important components and focus on measuring their
success in those.
Measures should be relevant to the main goals of the
project and should indicate success in reaching the
goals.
Slide 21
CHARACTERISTICS OF GOOD M & E PLANS
22. CADRE TRAINING | 22
3. Mixture of output & outcome measures
Counting activities and participants is a good place to
start when developing an M/E Plan, but falls short of
demonstrating quality.
Strong M/E plans include measures that demonstrate the
quality of the project in terms of changes in the behavior
or knowledge of the beneficiaries.
Slide 22
CHARACTERISTICS OF GOOD M & E PLANS
23. CADRE TRAINING | 23
4. Reasonable attribution
– Measures should be closely and obviously connected to
activities conducted and the project participants.
– Grant officers can advise Rotarians on what is
reasonable attribution of project outputs to outcomes.
Slide 23
CHARACTERISTICS OF GOOD M & E PLAN
24. CADRE TRAINING | 24
5. Resources are dedicated to M/E activities
Eligible items include:
– Contracted labor of a local org or paying stipends to community members to
collect data.
– Consulting costs to technical experts to review and validate data.
– To purchase data sets and other information necessary to establish a
baseline.
– Purchasing equipment such as phones, GPS equipment, and laptops to
monitor activities.
– Local travel costs, such as car rental, fuel, and train fare to conduct site
visits.
– Cost of electronic surveys; meeting costs for focus groups for evaluation
purposes.
– Printing costs for manuals and other materials needed for data collection.
Slide 24
CHARACTERISTICS OF GOOD M & E PLAN
25. CADRE TRAINING | 25
5. Resources are dedicated to M/E activities
Ineligible items include:
– International travel to visit the project site.
– Accommodations, meals and per diems.
– Producing project promotional or recognition materials.
– Any measurement expenses that are incurred prior to the grant approval
date, such as the costs of conducting a community assessment.
Slide 25
CHARACTERISTICS OF GOOD M & E PLAN
27. CADRE TRAINING | 27
SUSTAINABILITY
For The Rotary Foundation, sustainability means:
ensuring that grants provide long-term solutions
to community needs that the benefiting
community can maintain after grant funding ends.
28. CADRE TRAINING | 28
Legacy Grants;
Giving A Man A Fish
SUSTAINABILITY
29. CADRE TRAINING | 29
Rotary
Foundation
ensures
Sustainable
impact of
goods or
services.
Clubs and Districts
ensure sustainable
impact of goods and
services
Individuals &
organizations benefit
from sustainable
impact of goods and
services
Community or
organization ensures
sustainable impact of
goods and services
TRF ensures
sustainable impact of
goods and services
SUSTAINABILITY
Global Grants;
Teaching A Man to Fish
31. CADRE TRAINING | 31
SUSTAINABILITY FRAMEWORK
Community
Needs/
Strengths
Materials/
Technology
Funding
• Community
Needs
Assessment
• Meeting
beneficiary
needs
• Involving
community
partners
• Confirm local
funding for
long-term
support
• Compensate
project
participants
to ensure
continuity of
work
• Purchase
equipment
locally
• Involve
community in
selecting
technology
• Train community
to operate and
maintain
equipment
32. CADRE TRAINING | 32
SUSTAINABILITY FRAMEWORK
Knowledge Motivation
Monitoring/
Evaluation
• Provide
training to
beneficiaries
• Create plan to
transfer
knowledge to
new
beneficiaries
• Collaborate
with local
organizations
with expertise
• Give incentives
to beneficiaries
to provide
ongoing support
• Identify leaders
committed to
outcomes
• Prepare the
community to
own the project
when funds
expended
• Develop
measurable
objectives
• Identify ways
to collect data
• Establish
baseline data
to track the
impact of the
project
33. CADRE TRAINING | 33
SUSTAINABILITY FRAMEWORK
Community
Needs/
Strengths
Materials/
Technology
Funding
Knowledge
Motivation
Monitoring/
Evaluation
High incidence
of waterborne
illness among
children under
5 in a village
39. CADRE TRAINING | 39
SUSTAINABILITY MATRIX
Methodology
Assessment is comprised of eight sections
Each section has 15-30 indicator questions
Each grant is assigned a score for each indicator within
the applicable sections
Score is between 1-4
Each grant is evaluated independently
40. CADRE TRAINING | 40
SUSTAINABILITY MATRIX
Grant Management
The Grant Management section focuses on overall project management
Key indicators include:
Does the project have precise and finite objectives?
Is there an implementation workplan including milestones and
benchmarks?
Were sustainable activities incorporated into the original project
design?
Were the required progress reports submitted on time?
Was a sustainability plan included in the final report?
41. CADRE TRAINING | 41
SUSTAINABILITY MATRIX
Measuring Outcomes
The Measuring Outcomes section focuses on the measurability of a
project, and whether or not appropriate M&E was conducted during and
after the project
Key indicators include:
Does the project have clear and measurable outcomes?
Does the monitoring system include community involvement?
Are the project outcomes shared with the community and other
stakeholders in ways meaningful for sustained project support?
42. CADRE TRAINING | 42
SUSTAINABILITY MATRIX
Financial
The Financial section focuses on budget planning and the financial role
of the cooperating organization
Key indicators include:
Does the projected budget indicate sufficient revenue to fund the
workplan?
Does the project Spending Plan (Budget) align with the
implementation workplan line items (Activity Plan)?
Does the project budget indicate sufficient revenue to fund the
sustainability activities beyond the end of the grant?
43. CADRE TRAINING | 43
SUSTAINABILITY MATRIX
Community
The Community section focuses on community involvement and buy-in
for the project
Key indicators include:
Was a community needs assessment completed? Were
recommendations made?
Does the workplan include activities to create or promote local
community involvement and ownership?
Do grant reports describe community relationships developed which
support the continuation of project outcomes beyond the life of the
grant?
Does the grant identify community champions willing to assume
leadership roles for sustaining project outcomes after the life of the
grant?
44. CADRE TRAINING | 44
SUSTAINABILITY MATRIX
Beneficiary Organization
The Beneficiary Organization section focuses on the project’s
relationship with the beneficiary organization and the organization’s
capacity to take ownership of the project
Key indicators include:
Do written agreements define the terms of cooperation, roles and
responsibilities of the beneficiary organizations?
Does the documentation identify personnel willing to assume
leadership roles for sustaining project outcomes?
Does the local Club/District demonstrate strong partnerships with
beneficiary organizations as appropriate to objectives?
45. CADRE TRAINING | 45
SUSTAINABILITY MATRIX
Cooperating Organization
The Cooperating Organization section focuses on the project
management plan and structure of the relationship between the project
sponsors and cooperating organization
Key indicators include:
Does the documentation include analysis and/or justification of
cooperating organization(s) selection based on demonstrated
expertise or experience?
Does the design integrate into the cooperating organization(s)
regular systems or operations?
Does the cooperating organization agree in writing to participate in
program financial reviews?
Does the documentation include a succession plan to hand over the
project to the partner organization(s) at the end of the grant lifecycle?
46. CADRE TRAINING | 46
SUSTAINABILITY MATRIX
Training
The Training section focuses on the training and follow up associated
with the project
Key indicators include:
Does the training plan address gaps identified in the assessment of
beneficiaries targeted for training (doctors, teachers, unskilled labor)?
Are the training course design (delivery and materials) locally and
culturally relevant (e.g., language, practical examples)?
Is there a measurement process to assess learner outcomes after
training?
Is there a plan for keeping skills up-to-date?
Has the project resulted in demand for training outside the targeted
beneficiary group (organization, individuals)?
47. CADRE TRAINING | 47
SUSTAINABILITY MATRIX SECTIONS
Software/Equipment
The Software/Equipment section assesses whether culturally appropriate
technology was used during the project
Key indicators include:
Does the documentation include a justification of why equipment is
needed?
Does the proposed equipment respect local social and cultural
norms?
Was the availability of equipment on the local market considered?
Is there an inventory management system?
Do user guidelines exist for equipment use?
Are local spare parts and replacement equipment available?
Is equipment being used as intended?
50. CADRE TRAINING | 50
AREA OF FOCUS EVALUATIONS – FUTURE VISION PILOT
Global Grants:
Sample of all Global Grants approved during pilot, 35 site
visits across 5 areas of focus Nov 2013- Feb 2014
Nine countries on four continents
Focused on VTT and Humanitarian Projects, not Scholarships
10 Cadre and 4 Area of Focus Managers
51. CADRE TRAINING | 51
Pilot Evaluation - Area of Focus Site Visits
Korhan Atilla
52. CADRE TRAINING | 52
WHAT IS SUSTAINABILITY?
(is it QUALİTY?)
AND WHAT IT IS NOT?
(is it only QUALİTY?)
53. CADRE TRAINING | 53
MATTERS OF SUSTAINABILITY
1- Financial Sustainability
2- Maintenance
3- Proper Operation
4- Precautions during installation
5- Rotarian Involvement
6- Benefitting Community Involvement