Programme Evaluation training slides Osvaldo Nestor Feinstein

Bu proje Avrupa Birliği ve Türkiye
Cumhuriyeti tarafından finanse edilmektedir

Programme Evaluation for
Strategic Management
Osvaldo Néstor Feinstein
Program Evaluation Workshop
Ankara, 28-29 November 2013

Cumhuriyeti tarafından finanse
edilmektedir

Themes

1. Programme Evaluation and Strategic Management
2. Performance Management & Rating systems
3. Institutionalization of Evaluation in the Public
Sector
4. Assumptions Based Programme Evaluation
Framework

2

edilmektedir

Programme Evaluation and the
Strategic Management Cycle

• Policy Review 
• Strategic Planning and Selection of
Interventions/strategies
• Implementation (delivery of strategies)
• Performance reporting /Monitoring
• Performance auditing/EVALUATIONWider
PROGRAMME EVALUATION
• Policy Review 

edilmektedir

Programme Evaluation and
Effective Governance

• Public programmes are interventions that attempt to
achieve results, outputs, outcomes and impact,
using public resources.
• Are government interventions achieving their
intentions? Are they producing the intended results?
Can they be improved?
• Programme Evaluation provides answers to these
questions, contributing to effective governance.

edilmektedir

Relations between
Monitoring and Evaluation

• MONITORING: observing and collecting data on the implementation
of a programme, including inputs, outputs and outcomes, &
contextual factors
• EVALUATION: on the basis of monitoring and other data, assessing
in a systematic way programme effectiveness ( achievement of
outputs & outcomes),efficiency, relevance and sustainability
• Evaluation uses monitoring data and evaluation requirements
determines which data should be monitored ME & EM

edilmektedir

Performance monitoring and reporting and their
relation to evaluation

• Performance monitoring is a continuous process
of collecting and analyzing data to compare how
well a programme or policy is being implemented
against expected results
• Performance reporting is an instrument to convey
the results of performance monitoring
• Performance monitoring and reporting are inputs for
Programme Evaluation, which includes an
assessment of programmes.

edilmektedir

Differences with Auditing and Investigation

• AUDITING verifies if resources were allocated for
their intended uses. It does not assess objectives,
which takes as a given.
• INVESTIGATION assesses the concerns and
complaints with respect to the program
implementation, for example, in violating rights of
minorities or allegations of fraud.

edilmektedir

Key purposes of evaluation

• Evaluations for accountability: to show for which
purposes resources were used. Were the
programme‟s objectives (expected results)
achieved? Ex post
• Evaluations for learning: to draw lessons from
experience and apply them to improve
interventions. What knowledge or lessons can be drawn from
the program‟s experience? (Janus)

edilmektedir

Programme Evaluation as a tool to
learn from experience
Looking to the future, with a view to the past

THE PAST

THE FUTURE

edilmektedir

Program Evaluation and
Strategic Decision Making

• Decisions concerning
– which programs to close or cut, on the basis of
performance, results achieved or not achieved
(accountability function of evaluation)
– which programs should be supported, and/or improved,
in light of results achieved and strategic orientations
(learning function of evaluation)

• Decisions influenced by political factors and by
evidence on results provided by evaluations

edilmektedir

Programme evaluation and strategic
planning

• Decisions among alternative ways (technologies) to
achieve programme‟s objectives . Ex-ante use of
evaluation (for example, cost benefit analysis)
• Identifying contextual factors that may require
programme adaptation
• Improving stakeholder coordination during
preparation of strategic plans

11

edilmektedir

Program evaluation and
public management reform

• Program Evaluation (PE) as an instrument of public
management reform, to improve its implementation
by drawing lessons from the implementation
experience.
• PE provides evidence on effective and ineffective
programs so as to guide resource allocation (but
beware with mechanical links)

12

edilmektedir

Programs, projects and policies

• Projects: achieving objectives (e.g. reducing rural poverty) through a set of
instruments or components (e.g. microfinance)
• Programmes: set of projects or projects with several components. They include

objectives, resources and activities Programmes to be evaluated could be
interventions included in Institutional Strategic Plans.
• Policies: objectives that are implemented through a set of programmes or projects
• POLICIES  PROGRAMMES  PROJECTS

SOME EXAMPLES AMONG THE 25 PROGRAMS
INCLUDED IN THE 10th NATIONAL DEVELOPMENT PLAN

edilmektedir

• PROGRAM ON REDUCING THE INFORMAL ECONOMY
• THE COMMERCIALIZATION PROGRAMME IN PRIORITY
TECHNOLOGY AREAS
• IMPROVING THE HEALTH TOURISM PROGRAM
• TRANSFORMATION PROGRAM FROM TRANSPORTATION
TO LOGISTICS
• URBAN TRANSFORMATION PROGRAM ENHANCING
COMPETITIVENESS AND SOCIAL COHESION
• INCREASING THE EFFECTIVENESS OF THE LABOUR
MARKET PROGRAM

14

edilmektedir

Programme evaluation and the
evaluation of the NDP

• Turkey‟s 10th National Development Plan indicates that the
coordination of monitoring and evaluation of the progress of
the Development Plan will be performed by the Monitoring
and Direction Committee of the Development Plan, which will
be headed by the Undersecretary of the Ministry of
Development and comprised of high level administrators from
related ministries.
• In the process of planning and implementing new
investments, the NDP states that there is a need for
enhancing the capacity of public institutions in preparation,
implementation, monitoring and evaluation issues
• Discussion question: could it be convenient for strategic
management to separate evaluation from monitoring?

edilmektedir

Strategic Plans and Programme
Evaluation

• Strategic Plans include programmes although sometimes without
calling them “programmes” (like the French gentleman who spoke
prose without knowing it). “Implicit programmes”. Some examples
– SP of the Ministry of Energy, its “Aims” can be evaluated as
programmes (ex. “providing diversity in resources by giving priority to
domestic resources)
– SP of the Ministry of Justice, its “Objectives” can be evaluated as
programmes

• Performance indicators of strategic objectives are usually outputor input level indicators. Outcome indicators are rarely applied. In
many cases, performance indicators are measured by the level of
required inputs
• For discussion: how could better performance indicators support
strategic management?
16

edilmektedir

Strategic Plans and Monitoring &
Evaluation

• Monitoring and evaluation is one of the least
developed components of SPs among the pilot
institutions
• Performance Plans do not include any section for
the follow up and evaluation of the SP
implementation.
• For discussíon: why is it important for strategic
management to include an appropriate section
dealing with evaluation?
17

edilmektedir

•
•
•
•
•
•

Different types of evaluation
Ex-ante evaluation (appraisal)
Process evaluation
Mid-term evaluation
Completion evaluation
Ex-post evaluation – Impact evaluations
Formative (to improve)  Summative (to judge)
– “when the cook tastes the soup, that's formative;
– when the guests taste the soup, that's summative“
M.Scriven
18

Costs and Benefits of different types of
evaluation

edilmektedir

• Impact evaluations more expensive and less timely
for decision making
• Higher benefits of evaluations when their timing is
linked to decisions that will have to be made at
certain moments or periods
• Choose approaches that will allow to deliver the
evaluation in time for decision making
See “Monitoring and Evaluation: Some Tools, Methods and Approaches”

http://www.worldbank.org/ieg/ecd/tools/

edilmektedir

•

•
•
•

•

Public expenditure tracking surveys
(PETS)
Public expenditure tracking surveys (PETS) track the flow of public funds and

determine the extent to which resources actually reach the target groups. The
surveys examine the manner, quantity, and timing of releases of resources to
different levels of government, particularly to the units responsible for the delivery of
social services such as health and education (“follow the money”).
PETS are often implemented as part of larger service delivery and facility surveys
which focus on the quality of service.
Can be used to diagnose problems in service delivery quantitatively, to identify best
practices and to provide Providing evidence on delays, “leakage,” and corruption.
Supports the pursuit of accountability when little financial information is available
and improves management by pinpointing bureaucratic bottlenecks in the flow of
funds for service delivery.
http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPME
NT/EXTPCENG/0,,contentMDK:20507700~pagePK:148956~piPK:216618~theSite
PK:410306,00.html for more details, examples and other methodologies

20

edilmektedir

Cost-Benefit and Cost-Effectiveness
Analysis

• Tools for assessing whether or not the costs of a programme can be
justified by its results. Cost-benefit analysis measures both inputs and
outputs in monetary terms. Cost-effectiveness analysis estimates inputs in
monetary terms and outcomes in non-monetary quantitative terms (such as
improvements in student reading scores).
• Can be used to inform decisions about the most efficient allocation of
resources and/or to identify programmes or projects that offer the highest
rate of return on investment.
• Useful for convincing policy-makers and funders that the benefits justify the
programme but data for cost-benefit calculations may not be available, and
results may be highly dependent on assumptions made.
• http://ec.europa.eu/regional_policy/information/evaluations/guidance_en.cf
m

21

edilmektedir

Different evaluation
approaches/techniques

• Objectives based evaluation
• Goal-free evaluation
• Qualitative methods
• Quantitative methods
• Mix of quantitative and qualitative methods
• Triangulation
22

edilmektedir

Evaluation Criteria and
Evaluation Questions

• Relevance: were the objectives consistent with
beneficiaries‟ requirements, organizational and
country priorities & policies?
• Effectiveness: to which extent the program‟s
objectives were achieved, or are expected to be
achieved?
• Efficiency: how economically resources/inputs
(funds, expertise, time, etc.) were converted into
results? how reasonable were the costs?
23

edilmektedir

Four additional evaluation criteria

• Sustainability: will the results achieved be durable?
Is it likely that those results will be sustainable?
• Innovativeness: were innovative approaches
introduced by the program?
• Replicability and Scaling-up: to what extent the
program‟s approaches have been (or are likely to
be) replicated and scaled up?
• Equity: how equitable by gender, social group (or
other attributes) is the distribution of benefits?
24

edilmektedir

Case study exercise (I)
Evaluation of the Northern Someland Rural
Development Program

• With support from the XFund, the 10,000,000 euros Northern
Someland Rural Development Program aimed at providing
low prized fertilizers to 5,000 farmers in the northern region of
Someland, a land-locked country, where small farmers are
fruit producers.
• The objective of the program was to improve the welfare of
the farmers and their families. A program implementation unit
was set up to organize the delivery of fertilizers and to
establish links between the farmers and local institutions
involved in the provision of complementary services.
• How would you evaluate this program? What would you do
to evaluate the program? Please discuss with your
colleagues and prepare a 5 minutes presentation
25

edilmektedir

Alternative approaches to
program evaluation

• Goal-free evaluation
– Its strength in avoiding a limited focus on objectives can
be compensated by objectives-based evaluation with an
explicit focus on “unintended consequences”
– Weak on accountability.

• Impact evaluation with control groups
– Explicit counterfactual to model the without program
situation

26

edilmektedir

Counterfactual in evaluation

• What would have happened WITHOUT the
program? (had there been no program)
• Note that the program is “factual”, it is a fact, as a
program was implemented, whereas the
counterfactual is a situation that is not observed, it
has to be reconstructed.
• Explicit and implicit counterfactuals, not theoretical
but practical importance, rationale for the program.

27

edilmektedir

Comparisons
Before/ After With/Without
Without

With

Before

100

100

After

150

120

Look first only at the “with” column

28

edilmektedir

Comparisons
Before/ After With/Without
Without

Before
After

With

100

100

80

100

Look first only at the “with” column

edilmektedir

Establishing the “Before” & “Without”

• “Before” , or at the beginning of the program. Need
for a BASELINE, which could be part of the
diagnosis. Sometimes available census and/or
household surveys can be used.
• “Without”, requires a comparison group; “control
group” is a particular case. Statistical methods.
Comparative analysis.

30

edilmektedir

The “General Elimination Method”
GEM

• Simple and powerful, introduced by Michael Scriven in
2008
• Two key steps:
1. consider the set of “possible plausible causes” (including
components of the contexts and/or other programs or policies)
2. eliminate the less plausible possible causes

• See application and references in p.3 of
• http://www.undp.org/evaluation/documents/ADR/ADR_
Reports/Chile/ADR-Chile.pdf
31

How to manage and conduct program
evaluations

edilmektedir

• The first strategic decision is choosing WHAT to
evaluate. Decide (if you have degrees of freedom) on
the basis of expected value added for accountability
and/or learning
• The terms of reference should clarify expectations
concerning the evaluation that will be carried out.
Answers to WHY, HOW, and WHEN TO EVALUATE?
– See “Writing Terms of Reference for an Evaluation: A HowTo Guide”
– https://ieg.worldbankgroup.org/evaluations/evaluationcapacity-development-ecd
32

edilmektedir

Which type of evaluation should be
used?

• Selecting the type(s) of evaluation to be used
• Methods adequate to the circumstances (purpose,
availability of time and resources). Not only “fit-topurpose”. Pragmatic
• See “Monitoring and Evaluation: Some Tools,
Methods and Approaches”
• http://www.worldbank.org/ieg/ecd/tools

33

edilmektedir

Consultation with stakeholders

• Consulting with stakeholders
– Why? To get their insights, and
– to nurture their interest in the evaluation (which will
facilitate the use of the evaluation).

• Identifying key stakeholders:
– Partners, intended beneficiaries, government officers,
civil society representatives

34

edilmektedir

Reviewing relevant documentation

• In addition to PROGRAMME DOCUMENTS, reports
on the country, internet searches for thesis. other
evaluations and papers on the project/program area
and on the instruments used (e.g., rural
microfinance or agricultural extension in the area)

35

edilmektedir

APPLYING Evaluation Criteria

• Applying evaluation criteria
– Relevance, Effectiveness, Efficiency, Sustainability,
Innovativeness, Replicability and Scaling-up,
Performance of partners
– Ratings for the criteria
– Rating overall project achievement
– Provide a justification for the ratings

36

edilmektedir

Use of evaluation reports

• Using evaluation reports
– Evaluation reports are important INPUTS for
evaluations, as indicated before under “reviewing
documentation”.
– Evaluation reports are also key OUTPUTS of the
evaluation process. Show in a clear way the links
between findings and recommendations to facilitate the
use of the evaluation report.

37

edilmektedir

Case studies

• For in-depth understanding of how a program has
operated in a specific context
• Problem of eventual lack of representativeness
• Typology of contexts/situations, and case studies
corresponding to the different contexts/situations.
• Combining case studies with surveys

38

edilmektedir

Use of surveys

• Search for available surveys
• If surveys are needed and are not available, ensure
that professionals with the required expertise are
involved; possibility of subcontracting
• The methodology ACTUALLY used should be
described in detail (for replication & interpretation)
• Pilot trials are important

39

edilmektedir

Semi-structured interviews

• Identify key informants corresponding to different
perspectives
• Elaborate a brief interview guide
• Team should be opened to comments, opinions and
judgements not considered in the guide
• Systematize the results of the interviews

40

Case study exercise (II)
Evaluation of the Youth Employment Program for the Eastern
Region

edilmektedir

• Given the importance of youth unemployment a
program to increase the employability of the young was
designed and implemented in the Eastern region of the
country. The program included training in agricultural
activities, as well as training courses for the production
of furniture, which is an important economic activity in
that region. In addition there were also training courses
on informatics. An evaluation team was tasked to
evaluate the program.
• Discuss with your colleagues how you would conduct
the evaluation of the program. Prepare a brief
presentation with the conclusions of your discussion.
41

edilmektedir

e-reference materials

• Glossary of Key Terms in Evaluation and Results Based
Management
– http://www.oecd.org/dac/evaluation/evaluationandaideffectivenes
sseries.htm
– TURKISH VERSION:
– http://www.oecd.org/dac/evaluation/deerlendirmevesonuodaklyon
etimdeanahtarterimlersozluu.htm
– Monitoring and Evaluation: Some Tools, Methods and
Approaches

• http://www.worldbank.org/ieg/ecd/tools
• Impact Evaluation in Practice (2011) Eng/Fr/Sp.
– http://siteresources.worldbank.org/EXTHDOFFICE/Resources/54
85726-1295455628620/ Impact_Evaluation_in_Practice.pdf
42

edilmektedir

E-evaluation guides from the EC

• On the evaluation of innovations
• http://ec.europa.eu/regional_policy/sources/docgener/e
valuation/pdf/eval2007/innovation_activities/inno_activit
ies_guidance_en.pdf.
• On impact assessment
http://ec.europa.eu/governance/impact/commission_gui
delines/commission_guidelines_en.htm
• Several guides can be accessed at
• http://ec.europa.eu/regional_policy/information/evaluati
ons/guidance_en.cfm
43

edilmektedir

•
•
•
•
•

Performance Management
Performance Management,
Performance Measurement,
Performance Monitoring
Measure to Monitor & Monitor to Manage
Two frequent (and partially deceiving) quotes:
– “What gets measured gets done” Tom Peters
– “What gets measured gets managed” Drucker

• Measurement facilitates Management
44

edilmektedir

Performance
measurement and management

• Performance measurement
– What is happening with the programs?
– How are the programs performing?
– Are programs achieving their expected results? (the
“why” question answered by evaluations)

• Performance management
– making decisions about programs and project portfolio:
allocation of additional resources, stopping programs,
lessons for new programs, etc.
45

edilmektedir

PM in practice & Exercise III

• What should be measured and managed?
• Comparing what is actually measured with what should
be measured, which are the expected results?
• The results chain:
inputs--(processes)outputsoutcomes  impacts
• Exercise III: QUESTIONS FOR DEBATE:
• Are inputs “results” in the results chain? And outputs?
• What are the implications for accountability of different
definitions of results?
46

edilmektedir

Reviewing a portfolio of programs

• Identification of “PM gaps” (expected vs actual)
• In which (type of ) programs is performance higher
(lower) than expected?
•
•
•
•

What worked? What did not work?
Why? (an important evaluation question)
What influence had the context ?
Any patterns?
47

edilmektedir

Purposes of PM

• Accountability and learning (improvement)
• Whose accountability?
• Whose learning?
• Complementarity or trade-offs?

48

edilmektedir

Lessons on PM Systems (PMS)

• Importance of ownership of the PMS
• WHO are the intended users of PMS?
• Convenience of involving intended users in the
design and/or redesign of the PMS
• Opportunities created by PMS to facilitate
coordination and enhance effectiveness
49

edilmektedir

Lessons on PM Risks

• Nonuse and/or misuse (GAO‟s findings of nonuse
of performance information to manage; commitment
by leaders and communication of that commitment
is crucial)
• Mismeasuring and/or not measuring, not counting
what counts, leaving out what is important
• Inducing short-termism
• Bureaucratization, with PM being perceived as a
mechanical exercise, as a burden
50

edilmektedir

Issues of PM

• Participation of intended PM users in the review of
the system
• Programs, projects and portfolio, the trees and the
forest
• Partial achievement: PM when reporting period
does not correspond to “gestation period”. Actual
costs and likely benefits.

51

edilmektedir

Year

Issue of Partial Achievement

2011

Output
(reported
in year…)
2A

2012

2A

6,000

3,000

Total

4A

16,000

4,000

52

Cost
$

Unitary
cost $/A

10,000

Revised

5,000

6,000
A:2-2x.5

edilmektedir

Issues of PM

• PM and its use for resource allocation,
performance informed budgeting
• Quality control of data
• Performance Measurement & Management
• Links of PM with the M&E system
• PM, comparisons and benchmarking (with
baselines, targets, other organizations)

53

edilmektedir

•
•
•
•
•
•

Rating Systems
Trending theme: S&Poor
Why ratings?
Experience of WB & IFAD - ECG
Validation of self-evaluation ratings,
Evaluation of ratings, rating the raters
Contestability (dealing with disagreements)

54

edilmektedir

Program or Project Ratings

Relevance of anticipated outcomes
Effectiveness (achievement of outcomes)and
Efficiency on a six-point rating scale
Highly Satisfactory, Satisfactory, Moderately
Satisfactory.
• Moderately Unsatisfactory, Unsatisfactory., Highly
Unsat.
• (risks to) sustainability of outcomes based on an
assessment of four dimensions: financial, sociopolitical,
institutional framework & governance, and
environmental
•
•
•
•

55

edilmektedir

A six- point rating system for the
evaluation criteria

• A six-point rating system for the evaluation criteria
as a way to quantify qualitative judgements of
evaluators
– 6: highly satisfactory 5: Satisfactory
– 4: moderately satisfactory
– -----------------------------------------------– 3: moderately unsatisfactory
– 2: unsatisfatory
1: highly unsatisfactory

56

Ratings for innovation, scaling-up and
partners‟ performance

edilmektedir

• Good practice of IFAD‟s evaluation office providing explicit ratings for the
creativity, innovation and up-scaling potential of innovative projects and
distinguishing between agency and partners‟ performance ratings and
outcome ratings
• Promotion of pro-poor innovation,replication and scaling up: The extent to
which IFAD development interventions have: (i) introduced innovative
approaches to rural poverty reduction; and (ii) the extent to which these
interventions have been (or are likely to be) replicated and scaled up by
government authorities, donor organizations,the private sector and others
agencies.
• Performance of partners: assess the contribution of partners to project
design, execution, monitoring and reporting, supervision and
implementation support, and evaluation. The performance of each partner
will be assessed on an individual basis with a view to the partner‟s expected
role and responsibility in the project life cycle.

57

edilmektedir

Remembering the
performance evaluation criteria

• Relevance : The extent to which the objectives of a
program are consistent with beneficiaries‟ requirements
and country needs, institutional priorities and policies.
• Effectiveness: The extent to which the program‟s
objectives were achieved, or are expected to be
achieved, taking into account their relative importance.
• Efficiency: A measure of how economically
resources/inputs (funds, expertise, time, etc.) are
converted into results.
58

edilmektedir

Clarifications

• A satisfactory outcome means that relevant project
objectives are expected to be achieved with no, minor
or moderate shortcomings at the time of evaluation.
• A likely sustainability rating means that the project is
considered likely to generate continued benefits after
project implementation - with no or only moderate risks.
• (definition of sustainability: The likely continuation of
net benefits from a program beyond the phase of
external funding support. It also includes an
assessment of the likelihood that actual and anticipated
results will be resilient to risks beyond the program‟s
life).
59

edilmektedir

SMART criteria for indicators

• Specific relating results to the achievement of one
and only one objective.
• Measurable. all parties agree on what indicators
cover and on practical ways to measure them
• Achievable feasible targets
• Realistic levels of performance that are ambitious
but feasible..
• Time-Bound, not open ended but with dates
60

edilmektedir

Use (and Misuse) of ratings

• As an input, but not the only input, for decisionmaking, for performance management
• For comparisons, benchmarking
• For assessing and enhancing learning & quality
• For aggregation of information

61

edilmektedir

Issues concerning Ratings

• Identifying critical dimensions
applying “Occam‟s razor”
d1, d2, d3, …., dn
• Weighting the dimensions
w1d1, w2d2, w3d3
• Partial orders
limits to aggregation: partial comparisons
(Amartya Sen);
• Transparency (avoiding “secret algorithms”)
62

edilmektedir

Synthesis

• Performance management
1.
2.
3.

Measurement, availability of performance information is not enough
(possible nder-use, non-use and misuse);
Focus on expected results, identify PM gaps and patterns, present data with
a brief narrative
Involve intended users in (re)design of PMS

• Rating systems
1.
2.
3.

Different approaches to ratings
Validation of ratings
Transparency of ratings and benchmarking

• Links between performance management and rating systems
1.

Ratings provide management with comparative data on performance that
can be used for aggregation, learning and accountability

63

edilmektedir

Exercise IV: Case study example
Performance Measurement System for a Health Program

• A country wide health program was launched with
the objective of contributing to improve primary
health care of the population in urban and rural
areas. Different types of health interventions were
considered. You have been asked to develop a
performance measurement system for such a
program.
• Discuss with your colleagues and prepare with your
group a 5 minutes presentation.
64

edilmektedir

Learning from others

World Bank Independent Evaluation Group (IEG)
http://ieg.worldbankgroup.org/content/ieg/en/home.html
Project perf.assessment reports(PPAR); M&E
http://www.ifad.org/evaluation/process_methodology/do
c/manual.pdf
• http://www.thegef.org/gef/eo_office
• https://wpqr1.adb.org/LotusQuickr/ecg/Main.nsf/$defaul
tview/0906CAABDA17E70A48257731002A0656/$File/
PublicSectorGPS-revised.pdf?OpenElement
• (GPS Evaluation Cooperation Group)
•
•
•
•

65

edilmektedir

Additional reference materials

• DAC/IEG Sourcebook for evaluating global and regional programs
http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTGLOREGPAR
PROG/0,,contentMDK:21178261~menuPK:4426473~pagePK:64829573~pi
PK:64829550~theSitePK:4426313,00.html
• Program Assessment Rating Tool (PART) US
Gov.http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTEVACAP
DEV/0,,contentMDK:22321269~menuPK:4585748~pagePK:64829573~piP
K:64829550~theSitePK:4585673,00.html
Note: PART has been
discontinued but it‟s worthwhile
• A critique of the PART in
http://www.businessofgovernment.org/report/performance-managementrecommendations-new-administration
• Up-to-date information on the US gov. case:
http://www.businessofgovernment.org/blog/business-government/ombunveils-its-performancegov-website

66

edilmektedir

And some other useful references

• US (speech by OMB A.Director)
http://www.state.gov/s/d/rm/rls/rm/2010/147424.htm
• http://web.hbr.org/email/archive/managementtip.php?date=060611
(brief note)
• UNEG Good Practice Guidelines for Follow up to Evaluations
• http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=
610
• http://www.eipa.eu/files/repository/eipascope/20101022101218_Ei
pascope_2010_2_Article2.pdf
• Patria de Lancer Julnes et.al. (2008) “International Handbook of
Practice-Based Performance Management” Sage
• http://www.sustainability.com/library/rate-the-raters-phase-three

67

Institutionalization of Evaluation in the
Public Sector - Demand

edilmektedir

• Demand for evaluation
– from the Executive (Prime Minister Office, Ministry of
Development, MOF, Other M.)
– from the Legislative , Parliament, from Civil Society

• Types of demand:
– Actual demand
– Potential demand (there is an interest on evaluation but
funds are lacking)
– Latent demand (there is no awareness of the role of
evaluation but an interest on what evaluation can provide)
68

Institutionalization of Evaluation in the
Public Sector - Supply

edilmektedir

• Actual Supply of Evaluation
– Public sector managing the evaluations
– Public sector conducting evaluations
– Public sector outsourcing evaluations

• Potential Supply of Evaluations
– (organizations or professionals so far not involved in
evaluations but with capacities which can be used for
evaluation)

• Private consultants, universities, think tanks
conducting evaluations
69

edilmektedir

Institutionalization of Program
Evaluation in the Public Sector

• Who would lead a program evaluation initiative in
the public sector?
• A central Ministry, like the Ministry of
Development in Turkey, the Ministry of Planning or
of Finance in other countries
• Who will manage, conduct and use program
evaluations? Sector Ministries (Health, Education,
Labor…)

70

edilmektedir

International Experiences in Program
Evaluation

• Frequently the experiences of New Zealand, USA,
Canada, the Nordic countries and the UK are
highlighted. We will make reference to another
OECD case more relevant for Turkey: the case of
MEXICO and some other relevant experiences
• Mexico has developed a well functioning system of
evaluation of social programs, introducing some
innovations that are worthwhile to take into account

71

edilmektedir

The Mexican Experience on social
program evaluation

• The Mexican system for social program evaluation
is described in
http://www.coneval.gob.mx/Evaluacion/Paginas/Eva
luation-and-monitoring-en.aspx
• The National Council for the Evaluation of Social
Development Policy (CONEVAL), as set forth in the
General Law of Social Development, has the aim to
regulate and coordinate the evaluation of the Social
Development Policy and Programs implemented by
public dependencies.
72

edilmektedir

More on the Mexican Evaluation
Experience

• Mexico has also been developing evaluation
systems at the subnational level, with support from
CONEVAL, and the capital city of Mexico developed
an evaluation system on its own.
• For experiences of institutionalization of impact
evaluation see
www.worldbank.org/ieg/ecd/docs/inst_ie_framework
_me.pdf

73

edilmektedir

The experience of Austria

• From a recent diagnosis of Austrian Public
Administration: input rather than output (or
outcome). Concern with who gets how much,
instead of what has to be the outcome
• Shift in 2009 towards an outcome orientation.
Managing public adminstration based on
contributions to achieve outcomes (performance
management)

74

edilmektedir

The experience of Malaysia

• Malaysia public sector management reform shaped
by best practices
• Framing a reform as „best practice‟ was a way of
neutralizing potential opposition from the very
different and sometimes antagonistic stakeholders
that the reform had to satisfy
• Successive reforms have built on previous ones:
past failures have been the paradoxical foundation
for present success
75

edilmektedir

More on the Malaysian experience

• The current prime minister, or his deputy, has
individual meetings with the ministers responsible
for the seven national key result areas every month,
and with the rest of the Cabinet every six months,to
review their performance against key performance
indicators (KPIs) set by the prime minister himself
• Need to emphasize the importance of the
implementation stage of reform, and of an
appropriate diagnosis
76

edilmektedir

The evaluation of Turkey‟s Vocational
Training Programs

• The evaluation of Turkey‟s Vocational Training
Programs sought to assess the effectiveness of
improving vocational training programs in the context of
the rapid expansion of these programs in Turkey.
• Concept Note, Baseline Report and
• Evaluation Report available at
• http://web.worldbank.org/WBSITE/EXTERNAL/EXTAB
OUTUS/ORGANIZATION/EXTHDNETWORK/EXTHDO
FFICE/0,,contentMDK:23177755~pagePK:64168445~p
iPK:64168309~theSitePK:5485727,00.html
77

edilmektedir

Exercise V: Debate on the
Institutionalization of Evaluation

• Discuss the institutionalization of evaluation in the
public sector of Turkey and rate demand for and
supply of evaluation, as well as the links between
demand and supply of evaluation, and the use of
evaluation on:
– a four point rating scale, with 1
– as the lowest rating and 4 the highest.

• Demand:

Supply:

78

Links:

Use:

edilmektedir

Summing-up: 12 steps to conduct
program evaluation

1. Identify the program‟s objective (s)
2. Become familiar with the modalities of implementation
3. Identify the basic assumptions (the logic of the
intervention)
4. Decide which will be the key indicators (could be
those included in the program, eventually adding
some more)
5. Search and read bibliography and review the first four
steps in light of your reading
79

edilmektedir

Steps for conducting program
evaluation (continued)

6. Search for and review available information
sources.
7. Determine needs of additional information, if any
8. Decide which methods/techniques to use
9. Apply the evaluative criteria

80

edilmektedir

Steps for conducting program
evaluation (final)

10. Be open to the possibility of unintended
consequences, positive and/or negative
11. Analyse and synthesize results by implementation
modality and by area of implementation, identifying
best and worst
12. Triangulate results (cross-checking different
information sources)

81

edilmektedir

The end?

“This is not the end,
nor the beginning of the end,
but the end of the beginning”
- Winston Churchill

82

edilmektedir

Thank you!

teşekkür ederim
ofeinstein@yahoo.com

83

Programme Evaluation training slides Osvaldo Nestor Feinstein

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (11)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Programme Evaluation training slides Osvaldo Nestor Feinstein

Ähnlich wie Programme Evaluation training slides Osvaldo Nestor Feinstein (20)

Mehr von stratejikyonetisim

Mehr von stratejikyonetisim (8)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Programme Evaluation training slides Osvaldo Nestor Feinstein