1) The document discusses results monitoring for Phase I and Phase II of the FLEG program. It outlines challenges in Phase I monitoring due to a lack of measurable indicators and baseline data.
2) For Phase II, a detailed results framework was developed with indicators to track progress at regional, national and sub-national levels. The framework includes project development objectives (PDO) and intermediate results indicators.
3) Reporting for Phase II will include country-level, organizational-level and program-level reporting according to set frequencies and guidelines. This will allow monitoring of progress toward intended objectives.
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Nina rinnerberger results monitoring and reporting
1. 1st Steering Committee Meeting
Minsk, October 1-3, 2013
Results Monitoring at Program and
Country Levels
2. Overview
1. Results Monitoring & Challenges in
Phase I
2. Lessons Learned & Results Framework
for FLEG II Program
3. Reporting on Program & Country Level
3. Results Monitoring - FLEG I
• FLEG I Program monitored progress for
seven results areas
– Tracked results in each country and regionally
– Based on qualitative assessments by the country
teams and anecdotal evidence
• No results framework with measurable
indicators was developed
4. Challenges
• How to monitor the impacts of FLEG Program
interventions?
– With hundreds of activities implemented, there is anecdotal
evidence that the impact of FLEG I has been large
– No baseline survey conducted
– Progress towards higher level outcomes is difficult to judge
quantitatively
E.g., development of forestrelated educational materials in
Azerbaijan
(“Young Foresters‟ School”)
6. Lessons Learned
Robust and comprehensive results monitoring
is key to measuring impacts and judging higher
level impacts quantitatively
7. Measuring Results for FLEG II
• New Feature for Phase II:
Detailed Results Framework developed
• Indicators track progress at regional, national, and subnational level
• Responsibility for monitoring and data collection: PCTs
and/or PMT
8. Types of Indicators
• Project Development Objective (PDO)
Indicators
– 3 Core Sector Indicators (World Bank-wide)
– 4 Custom Indicators (FLEG II Program specific)
• Intermediate Results Indicators
– 7 Custom Indicators
(FLEG II Program specific)
9. Project Development Objective Indicators
Core Sector Indicators (CSI)
1. Reforms in forest policy, legislation or other
regulations supported
2. Government institutions provided with capacity
building to improve management of forest
resources
3. Forest users trained (by gender and ethnicity)
10. Reforms in forest policy, legislation or other
regulations supported
• Linked with PDO 1 (regional level): make progress
implementing the 2005 St. Petersburg FLEG Ministerial Declaration
in the participating countries and support the participating
countries commit to a time-bound action plan to ensure its
implementation and follow-up activities.
– Measures whether a project has supported forest
sector reforms (Yes/No)
• General ongoing policy dialogue with stakeholders
should not be included
– Baseline: „No‟
– Frequency: annual
Core Sector Indicator 1
11. Government institutions provided with capacity building to
improve management of forest resources
• Linked with PDO 1 (regional level)
– Covers capacity-building aimed at strengthening
forest administration institutions to deliver services to
the forest sector
• Refers to the number of national or sub-national
institutions that have received capacity building
• Targeted institutions may be outside of the forestry
sector
– Baseline: „zero‟
– Frequency: annual
Core Sector Indicator 2
12. Forest users trained
• Linked with PDO 1 (regional level)
• Refers to the number of forest users and
community members that have received
capacity building through training
– 2 Sub-indicators by:
• gender (female)
• ethnic minority/indigenous people
– Baseline: „zero‟
– Frequency: annual
Core Sector Indicator 3
13. Custom Indicators (PDO level)
Results Framework
Project Development Objective Indicators
Cumulative Target values
Indicator Name
Credible process
toward the
implementation of the
St. Petersburg
declaration launched
Understanding and
implementation of
FLEG principles by
forest practitioners and
other stakeholders
improved
Core
Unit of
Measure
Percentage
Responsibility
Data Source/
Baseline
End Frequency Methodology for Data
YR1 YR2 YR3 YR4
Collection
Target
0.00
Percentage x% Based
of
on survey
practitioners results
100
annual
Program
reports
PCT/PMT
TBD
Twice,
baseline
and final
surveys
Repeated
perception
surveys of key
decision
makers
PMT
14. Custom Indicators (PDO level)
Results Framework
Project Development Objective Indicators
Cumulative Target values
Indicator Name
Uptake of best practice
models on sustainable
forest management
Increased awareness
of decision makers of
modern technology
and information to
improve forest law
enforcement and
governance
Core
Unit of
Measure
Yes/No
Responsibility
Data Source/
Baseline
End Frequency Methodology for Data
YR1 YR2 YR3 YR4
Collection
Target
No
Percentage x% Based
of Decision on survey
Makers
results
Yes
annual
Program
reports
PCT/PMT
TBD
Twice,
baseline
and final
surveys
Repeated
perception
surveys of key
decision
makers
PMT
16. Intermediate Results Indicators
Results Framework
Intermediate Results Indicators
Cumulative Target values
Indicator Name
Core
Unit of
Baseline
Measure
YR1 YR2 YR3 YR4
Responsibility
Data Source/
End Frequency Methodology for Data
Collection
Target
Monitoring plan for
implementation of St.
Petersburg declaration
No
designed and regularly
updated in participating
countries
Number
0.00
7.00
annual
Program
reports
PCTs
Regional studies under
the framework of the
St. Petersburg
declaration undertaken
and disseminated
Number
0.00
TBD
annual
Program
reports
PCTs, PMT
No
17. Intermediate Results Indicators
Results Framework
Intermediate Results Indicators
Cumulative Target values
Indicator Name
Unit of
Core
Baseline
Measure
YR1 YR2 YR3 YR4
Responsibility
Data Source/
End Frequency Methodology for Data
Collection
Target
EU Member States‟
forest sector knowledge
is made available to
participating countries
and knowledge exchange No
between participating
countries and EU
Member States is
ongoing
Number
0.00
TBD
(7)
annual
Program
reports
PCTs, PMT
Sustainable forest
management and
improved forest
governance best practice
No
models developed, tested
and used for
demonstration purposes
by the Program
Number
0.00
TBD
annual
Program
reports
PCTs
18. Intermediate Results Indicators
Results Framework
Intermediate Results Indicators
Indicator Name
Cumulative Target values
Responsibility
Unit of
Data Source/
Core
Baseline
End Frequency Methodology for Data
Measure
YR1 YR2 YR3 YR4
Collection
Target
Modern technology trialed
and systems to improve
No
forest governance
developed by the Program
Awareness, ownership
and capacity of key
stakeholders enhanced
Media coverage of FLEG
issues has increased
TBD
annual
Program
reports
PCTs
x%
Percentag
Based
e of
No
on
stakeholde
survey
rs
results
TBD
Twice,
baseline
and final
surveys
Repeated
perception
surveys of key
decision
makers
PMT
No
7.00
annual
Repeated
media
monitoring
PCTs
Number
Number
0.00
0.00
19. Defining End Targets
• Action needed by PCTs, PMT to determine
end targets for remaining indicators based on
– proposed country and regional activities
– planned baseline survey
• Discussion at SC meeting
END TARGETS?
20. Reporting under FLEG II Program
• Three streams of reporting:
Program-level
Reporting
• Country-level (PCT to PMT)
• Organizational-level (WWF & ICUN to WB)
• Program-level (WB to EC)
• Internal Reporting by the World Bank
21. Reporting under FLEG II Program
Country-level
Organizational
Organizationallevel
level
Program-level
PCTs
IUCN &
WWF
WB
PMT
WB
EC
Internal WB Reporting
22. Country-level Reporting
• Narrative Progress Reports
Country-level
PCTs
PMT
– Quantitative (on the basis of the
Results Framework and indicators)
– “Results Stories”, quotes
• Frequency: semi-annual
– Reports for period July to December due
January 31
– Reports for period January to June due
July 31
– Final report needs to be prepared prior to
activity completion
23. Organizational-level Reporting
Organizational
level
• Organizational Reports
• Frequency: semi-annual
– Reports for period July to December
due January 31
– Reports for period January to June due
July 31
– Final narrative report due 6 month after end
date of activity implementation, but can be
prepared at the end of activity
implementation
In accordance with the
Grant Agreements
24. Program-level Reporting
• Narrative Progress Reports:
Program-level
• Summary and Context of Project
• Activities carried out during the reporting
period
• Difficulties encountered and measures
taken to overcome challenges
• Changes introduced in implementation,
including in the procurement plan
• Final Narrative Report
In accordance with the
Administration Agreement
• 6 months after end disbursement
date
Results monitoring under the first FLEG Program focused on seven results areas, including: 1 – increased awareness and commitment of key stakeholders on FLEG2 – effective national and regional FLEG action processes in place3 – increased national ownership and capacity4 – improved regional and sub-regional collaboration and knowledge sharing5 – effective engagement of key trading partners6 – continuation of the formal official Europe-North Asia FLEG process7 – sustainable forest management practices implementedResults areas were monitored in country and regional programs based on qualitative assessments by the country teams and percentage completion of activities, as well as their success and outcome. Narrative reports were prepared to report on progress to the PMT. However, there was no actual results framework with measurable indicators.
While there were hundreds of activities with real impact, and while the impact of the Program has been large, this is based on anecdotal evidence. Overall, progress towards higher level outcomes is difficult to judge quantitatively. Example from Azerbaijan: The Ministry of Ecology and Natural Resources, as the main partner, was intensively involved during the development of forest-related educational materials (e.g. teacher support pack “Young Foresters’ School”) and other Program activities including trainings/seminars and studies. FLEG issues are a complex we of different causes and effects: e.g. addressing the issue of illegal logging simply by increasing enforcement or penalties (without addressing the root causes and underlying culture), it is likely that the impact on governance will in fact be worse than doing nothing. Because the drivers for illegal logging (such as poverty, lack of legal or alternative supply) will still be there creating the demand and so, increasing enforcement may simply create opportunities for corruption and collusion. Because of this complexity, it is difficult to select appropriate indicators.
Lessons LearnedNevertheless many achievements, which are summarized in the Final Report of the Program.
What is different in FLEG II? We now have a detailed Results Framework with indicators to track progress at the different levels – regional, national, and sub-national. Some of the indicators are directly linked to the PDO.
What is different in FLEG II? We now have a detailed Results Framework with indicators to track progress at the different levels – regional, national, and sub-national. Some of the indicators are directly linked to the PDO.
There are two sets of indictors: one on the PDO level and then Intermediate results indicatorsFirst set of indicators, i.e. PDO-level indicators include: core indicators developed Bank-wide and custom indicators
There are two sets of indictors: one on the PDO level and then Intermediate results indicatorsFirst set of indicators, i.e. PDO-level indicators include: core indicators developed Bank-wide and custom indicators.Results framework at the bottom will be used for reporting. Baseline for CSI 1 is No, Target values are tracked each year, annually. End target is yes.
CSI #1 – linked with PDO one Some Bank projects aim at supporting reforms of forest policies, forest products trade policies as well as legal and institutional frameworks in client countries. In some federal states, the Bank support to reforms may be delivered at lower, sub-sovereign level12. This indicator measures whether a project has supported forest sector reforms. This includes support to revised policies or legal and institutional reforms that have been adopted by the client. It also includes well-defined, time-bound phased action plans that have been launched with the objective of achieving such forest sector reforms. The processes have to be formalized through documented official endorsement. Adoption of reforms can be indicated by approving new legislation or by issuing implementing regulations or decrees. They also have to inclusive and consultative.13
CSI #2:This indicator covers capacity-building projects aiming at strengthening forest administration institutions and other institutions to deliver services to the forest sector. The targeted institutions may also be outside the forest sector (sensu stricto) and they may cover other public institutions or service delivery and law enforcement organizations in the rural landscape. This could include support to the implementation of trade policies during Voluntary Partnership Agreement (VPA) implementation. The baseline value for this indicator is expected to be zero. Guidance on “Government institutions provided with capacity building support to improve management of forest resources”: This refers to the number of national or sub-national institutions (e.g. forest or environmental departments at national, state or province-level) that have received capacity building as a result of the project. This includes support to training of officials, support to operations, information management or investments in physical infrastructure or other facilities. If sub-sovereign units14 are counted separately, it is essential that they have “independence” of each other. As a comparison, if forest policy implementation is piloted in two districts, but they are effectively field organizations for one agency or department, this counts as one institution supported.
CSI #3: Important indicator because one of the ways to improve forest governance and forest management is through capacity building and this indicator also will be reported at the corporate WB-level. Results from FLEG II will feed into/contribute to Bank-wide monitoring of impacts of projects. Often World Bank projects do not directly invest in the management of specific land or forest areas. Instead, these capacity-building projects aim at strengthening communities to improve forest management and their livelihood. The expected baseline value for this indicator is zero. Data must be disaggregated by ethnicity10 and gender. Guidance on “Forest users trained (number)”: This refers to the number of forest users and community members that have received capacity building through training as a result of the project. The concept may need to be adjusted according to local practice or national legislation. Training needs to be targeted to specific audience. General media or public awareness campaigns are not included. When estimating the number of people trained, it is essential to avoid double counting if same individuals have participated in a series of training events.
Custom indicators (4 in total) were developed to correspond to the three specific PDOs.
Custom indicators (4 in total) were developed to correspond to the three specific PDOs.
There are two sets of indictors: one on the PDO level and then Intermediate results indicatorsFirst set of indicators, i.e. PDO-level indicators include: core indicators developed Bank-wide and custom indicators.Results framework at the bottom will be used for reporting. Baseline for CSI 1 is No, Target values are tracked each year, annually. End target is yes.
Second set of indicators are intermediate results indicators: these include only customized indicators specific to the FLEG II Program (7 in total).
Second set of indicators are intermediate results indicators: these include only customized indicators specific to the FLEG II Program (7 in total).
Second set of indicators are intermediate results indicators: these include only customized indicators specific to the FLEG II Program (7 in total).Each country as part of their public awareness strategy will need to monitor media coverage of forestry and FLEG issues. A simple count of the number of media articles etc. can demonstrate how FLEG and Forestry coverage is rising.
Missing indicators: 2 PDO level and 5 intermediate results indicators
Content of Narrative Progress Reports same as that required for reporting to EC:Summary and Context of ProjectActivities carried out during the reporting periodBoth quantitative and narrative End date for activity implementation: IUCN and WWF: December 31, 2016 – final report due June 30, 2017 (PCTs)WB: February 28, 2017 – final report due August 31, 2017 (WB PCT) End disbursement date: June 30, 2017.
Content of Narrative Progress Reports same as that required for reporting to EC:Summary and Context of ProjectActivities carried out during the reporting periodBoth quantitative and narrative End date for activity implementation: IUCN and WWF: December 31, 2016 – final report due June 30, 2017 (PCTs)WB: February 28, 2017 – final report due August 31, 2017 (WB PCT) End disbursement date: June 30, 2017.
Narrative reports to accompany each funding request.Reports are to provide for comparison of the objectives of the Trust Fund, the results expected and obtained and the budget details for the ProjectFinal Narrative Report (due December 31, 2017)