The World Bank’s Involvement in GRPPs
•Als PPTX, PDF herunterladen•
0 gefällt mir•493 views
Presentation at the 2011 Evaluation Week
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie The World Bank’s Involvement in GRPPs
Ähnlich wie The World Bank’s Involvement in GRPPs (20)
Mehr von Independent Evaluation Group /World Bank Group
Mehr von Independent Evaluation Group /World Bank Group (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
The World Bank’s Involvement in GRPPs
- 1. The World Bank’s Involvement in GRPPs: The Current State of GRPP Evaluation Practice Chris Gerrard Evaluation Week October 24, 2011
- 2. What are GRPPs? ►Programmatic partnerships in which: • The partners dedicate resources (financial, technical, staff, reputational, etc.) toward achieving agreed objectives over time. • The activities of the program are global, regional, or multi- country (not single country) in scope. • The partners establish a new organization with shared governance and a management unit to deliver these activities. ►40 percent are located in the Bank, 35 percent in other international/partner organizations, and 25 percent are free-standing independent legal entities.
- 3. The Bank Plays Multiple Roles . . .
- 4. . . . But Is Not a Major Source of Funding
- 5. Relationship Between GRPPs, Trust Funds, and the DGF Global and Regional Partnership Programs (shared governance) Development Grant Facility Bank-Administered Trust Funds
- 6. What Are The Programs Doing? Location of the Secretariat In the In Another Independent Principal Activities World Bank Partner Legal Entity Total Knowledge Networks 11 31 11 53 Technical Assistance For NPGs 20 4 6 30 For GPGs/RPGs 1 1 2 Financing Country-Level Investments For NPGs 3 3 For GPGs/RPGs 13 3 6 22 Financing Global 1 2 4 7 Investments for GPGs Total 48 41 28 117
- 7. How Much Money Are They Spending? (Millions of U.S. Dollars) Location of Secretariat In the In Another Independent Principal Activities World Bank Partner Legal Entity Total Knowledge Networks 52 68 52 171 Technical Assistance For NPGs 184 3 94 282 For GPGs/RPGs 14 241 255 Financing Country- Level Investments For NPGs 291 218 For GPGs/RPGs 970 124 4,045 5,139 Financing Global 580 66 180 826 Investments for GPGs Total 2,077 274 4,612 6,963
- 8. GRPPs Are Concentrated in a Few Sectors . . .
- 9. . . . And a Few Vice Presidencies
- 10. GRPPs Challenge the Bank’s Country- Based Model ►They challenge the Bank’s traditional financial accoun- tability mechanisms. Their legal and governance arrange- ments do not always confer clarity on how collective responsibility for results is supposed to work in practice. ►Bank is dedicating more and more senior management time to the governance of these programs. ►Bank expects programs’ activities to be linked to the Bank’s country operations. But the way in which GRPPs plan their activities is different from the way in which country teams plan theirs.
- 11. GRPPs Present Challenges for Evaluation ►Programs have shared governance and accountability for results – to achieve something collectively that individual partners could not achieve by acting alone. ►Programs evolve over time – do not usually have a fixed end-point. ►Programs operate at multiple levels – global, regional, national, and local. ►Programs are diverse in size, age, sectoral focus, objectives, and activities (knowledge, technical assistance, investments).
- 12. There Is a Clear Trend Toward Stakeholder Models of Governance Inter- Low- and Civil Commer- national/ Private No. of Donor middle- society cial regional foun- programs countries income organi- private organi- dations countries zations sector zations Share- 24 holder 24 17 5 2 1 models (WB 24) Stake- 72 holder 90 62 22 55 49 27 models (WB 60) 96 Total 114 79 27 55 51 29 (WB 84)
- 13. Most Programs Have Been Evaluated, But Transparency & Accessibility Have Been Weak ►Of the 93 programs that are five years older or more, 77 have had evaluations. ►However, only 30 evaluation reports have been posted on the programs’ external Web sites and only 12 programs have posted a “management response” to the evaluation. ►By way of comparison, 57 programs have posted their charters on their external Web sites.
- 14. Independence, Quality and Impacts – Based on 20 GRPPs Reviewed since 2006 Location of the Secretariat In the In Another Partner Independent Legal Principal Activities World Bank Organization Entity Knowledge, CGAP ADEA (AfDB) GDN (New Delhi) advocacy & IAASTD GISP (CABI-Nairobi) GFHR (Geneva) standard-setting MAPS ILC (IFAD) GWP (Stockholm) networks PARIS21 (OECD) ProVention (IFRC) Providing country- Cities Alliance level technical MDTF-EITI assistance PRHCBP TFSCB Financing CEPF (CI) Development Gateway investments Stop TB (WHO) (Washington, DC) (global or Global Fund (Geneva) country-level) MMV (Geneva)
- 15. Evaluation Independence ►14 of 20 evaluations were effectively independent throughout the evaluation process: • Organizationally and behaviorally independent of program management • Protected from outside interference • Avoided conflicts of interests ►Independence of 2 evaluations was compromised during one stage of the evaluation process: commissioning, team selection, oversight, or review ►Independence of 4 evaluations was compromised at more than one stage
- 16. Evaluation Quality ►The quality of the evaluations was satisfactory in 8 of 20 cases in terms of evaluation scope, design, methods, and feedback ►The evaluations had moderate shortcomings in 9 cases, and significant shortcomings in 3 cases ►Most common issues adversely affecting quality were: • An unclear terms of reference • Insufficient budget and time • Weak M&E frameworks for the program • Lax evaluation methodology and tools
- 17. Evaluation Impacts on the Programs ►Evaluations have had notable impacts on the programs. ►Most common impacts have been: • Half the programs modified their governance arrangements in some way, such as establishing an executive committee. • Almost half the programs revised their strategies. • About one-quarter improved their M&E systems.
Hinweis der Redaktion
- IEG’s recent report had four sets of findings:Independence and quality of GRPP evaluationsThe development effectiveness of the programs reviewedThe progress that Bank management has made in implementing Bank-wide systems and accountabilities for managing and overseeing its portfolio of GRPPsThe effectiveness of the Development Grant Facility in providing grant financing for GRPPs.This being “Evaluation Week,” this particular presentation focuses on the first set of findings. See the full report for the other findings and recommendations.
- Historically, bilateral donors and other international organizations have been the principal partners. Increasingly, recipient countries and civil society organizations are also involved in program governance.
- The Millennium Declaration appears to have had a significant impact on the establishment of large GRPPs that are financing country-level investments such as GAVI, the Global Fund, Education for All—Fast Track Initiative, the Climate Investment Funds, and the Global Agriculture and Food Security Program. These programs are pooling donor resources earmarked for a specific purpose.These programs create the potential for competition or collaboration with the Bank’s own lending portfolio.
- The essence of a GRPP is shared governance, with dedicated resources, and active in more than one country or region. This also means shared accountability for results, wherever the program management unit (the secretariat) is located – in the Bank, in another partner organization, or as a free-standing independent legal entity.
- The Bank is a voting member of 79 out of 117 programs, and a non-voting member or official observer in another 13 programs – as of fiscal year 2011. The Bank chairs or co-chairs 24 programs.The Development Grant Facility has provided some financial support (past or present) to 71 programs (61 percent).Bank-administered donor trust funds are currently providing support for 71 programs – including 11 GRPPs being supported by FIFs. (Now 12 FIF-supported programs, with the addition of the Nagoya Protocol Implementation Fund.)48 programs are located in the Bank, including 4 programs in the GEF Secretariat. (Now 5, including Nagoya Protocol.)
- The Bank provides only 2.5 percent of the financing from its own resources (mostly from the Development Grant Facility), but administers almost 80 percent of the donor trust funds supporting the programs.
- This shows the relationship between GRPPs, Trust Funds and the DGF. Some DGF grants and many trust funds are not supporting GRPPs. Some GRPPs in which the Bank is involved are not being supported by either DGF grants or trust funds.DGF grants and trust funds are largely independent sources of finance for GRPPs, since DGF grants generally flow to partnerships outside the Bank and TFs generally support partnerships and other activities inside the Bank. Only 9 out of 117 GRPPs were supported by both DGF and TFs in FY2010. Including the four GEF programs, 46 out of 71 trust-funded GRPPs were located inside the Bank.Trust fund disbursements are actually 50 times the size of the DGF disbursements. About 60 percent of TF disbursements in FY10 were from GRPPs (including FIFs). Of the remainder, 80 percent were from country-specific trust funds, and 20 percent from Bank-managed multi-recipient country trust funds.
- The largest number (45 percent) of programs are knowledge, advocacy, and standard-setting networks, most of which are supported by the DGF and are located outside the Bank. Most spend less than $5 million annually.Programs that are supporting the provision of national public goods in sectors such as agriculture and rural development, urban development, transportation, water, energy and mining, education, and social protection tend to limit their support to technical assistance in order not to compete with the Bank’s country lending operations. These programstend to be located in the Bank due to the Bank’s country presence to supervise such technical assistance. These are somewhat larger programs, averaging $16 million a year.The overwhelming majority of investment programs are supporting the provision of global and regional public goods. EFA-FTI and GAFSP are two notable exceptions, justified on the basis of the MDGs.
- The four largest programs – the Global Fund, the GEF, the CGIAR, and GAVI – account for 73 percent of the entire $7.0 billion being spent by all 117 programs.
- Four sectors account for 54 percent of the programs – environment, health, agriculture and rural development, and public sector governance. Sector Boards are generally responsible for approving new programs.
- Two Network Vice-Presidencies account for 54 percent of the programs – the Sustainable Development Network and the Human Development Network. The Africa Region accounts for 11 percent of the programs. Vice-Presidencies are generally responsible for oversight and management of the programs going forward.
- Unlike the Bank’s bilateral financial relationship with client governments, GRPPs are collective entities in which the partners have collective responsibility for the financial resources that they have dedicated to the global and regional activities of the programs.Programs generally seek the highest possible level of Bank representation of their governing bodies. The Bank’s representation on partnership Boards is at the Director level or higher for half the 92 programs for which the Bank is member or official observer.
- These challenges arise from the nature of GRPPs – as partnerships, and as programs that are global or regional in scope – and from their diversity. An increasing number of programs are adopting 4-5 year strategies with specific objectives for which the program is accountable for achieving during the strategy period. This is generally a good practice which provides a foundation for periodic evaluations.
- This refers to representation on the lowest, executive tear of governing bodies. The Bank is on 84 executive bodies and 92 overall.Shareholder models limit representation to contributing partners. Stakeholder models include non-contributing stakeholders. Governing bodies have become more inclusive in order to enhance legitimacy. But IEG has not been able to conclude that stakeholder models are more effective than shareholder models. Direct representation does not necessarily translate into effective voice. Non-contributing stakeholders may be able to express themselves more effectively in other ways. The real question is whether the governing bodies and management units are legitimately and effectively performing their designated functions.The lesson for the Bank: We had better used to having developing countries and CSOs on the governing bodies of GRPPs.
- Periodic evaluations have become a principal instrument of accountability.Some programs post their evaluation reports only on their internal Web sites – accessible only to members – not to the general public.That evaluation reports will be publicly disclosed should be stated clearly in the evaluation terms of reference. Both commissioners and evaluators should insist upon this.
- The programs that IEG has so far reviewed are representative of the Bank’s GRPP portfolio.Each Global Program Review does three substantive things:Assesses the independence and quality of the evaluation on which the Review is based.Provides a second opinion on the effectiveness of the program.Assesses the performance of the Bank as a partner in the program.Then it draws lessons based on these findings for the Bank’s involvement in GRPPs more generally.All but two programs are still operational – the International Assessment of Agricultural Knowledge, Science and Technology for Development and ProVention. Of those that are still operational, the Bank remains involved inn all but the Global Invasive Species Program and the Development Gateway Foundation. The Bank remains engaged in the International Land Coalition and the Global Water Partnership, even though it is no longer providing financial support to these programs.
- Monitoring is a management responsibility, and evaluation is a governance responsibility. There is a clear trend toward more independent evaluations. To preserve the organizational independence of the evaluation, IEG recommends that the programs’ governing bodies should commission the evaluation, approve the evaluation terms of reference, select the evaluation team (ideally using competitive methods), and receive the final evaluation report — or establish an oversight subcommittee for these purposes. If the governing body does not have the institutional capacity to undertake all these things independently of the management (secretariat) of the program, then the secretariat should act purely as an agent of the board in these respects. Even if organizational independence is somewhat compromised, the evaluation may still be independent if behavioral independence is sound, that is, if the evaluators behave independently of management, notwithstanding the challenges of doing so. IEG has observed that the pool of candidates with the required technical skills, knowledge, and experience to evaluate large GRPPs like the Global Fund and the Global Environment Facility, who have never done any work for the program, is often limited due to the program’s overwhelming presence in the sector. In such cases, the key is to identify and manage conflicts of interest transparently.
- Evaluation scope refers to relevance, efficacy, efficiency, governance and management, and sustainability. Appropriate evaluation methodology depends on the types of activities being undertaken. Field visits are needed if the program is financing country-level technical assistance or investments.An incomplete, broad or vague TOR that did not set a clear framework for the evaluation was the most common problem at initiation.IEG suggests, as an initial benchmark, that a program-level evaluation should be budgeted at 1–3 percent of annual program expenditures — closer to 3 percent for smaller programs and closer to 1 percent for larger programs. The size of the budget should also be influenced by the scope and methodology that have been specified for the evaluation (those requiring field work cost more) and by the quality of the program’s M&E system (those with weaker M&E systems cost more to evaluate). Weak M&E frameworks made it difficult to assess the overall effectiveness of the programs: Little systematic evidence was available on achievements at the outcome level, or the extent to which outcomes could be attributed to the program’s activities. With two exceptions (GDN and GFHR), little effort was made to assess the efficiency or cost-effectiveness of the program’s individual outputs. All the external evaluations had to spend some resources — if resources were available — attempting to reconstruct the historical inputs, outputs, and outcomes of the program.
- Led by the global health partnerships, individual GRPPs are slowly developing a culture of evaluation. There is a growing awareness of the importance of independence in evaluation and a clear trend toward more independent evaluations. Commissioners of evaluations are identifying the important issues to be addressed (including governance and management issues), and evaluators are developing instruments to address them. But many programs continue to regard periodic evaluations as a substitute for the hard work of putting in place adequate monitoring systems to track outputs and outcomes in relation to plans, to identify reasons for success or failure, and to take the necessary actions to improve performance. Few programs have established adequate M&E systems prior to or even immediately after their first evaluation. And there is still the sense that some program management units are going along with the pressure from donors to undertake regular evaluations more to mobilize funds for the programs than to learn lessons to improve the program’s effectiveness.