Presentation by Philippe Ellul, Senior Science Officer, Consortium/System Office on the monitoring, evaluation, and impact assessment framework in CGIAR and the role of monitoring and reporting

1. Philippe ELLUL, Senior Science Officer
ECOP meeting
Rome, 12-13 Nov. 2015
Consortium/System Office Science Team
Monitoring, Evaluation & Learning

2. System Office and new SRF
Finance
IP & Legal
Strat. & Partn.
ICT & KM
Science
System Council
Portfolio / CRPs / Centers
independent...
IEA
ISPC
IAU
NewSRF

3. ScienceTeam at the CRP level
To implement the reform process in alignment with SRF by
– maintaining an ongoing direct dialogue with CRPs to provide a sounding
board and share best practices – concerns across programs
– ensuring that top quality and effective R4D proposals are designed and
proposed to CB/ FC for approval (Extension, draft & pre-proposal phase2)
– assessing annually the planning (POWB) and reviewing performance of
CRPs (Annual Reports) by monitoring progress along impact pathways
and advise on resource re-allocation
– advising on the potential use of technological and behavioural innovations
(eg pre-breeding approaches, policies issues on germplasm exchange, big
data opportunities, system dynamics or agro-biodiversity studies
– supporting CRPs to set up an efficient Result-Based Management (RBM)
framework, promoting an outcome-based approach; MELCoP

4. 1)The ST supports CGIAR
scientists & stakeholders
in shaping a coherent,
innovative and well-
balanced Portfolio of
CRPs that
• enhances synergies
across the system
• address research
gaps
• minimize
duplication
ScienceTeam’s role at the Portfolio level

5. 2)The ST produces and facilitates
adherence to CGIAR research
policies, guidelines, and
standards that facilitate increased
coherence and impact
• Harmonized reporting templates
(POWBs andAnnual Reports)
• Guidelines for Extension Proposals
2015-16
• Guidelines for CRP 2nd call 2017-22
• CB/FC Gender Strategy
• OpenAccess &Open Data strategy
(harmonized implementation)
ScienceTeam’s role at the Portfolio level

6. 3)The ST deals with increased Portfolio efficiencies by steering the
entire package towards increased efficiency, relevance, and
scientific innovation and quality by:
• identifying concerted actions (pre-breeding, GB++, RBM, etc...)
• harmonizing responses on cross-cutting strategic issues
• coordinating strategic sharing of research infrastructure across CRPs
 Site/country integration (Partnership)
 Data repositories, metadata, interoperability (OA-DM, ICT)
 User-friendly flexible IT system for RBM results information
 Innovation platforms, Phenotyping platforms
 Etc...
ScienceTeam’s role at the Portfolio level

7. ScienceTeam’s role at the Portfolio level

8. 1st MEL CoP meeting (Rome, 10-11 November 2015)

9. System Office & CRP performance monitoring
CCEE
Monitoring
Evaluation
CRP Life Cycle
Proportionofresearchportfolio
Impact Assessment
CONSORTIUM –
SYSTEM-OFFICE
IEA
IAU

10.  But generic ToC far too simplistic about how science can
influence rest of world
 Essential to evolve realistic & dynamic ToC overtime (e.g
private seed companies w/ 5-year plan revisited annually)
 Need + flexibility, + willingness to learn from mistakes,
asking key questions, re-inventing program
3rd round of CRP Annual Reports (2012-2013-2014)
•  quality reporting
• + structured CRPs
•  visibility on progress towards IP
• Performance reporting strategy

11. CRP Annual Reports 2013, 2014 and 2015
A. KEY MESSAGES (1 ½ page)
B. IMPACT PATHWAY AND INTERMEDIATE DEVELOPMENT OUTCOMES (IDOS) (1/4 page)
C. PROGRESS ALONG THE IMPACT PATHWAY
The CRP should complete Table 1, in Annex 1 and provide a narrative (C.1 to C.3),
referring to those indicators from the table in Annex 1 that are relevant to the CRP
C.1 Progress towards outputs ( 2 pages)
C.2 Progress towards the achievement of research outcomes and IDOs ( 2 pages)
C.3 Progress towards Impact (1/4 page)
D. GENDER RESEARCH ACHIEVEMENTS (1 page)
E. PARTNERSHIPS BUILDING ACHIEVEMENTS (1 page)
F. CAPACITY BUILDING (1/2 page)
G. RISK MANAGEMENT (less than 1/2 page)
H. LESSONS LEARNED (1 page)
I. CRP FINANCIAL REPORT (L series, 9 Excel files)

12. CRP Annual Reports 2013, 2014 and 2015
• KNOWLEDGE,TOOLS, DATA (12)
• CAPACITY ENHANCEMENTAND INNOVATION PLATFORMS (5)
• TECHNOLOGIES/PRACTICES INVARIOUS STAGES OF
DEVELOPMENT (10)
• POLICIES INVARIOUS STAGES OF DEVELOPMENT (5)
• OUTCOMES ONTHE GROUND (2)
34 indicators inTable 1 / Annex 1

13. Indicator #9 Number of publications in ISI journals produced by CRP
 Web-link to the complete list of papers published in
2014
 Funding acknowledgement (W1, W2, W3 or
bilateral) is suitable in all publications (mapping the
publications to CRPs’ contribution)
 A classification based on ISI impact index would
provide information on science quality and content
 Present a summary on the CRP research staff
(numbers, grades, gender; see FTA AR 2014)
2014 CRP ISI papers
1 FTA 328
2 GRiSP 218
3 WLE 150
4 A4NH 137
5 CCAFS 114
6 GL 108
7 WHEAT 107
8 PIM 98
9 RTB 92
10 DS 72
11 MAIZE 64
12 AAS 49
13 L&F 48
14 DC 47
15 HT 44
TOTAL 1676

14. CRP Performance / Science quality
Informed by
Elsevier bibliometric analysis
2014

15. 34 indicators of the Annual Report (Table 1 / Annex 1)
TECHNOLOGIES/PRACTICES INVARIOUS STAGES OF DEVELOPMENT
18. Number of technologies/NRM practices under research in the CRP (Phase I)
=> Novel genes, promoters or QTLs with known major effect(s) on specific traits,
molecular markers, breeding or lines with improved traits (NILs, RILs, transgenic, …),
DHLs, animal lines with specific traits
23. Number of technologies /NRM practices field tested (phase II)
=> Superior genotypes, breeds, lines, crosses or hybrids with improved traits,
transgenic lines, improved variety for which certification is being sought
27. Number of technologies /NRM practices released by public and private
sector partners globally (phase III)
=> Improved varieties, cultivar, lines or breeds with various desirable traits,
management practices, ready to be dessiminated

16. TECHNOLOGIES /
PRACTICES IN
VARIOUS STAGES
OF
DEVELOPMENT
18. Number of
technologies/NRM
practices under
research in the CRP
(Phase I)
23. Number of
technologies /NRM
practices field tested
(phase II)
27. Number of
technologies /NRM
practices released by
public and private
sector partners
globally (phase III)
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
WHEAT
MAIZE
DC
GL
RTB
Phase I nº18 Phase II nº23 Phase III nº27
WHEAT MAIZE DC GL RTB
Phase I nº18 200000 34123 6329 1684 78
Phase II nº23 2151 2063 2173 23 48
Phase III nº27 24 70 8 32 22

17. From guidelines to creativity... MAIZE Annual Report 2014

18. Only 2 outcome-based indicators (Table 1 / Annex 1)
OUTCOMES ONTHE GROUND
33. Number of hectares under improved technologies or
management practices as a result of CRP research
34. Number of farmers and others who have applied new
technologies or management practices as a result of CRP
research

19. Outcome indicators 33 (a) and 34 (b)
From CGIAR Research Programs (CRP) Annual Reports 2013 and 2014
CRPs Number of hectares (a) Number of farmers reached (b)
2013 2014 2013 2014
A4NH 231 135,000 645,075 1,089,139
AAS (c) 242,133 7,408 584,953 -
CCAFS 450,000 (d) 185,000 2,000,000 3,900,000
Dryland Cereals 80,000 267,800 70,000 377,000
Dryland Systems 4,000,000 7,525,065 4,000,000 469,348
FTA 196,500,000 65,000,000 870,000 2,000,000
Grain Legumes 253,224 4,763,000 1,717,249 2,451,917
GRiSP 4,763,000 91,996 7,741,000 7,741,000
HumidTropics 12,000 430,000 94,466 91,000
Livestock & Fish (c) 331,070 632,169 24,000 488,000
MAIZE 417,000 2,305,462 1,100,000 4,089,000
PIM - - - -
RTB 398,000 430,000 1,180,000 1,180,000
WHEAT 1,650,000 2,096,240 2,802,000 10,310,000
WLE 15,471,050 4,845,000 35,000 205,946
Total 224,567,708 88,714,140 22,863,743 34,392,350

20. Intermediate users’ adoption
End of
ADB
project
HRDC
Formed
38 members
HRDC w/ 68 public & private members
Kindly provided by Dr. Achim Dobermann (IRRI)

21. End users’ adoption
Drought Tolerant Maize for Africa (DTMA) project

22. DroughtTolerant Maize for Africa (DTMA)
http://www.cgiar.org/consortium-news/partnerships-lead-to-measurable-impacts-for-drought-tolerant-maize-for-africa/
DTMA Project indicators 2013
Number of new varieties released (2007-2013) 140
Seed production in 2013 (new varieties) 30,000 t
Number of NARs - or companies - involved in seed production 13
Number of stakeholders adopting the new varieties: cooperatives;
regional, national, small & medium enterprises (SMEs);
international enterprises; large companies
118
Hectares planted with the new varieties 1,230,000 has
Number of households reached 2,900,000
Number of consumers benefiting these adoptions (Millions) 20 M

23. CRP Portfolio Report 2014

24. CRP Portfolio Report 2014

25. CRP Portfolio Report 2014

26. CRP Portfolio Report 2014
8. RISKS TO PORTFOLIO PROGRESS AND THEIR MITIGATION
Risks of a CRP portfolio biased towards short-term priorities
and results
 New financial risk
 Risk of continued disconnection between financial and
programmatic planning and management
 Risk of mobilizing insufficient human resources to
undertake research on complex systemic approaches
 Risk of a CRP leadership not fully empowered
 Risk of a patchy approach to integrity and ethics in research
for development at portfolio level

27. ScienceTeam’s role at the CGIAR-Institutional level
How to optimize the implementation of the CGIAR reform process?
 To participate to the design of the new Strategic Research
Framework (SRF) for the CGIAR
 To explain the way in which the new System Council functions,
what it decides and the rationale in the decisions, to communicate
clearly to the CRPs/stakeholders what donors’ expectations are
 To communicate and work + collaboratively with ISPC, IAU, IEA,
regional organizations (GCARD3), donors, development agencies
 To maintain and build external partnerships (to provide
leadership in agricultural development, share / learn from others)
 To ensure a system-wide focus demonstrating global visibility on
development impact for smallholders’ benefit

28. Thanks a lot for your attention...