JPI Conference Dublin - Edvard Beem - Evaluation and Monitoring Framework

1. EU Joint Programme - Neurodegenerative
Diseases Research (JPND)
Developing a framework for monitoring and evaluation
Joint Programming Conference 2013
Dublin, 28 February & 1 March 2013, Parallel Session 8
Edvard Beem, Co-director ZonMw, member of the Executive Board of JPND

2. This presentation
• Rationales and objectives of JPND
• Development of a monitoring and evaluation framework
• Logical Framework Analysis as a systemic tool
• Overview of the defined set of indicators of performance
• Execution of monitoring to date
• Key issues, challenges and lessons learned

3. Alzheimer’s Disease in Europe
40
Number of Societal costs (€)
cases
30 Dementia 2010 6,000,000 72,000,000,000
Cases / 100
20 2040 12,000,000 144,000,000,000
Stroke
10
Parkinson's
0
A major societal challenge
60 70 80 90 100
Age (years)
for the coming years
From Rocca W et al, Eurodem

4. A major societal
« Grand Challenge »
Energy
Climate Change
Food & Health
Neurodegeneration

5. Scope of JPND
Scientific
Scientific
• Animal models Medical
Medical
Animal models
• Biobanks
• Biobanks • Early diagnosis
Early diagnosis
Focus on • Prevention
• Cohorts/registries
• Cohorts/registries • Prevention
Three Domains • Disease pathology
• Disease pathology • Clinical trials
• Clinical trials
Social
Social
• Health care delivery
Health care delivery
• Home automation
• Home automation
• Health economics
• Health economics
• Ethics
• Ethics
Diseases Targeted
Alzheimer’s Disease and other Parkinson’s Disease & related disorders
dementias Huntington’s Disease
Motor Neurone Disease Spinocerebellar Ataxia (SCA)
Prion Disease Spinal Muscular atrophy (SMA)

6. What are the JPND goals?
• Increase coordinated investment in ND research aimed
at finding causes of disease, developing cures, and
identifying appropriate ways to care for those with ND
• Our immediate goals:
• Launch activities that add value to national research efforts
• Leverage additional resources through partnerships
• Link and support national plans with relevance for ND

7. Why JPI?
• Awareness
• common challenge, common vision
• Mapping
• Inventory running national programmes (database)
• Overlaps and gaps, opportunities for joint actions
• Baseline for monitoring
• SRA (>10 years)
• Roadmap for EU-wide investment
• Implementation plan phase 1 2012-14
• Joint actions: calls, enabling activities, action groups

8. CALLS
Total commitment 2011-2014
• Pilot Joint Transnational Call 2011 = €15M
• CoEN program call 2011 (€ 4M) and calls 2012-2014 (€12M)
• Joint Transnational Call 2012
• “identification of genetic, epigenetic and environmental risk and protective
factors” = €19M
• “evaluation of health care policy, strategies and interventions = € 10M
• Estimate commitment 2013 = €25M
• Estimate commitment 2014 = €25M
• Total projected investment: over €100M in cash contribution
(no in-kind resources included)

9. Enabling activities
JPND will execute the SRA within the next ten years through a combination
of:
• Knowing our research • Capacity
capability building
• Opportunities for • Education &
infrastructure and platforms
training
• Working in partnership
with industry • Connection to
• Working with the regulators policy makers
• Global partnership (third • Communication
country collaboration) and outreach

10. Action groups
• Action Groups to determine transnational research needs and
opportunities in specific scientific areas
• Longitudinal Cohort Studies (disease- & population-based)
• Animal and Cell Models
• Assisted Living Technologies
• Action Groups to promote engagement, commitment and
partnerships
• Engagement and Partnership with Industry
• Engagement and Partnership with the EC and beyond
• User and Public Involvement in ND Research
• Alignment of National Plans and activities

11. Why monitoring and evaluation?
Critical questions
• Does a co-ordinated EU-wide approach makes a difference
achieving the outcomes?
• Will the outcomes of a concerted approach be more than the
sum of the outcomes of national approaches?
• Can we reduce fragmentation?
• Can we recognize and learn from successes and failures?

12. Monitoring and evaluation framework
• Analysis of the links between (societal) challenges,
objectives and activities of a programme or initiative
related to the expected effects and impacts
• By mapping the intervention logic it will be easier to
assess whether the objectives have been met and what
the success criteria are
• However it is “not as easy as it looks”
• Different objectives and effects
• Changing objectives
• Unexpected effects

13. Logical Framework Analysis (LFA)
The logic:
• A public action is undertaken for a reason (rationale)
• It has objectives which address (societal) needs
• It provides inputs which lead to activities
• These activities create (direct) outputs
• Which on their lead to outcomes/results and finally to
(long-term) impacts

14. Intervention logic
Mission
Long-term
Impacts
Specific
Result Outcomes or goals
chain results
Programme
logic
Operational
Outputs goals
Interventio
n
logic
Inputs Activities
Rationale or
reasons
(Societal)
challenges

15. Logical Framework of JPND

16. Indicators of performance
• Based on the Logical Framework Analyse, indicators of
performance have been developed:
• Type A indicators monitor the effect of JPND on (European)
research programming, research policy and funding (the concept of
Joint Programming)
• Type B indicators monitor the scientific and societal impact of
research on neurodegenerative diseases conducted in the context
of JPND
• Distinction is made between:
• Input indicators
• Output indicators
• Outcome indicators
• Impact indicators

17. Development of indicators

18. Examples of type A indicators
• Input indicators
• Attitude towards JPND goals and objectives
• The entry or drop out of countries
• Output indicators
• The national research (funding) priorities based on JPND’s Research
Strategy
• Outcome indicators
• The amount of JPND common research funding for neurodegenerative
diseases as share of total EU research funding
• Impact indicators
• The number of publications in high impact journals of European
researchers in neurodegenerative diseases research

19. Examples of type B indicators
• Input indicators
• The number of collaborative research projects funded through JPND
• The number of non-project funded activities
• Output indicators
• The number of Europe-wide population-based studies with contribution
of JPND
• Outcome indicators
• Regular interactions between JPND and stakeholder groups
• Impact indicators
• Exchange of practices across different types of research (basic,
clinical and healthcare)

20. Execution of monitoring to date
1. Monitoring reports based on indicators
• Quantitative indicators based on desk study, information made
available by other work packages
• Qualitative indicators obtained through a questionnaire
investigating the attitudes towards JPND:
• Expectations regarding JPND
• Opinions on current JPND activities
• Views on the future of JPND
1. External review of experts
• Review focusing on JPND’s governance and and processes of
strategic policy programming (strategic direction) that have led
to the development of the Research Strategy (SRA) and the
implementation plan

21. Key issues, challenges and lessons
learned
• Special attention should be paid to monitoring and evaluation as an
important element to show progress and results of the initiative to
various audiences
• Development of monitoring and evaluation framework should be
aligned with the development of the Research Strategy
• Monitoring and evaluation is NOT just listing a set of indicators and
start measuring, but takes some time and effort to develop according
to intended objectives and achievements of the initiative
• The systematic use of a Logical Framework Analysis approach is
considered to be very helpful in (re)defining
the objectives of JPND

22. More information
• The full version of the monitoring and evaluation framework
to be found on the website of JPND:
http://www.jpnd.eu
The team responsible for monitoring and
evaluation of JPND:
• Edvard Beem beem@zonmw.nl
• Bastian Mostert bastian.mostert@technopolis-group.com