Committee and Council Members induction By Luke Moody, Deputy Head of Evaluation, Corporate Strategy and Analysis

1. ESRC Evaluation Strategy
Mr Luke Moody, Deputy Head of Evaluation, Corporate Strategy and
Analysis

2. Content
▶ ESRC Evaluation Committee
▶ ESRC Evaluation Strategy
▶ Project Evaluation
▶ Large Investment and Policy Evaluation
▶ International Benchmarking
▶ Impact Evaluation

3. ESRC Evaluation Committee
▶ Oversees evaluation of quality and impact of all
ESRC investments
▶ Responsible for advising Council on the
successful achievement of its corporate strategy
▶ Operates independently of the Policy
Committees, reports directly to Council

4. Evaluation Strategy – cuts across all
Committee and Network remits
▶ Impact through World Class Research: Evaluations of the academic quality
and impact of ESRC research and UK Social Sciences more generally – to
advise Council on the impact and quality of its research portfolio, and to
advise the Research Committee on further investments
▶ Impact through Skilled People: Evaluations of ESRC funding schemes and
training investments – to advise Council and the Training and Skills
Committee on the impact and further development of research training
initiatives
▶ Impact through Infrastructure: Evaluations of ESRC funding schemes and
resource investments – to advise Council and the Methods and
Infrastructure Committee on the impact and further development of research
methods and infrastructure initiatives
▶ Impact through International Leadership: Evaluation of internationally
focused ESRC funding schemes and investments – to advise Council and
the Policy Committees on the impact and further development of initiatives
that promote international collaboration
▶ Impact through Partnerships: Evaluations of the impact of ESRC
partnerships and investments on policy and practice – to advise Council and
the Policy Committees on the practical impact of its partnership building, and
on ways to increase this contribution

5. Why evaluate?
▶ The main purposes of evaluation are to:
▶ provide an assessment of accountability, i.e. whether
public funds were spent as agreed
▶ assess whether a project has been conducted
effectively, whether it has met its objectives and to
make an assessment of the quality and impact of the
research
▶ provide award holders with some feedback about the
management, quality and rigour of the research, and
to provide comments on uses or potential uses of
the research
▶ Learn lessons to inform ongoing and future activities

6. Project Evaluation
▶ All ESRC grant holders must provide:
▶ End of Award Report (3 months after grant end).
▶ Impact Report (12 months after grant end).
▶ Subject to external peer review and grade.
▶ Cumulative results reported annually to Council.

7. Investment and Policy Evaluation
▶ Annual programme of large investment, scheme
and policy evaluations.
▶ Consultation with Policy Committees.
▶ Undertaken by external evaluators.
▶ Reports presented to Council and relevant
Committees, alongside synthesis of key findings
and recommendations
▶ Mechanism in development for tracking
recommendations

8. International Benchmarking
▶ Assess comparative performance of UK Social
Science disciplines.
▶ In partnership with Learned Societies.
▶ Review series aims to:
▶ - highlight the standing and contribution of UK
disciplines
▶ - identify ways of enhancing performance and
capacity
▶ - promote future research agendas

9. Impact Evaluation - Aims
▶ To identify and analyse evidence of research
impact on policy and practice.
▶ To understand how impact is generated, and
help the ESRC to improve its performance in this
area.
▶ To develop impact evaluation methods.

10. Impact Evaluation – Critical Issues
▶ What are the problems?
– Timing – how long to wait?
– Attribution – what role has research played in change?
– Additionality – what is ESRC’s contribution?
▶ Different types of Impact:
– Instrumental
– Conceptual
– Capacity Building
▶ Looking for demonstrable impact on policy and
practice

11. Impact Evaluation – Case Studies
▶ Experimental Policy and Practice Case Studies –
2005 onwards
▶ Aims:
– to trial methods - ‘what works’
– produce evidence – ‘demonstrating impact’
– improve understanding - ‘how does impact occur’
▶ Testing:
– wide range of methods
– wide range of contractors
▶ Key Requirements:
– conceptual framework
– understanding the ‘how’ as well as the ‘what’

12. Drivers of Impact
▶ Established relationships and networks with user
communities
▶ Involving users at all stages of the research
▶ Well-planned user-engagement and knowledge
exchange strategies
▶ Portfolios of research activity that build reputations
with research users
▶ Good infrastructure and management support
▶ Where appropriate, the involvement of
intermediaries and knowledge brokers as translators,
amplifiers, network providers

13. Economic Impact - Pilot
▶ Pilot Study: Some success in valuing outputs,
but quantifying the wider impact of research was
more challenging
– Two main barriers:
▪ Disentangling multiple contributors to policy development
▪ General lack of evidence on economic impact of government policies
– Access to robust evaluation data on the impact of
individual policies is needed in order to quantify
ESRC’s contribution to those policies

14. Economic Impact – Tracking Back
▶ Tracking Back Studies – working backwards
from policy or practice initiatives that have been
subject to evaluation
▶ Assessing ESRC research contributions to the
initiative
▶ Estimating economic value of ESRC’s impact,
using national evaluation data as a benchmark
▶ EMA and Pathways to Work

15. Developing Impact Evaluation
▶ Conceptual Impact
– Child Poverty Policy
▶ Impact of Infrastructure
– Economic impact infrastructure investment: the
Economic and Social Data Service (ESDS)
▶ Impact of Skilled People
– People-flow impacts in the Welsh Government

16. Further Information
▶ ‘Taking Stock’ and ‘Branching Out’:
http://www.esrc.ac.uk/impacts-and-
findings/impact-assessment/developing-impact-
evaluation.aspx
▶ Policy and Practice Case Studies:
http://www.esrc.ac.uk/impacts-and-
findings/impact-assessment/policy-practice-
impacts.aspx
▶ Economic Impact Evaluation:
http://www.esrc.ac.uk/impacts-and-

17. ESRC’s Contacts
▶ Speaker:
– Mr Luke Moody, Deputy Head of Evaluation, Corporate
Strategy and Analysis
luke.moody@esrc.ac.uk
▶ ESRC website
– www.esrc.ac.uk