2. 2
Context Around Evaluation
• Historically we have received a low number of
Excellence applications
• Evaluation was seen as a barrier to applying to
Excellence Strand
• Under the Innovation and Strategic
Development Strands some organisations
scored low on evaluation questions
3. 3
Evaluation
• We would expect you to conduct an evaluation
of your IESD project using the funding you
have allocated from within your approved grant
• If you plan your evaluation at the start of the
project you can ensure it is built into the
planning and delivery of your project
4. 4
Benefits of Evaluation
• Evaluation should be seen as a tool to help
improve your project.
• It can help you to understand why and how
your project is working.
• It enables you to involve stakeholders in giving
and collecting feedback.
• It provides evidence to future commissioners
about the success of your project
5. 5
Monitoring vs. Evaluation
• Evaluation involves an assessment or
judgement of your project based on the
information you collect about it.
• It focuses on ‘how’ processes or projects work
and ‘why’ they work and helps you to
understand what worked or did not work and
why.
6. 6
Monitoring vs. Evaluation 2
• Whilst Monitoring helps you to see whether
your aims and objectives are being met,
evaluation helps you to understand why and
how.
7. 7
Who undertakes the evaluation?
• Internal evaluation – which can be carried out
by internal staff or volunteers.
• External evaluation - by a university or another
vcs etc.
• The advantages of using an external evaluator
are that you will have an independent view and
that it will not take up staff resource
• Need to subcontract through a competitive
tendering process
8. 8
Evaluation specification
Needs to include the following:
• Overview of your requirement
• Aim of the evaluation and questions to be answered
• Background and context to the project
• Required methodology or approach
• Outputs and deliverables
• Timetable
• Requirements for dissemination
• Available budget (optional)
9. 9
What to evaluate
Every evaluation will be looking at something
different, so there is no set answer. Examples
include:
• The delivery model
• Service user outcomes and impacts
• Other/wider impacts
• Stakeholder involvement
• Partnership working
• Sustainability
• Cost effectiveness or value for money
• Good or effective practice
10. 10
How to Measure Impact
• Local health data from DH website
• Physical health indicators
• Lifestyle indicators
• Measures of how many falls or hip fractures
• Perceptions of health
• Perceptions of life satisfaction
When pulling together your bid for funding ensure you build in the costs of evaluation. It is good practice to set aside 3-5% of project value. Planning evaluation helps to ensure the right data is collected, for example keeping records of how many attendees at events, what their views are and how their views may change over time. Leaving evaluation until the end of the project can mean that useful information is lost or opportunities to collect important data are missed.
Please don’t see it as a necessary evil.
Tendering for this work will ensure complete independence, transparency and value for money. It could be a restricted tendering process which will involve inviting a small number of suitable organizations to respond to a specification.
Ideally you should issue the specification several weeks in advance of when the evaluator needs to be appointed. Hopefully they should then be in place for when your project goes live. Overview – give them what your project is trying to achieve with a background to any issues Key questions to be answered How effective was it in meeting its aims and objectives What outputs and outcomes have been achieved and how did they compare to targets Were there any unintended effects Methodology – you should list the data which will be collected during the lifetime of the evaluation but usually your evaluators will specify their own methodology. As a minimum it should include both quantitative and qualitative information. Reporting requirements – how often do you want interim reports Resources and timing – the evaluation should cover the lifetime of the project, you could specify a staged approach for example Phase 1 baseline and framework development Phase 2 interim evaluation Phase 3 final evaluation
Delivery Model - how did your project model work, what processes did you use? How was it delivered and managed? What worked well or not so well Service User outcomes and impacts – What was the effect of the project on service users? Were there any unintended outcomes, positive or negative? How much of the change can be attributed to your project? Other/Wider impacts – such as on your organisation as a whole or on the local community etc Stakeholder involvement – how effectively were stakeholders involved (service users, statutory or non statutory partners). Partnership Working – who were the partners? What were their roles, what worked/didn’t. Sustainability – What aspects of the project will be sustained and how VFM – was it cost effective and efficient, What added value di you deliver? How much money did you spend complare with the outputs/outcomes your project achieved? Good Practice – Have you learnt any promising or effective practice that will improve your service and can be shared with others.
Using indicators for health/care project might be local health data from DH website such as early death rates. These can be compared to national averages and changes in them measured over time. Physical health indicators including weight/BMI/body/waist measurements, blood pressure. Lifestyle indicators such as adult or childhood obesity, indicators for smoking or drugs misuse. Measures of how many falls or hip fractures Perceptions of health Perceptions of life satisfaction