3. AGENDA
MORNING SESSION
Time Agenda Item Speaker
10.00am Tea and coffee available
10.30am Welcome and introduction Nick Ockenden
Head of Research, NCVO
10:35am ‘Why is measuring impact important
now?’
Karl Wilding,
Director of Public Policy and Volunteering,
NCVO
11:05am ‘Planning your impact’ Georgina Anstey,
Consultant, NCVO
11.35am Roundtable discussions and
feedback
Chair
12.15pm Lunch
4. AGENDA
AFTERNOON SESSION
Time Agenda item Speaker
1.15pm ‘What difference do your volunteers
make? Top tips on how to undertake a
volunteering impact assessment’
Joanna Stuart,
Senior Research Officer, NCVO
1.45pm ‘Challenges & tools’ Claire Knight, Head of Volunteering,
Breast Cancer Care
Rebecca Stewart, Head of Volunteering,
Age UK
Representing the National Network of
Volunteer Involving Agencies (NNVIA)
2.05pm Q & A Chair
2.20pm ‘Volunteering in care homes’ Matt Hill,
Senior Research Officer, NCVO
2.40pm ‘Research into impact on
health & wellbeing’
Daiga Kamerade,
Work and employment researcher,
University of Birmingham
3.00pm Q & A Chair
3.15pm Closing remarks Chair
12. WHAT IS IMPACT?
• Impact is overall change we seek to
achieve.
• What we do – our outputs – designed
to deliver impact.
• Impact significantly bigger and more
important than evaluating, or
monitoring, or measuring.
33. WHY PLAN YOUR IMPACT?
Strategy
Communication
Fundraising
Measurement
34. LANGUAGE AND LOGIC
Resources
used to
create the
services
offered
What is done
with the inputs,
to enable it to
develop and
deliver its
outputs
Products,
services or
facilities that
result from
activities
The changes,
benefits,
learning or
other effects
that result from
what the
project or
organisation
makes, offers or
provides
Broader or
longer-term
effects of a
project’s or
organisation’s
activities,
outputs and
outcomes
Input Process Output Outcome Impact
35. More clients get and sustain jobs
Improved work experience
Skills
workshops
Improved basic skills
More
appropriate
behaviour at
work
Greater
confidence
Increased
knowledge of
job
market
Work
placements
Peer mentoring
Reduction in youth unemployment regionally
YOUTH
EMPLOYMENT
PROJECT
36. HOW DO YOU DO IT?
Describe WHAT
change needs to
happen first
Show HOW
changes will be
made by
delivering outputs
Long-term
impact
Necessary
pre-condition
Necessary
pre-condition
Necessary
pre-condition
Necessary
pre-condition
Necessary
pre-condition
Outcomes
must be
achieved
BEFORE long-
term impact
Output Output
37. MONITORING AND EVALUATION FRAMEWORK
3
7
CORNERSTONE OF (SELF) EVALUATION:
Outcomes Outcome
indicators
Information
collection
methods
When and by
whom
How to report
and use
Outputs Output
indicators
Information
collection
methods
When and by
whom
How to report
and use
38. “NCVO CES offers consultancy, training and
publications to the voluntary sector and its
funders.
This increases knowledge and skills in
evaluation, and helps people collect better
data to inform their decision making.
This helps them improve their services
which improves the lives of vulnerable
people at the frontline.”
SUPER QUICK THEORY OF CHANGE
39. EXERCISE:
You have 3 minutes to create your own mini
theory of change, in three sentences:
1. Describe what you do
2. The changes you bring about
3. The long-term, wider change that your
work contributes to
Max 20 words a sentence!
Share it with your neighbour
See if you can help improve each others’
40. FIND OUT MORE
Theory of change:
• Making Connections: Using a theory of change to develop
planning and evaluation
Free download at www.ces-vol.org.uk
• Creating your theory of change: NPC’s practical guide
Free download at www.thinknpc.org
Impact practice:
• Inspiring Impact - http://inspiringimpact.org/
Email: ces@ncvo.org.uk
Tel: 020 7520 3193
Twitter: @CESOnline
46. IDENTIFY WHAT YOU ARE TRYING TO
ACHIEVE/USE YOUR THEORY OF
CHANGE
INVOLVE OTHERS AND WORK
TOGETHER
IDENTIFY CLEARLY WHY YOU WANT
TO ASSESS IMPACT AND WHO IS IT
FOR?
BE REALISTIC AND PROPORTIONATE
PLAN
47. MAKE USE OF AVAILABLE TOOLS AND
FRAMEWORKS
BE PROPORTIONATE – WHAT IS
NECESSARY? WHAT IS MOST
USEFUL?
USE EXISITING INFORMATION AND
EVIDENCE
DO
49. BUILD IN ENOUGH TIME FOR
ANALYSIS AND MAKING SENSE OF
DATA
THINK ABOUT POSITIVES AND
NEGATIVES – BE HONEST AND OPEN
ASSESS
50. BE CREATIVE IN HOW YOU SHARE
AND USE YOUR FINDINGS
USE THE LEARNING – BE WILLING TO
CHANGE YOUR VOLUNTEER
PROGRAMME/PROJECT AND ACT ON
WHAT YOU FIND
REVIEW
FEEDBACK TO THOSE INVOLVED
CELEBRATE!
51. IVR undertakes research, consultancy and
evaluation on volunteering and is part of NCVO.
www.ivr.org.uk
For further information on the toolkit:
www. ncvo.org.uk/practical-
support/publications/volunteering
Next toolkit training will be on July 11th in London
www.ncvo.org.uk/training-and-events
5
52. Impact – Challenges and Tools
Claire Knight – Breast Cancer Care
Rebecca Stewart – Age UK
May 2016
54. Challenges
Third Sector purpose is to make social progress
£43Bn income – means that we can (and do!)
change millions of lives
Donations, and funding, means scrutiny
Every project should ask: ‘what difference is being
made ?’
Impact, and not simply delivery
55. Standardising impact
assessment
Umbrella bodies are well-placed to drive
standardisation
Rationale – help the sector make more informed and
visible comparisons
In order to do more that works
60. Where did we start?
• Designed a process for capturing our views
• Created a dummy project to test each tool
• Established criteria to measure against
• Created a list of tools to consider
• Invited Sally Cupitt to talk to us
61. Which tools & resources
would we review?
• VITA / VIAT
• VODG / NCF
• Better Impact
• Volunteers Count
• Inspiring impact
• Social Return on Investment
• Balanced scorecard
• Theory of change
62. What did we learn?
• Information & guidance is easily available
• There are plenty of tools to choose from
• It’s important to pick the right tool for the right job
• Plan ahead to ensure the right data is collected
• Beware the common pitfall: measuring outcomes of
projects, instead of impact
63. Next steps
• Complete review of all tools
• Share with all NNVIA members (September 2016)
• Asked NNVIA members if anything missing?
• Decided to invite Sally Cupitt to speak to all NNVIA
members (June 2016)
• Create a “Which Guide” to share
66. Volunteering in care homes pilot project
• 3-year Department of Health funded project
• Place volunteers in befriending and activity-
based roles in care homes
• Partnership model between NCVO, Volunteer
Centres (5) and Care Homes (15)
• Over 259 volunteers, almost 10,000 hours
over two years
• Evaluation aimed to draw out good practice
learning and assess impact
67. The context
• Burdens on care homes
• Exceptionally high time pressure
• Many new to volunteering
• One-step removed from evaluation
• Particular user group - especially dementia
• 80% of residents dementia or severe
memory problems
• Capacity to consent
• Has to be face-to-face
68. Changing and flexible pilot
• Not a set ‘intervention’
• Uncertainty around focus of impacts
• Two stage process
• Interim evaluation – explore impacts and
reflect
• Project completion – assess impact
• Further research
69. Social & emotional vs mental & physical
Social & emotional vs mental & physical
• Take a broad approach initially
• Draw on existing frameworks
• VIAT
• ASCOF, Senses and CQC
• Combining can invalidate the tools and
reduce comparability
• Try to explore all types of impact
70. Subjective vs objective
• Practical challenges for objective
measures
• care home burden / non-disclosure
• residents
• Subjective reliable for wellbeing
• Subjective but systematic
• Link with existing evidence
• … but does not satisfy everyone
71. Staff perception of impact
on residents (n=29 of 38)
(%)
Major
positive
Moderate or
minor
positive
No
impact
Major,
moderate or
minor negative
Emotional wellbeing 62 31 3 3
Social wellbeing 59 38 3 0
Mental wellbeing 48 45 7 0
Physical wellbeing 41 38 21 0
72. Return on investment
• Volunteer Investment and Value Audit
• Investment – staff time, training, expenses
• Value – equivalent wage rate per volunteer
hour
• Advantages – very simple, total output, gives
a monetary value, comparability
• Limitations – only measures output,
monetising a non-monetary phenomenon,
adopts market conception of value
74. Quant vs qual (stats vs stories)
• Not an either or
• … but triangulation of data
• Qualitative case studies and in-depth
interviews allows deep understanding of
impacts
• …and the factors that underpin them
• Involved residents and relatives
75. "If I didn't have the volunteer my morale
would be very low. I'd be shouting at
everybody - all of us in here have some
form of dementia. I ring this bell and
nobody comes and nothing happens and
then I scream at the matron and say you
haven't got enough staff and she gets
very annoyed when I say that" [Resident]
76. “instead of sitting in their rooms…. keep
people thinking the whole time and
keeping the brain alert instead of being a
dodo” [Resident]
77. “The communication has broken down a
little bit.. we seem a little bit out of the
loop…it’s a bit disappointing really. Cos we
felt like we were building a rapport with the
volunteers – it seems to have been a bit of
a breakdown really… We need to go back
down to basics …. and start from scratch”
[Home Manager]
79. Communicating impact
• Reports and presentations
• Video:
• Need to convince people to engage with
research outputs
• Helps give the context for understanding
impact
• Promotional video but underpinned by
evaluation findings…
81. References:
Cabinet Office (2015) Community Life Survey 2014-15, Cabinet Office
Naylor, C., Mundle, C., Weaks, L., & Buck, D. (2013) Volunteering in health and care: Securing a sustainable future,
King’s Fund
Alzheimer’s Society (2014) Statistics. Available at
<http://www.alzheimers.org.uk/site/scripts/documents_info.php?documentID=341>
See details of the project. Available at https://www.ncvo.org.uk/practical-support/volunteering/volunteering-in-care-
homes
Davis Smith, J., Gaskin, K., Ellis, A. & Howlett., S. (2015) Volunteering Impact Assessment Toolkit: A practical guide for
assessing the difference that volunteering makes, Institute for Volunteering Research. Available at
https://www.ncvo.org.uk/component/redshop/1-publications/P78-volunteering-impact-assessment-toolkit
Nolan, M. R., Brown, J., Davies, S., Nolan, J. And Keady, J. (2006) The Senses Framework: improving care for older
people through a relationship-centred approach. Getting Research into Practice (GRiP) Report No 2. Sheffield Hallam
University
Gardiner, C. & Barnes, S. (2016) The impact of volunteer befriending services for older people at the end of life:
Mechanisms supporting wellbeing Progress in Palliative Care
Fujiwara, D., Oroyemi, P. & McKinnon, E. (2013) Wellbeing and civil society Estimating the value of volunteering using
subjective wellbeing data Cabinet Office
Kirkman, E., Sanders, M. & Emanuel, N. (2015) Evaluating Youth Social Action, An Interim Report: Does participating
in social action boost the skills young people need to succeed in adult life? Cabinet Office
Ellis Paine, A., McKay, S. & Moro, D. (2013) Does volunteering improve employability? Evidence from the British
Household Panel Survey Third Sector Research Centre
82. Hostedby:
Impact of volunteering on
wellbeing and mental health
Dr. Daiga Kamerāde, Third Sector
Research Centre, University of
Birmingham
d.kamerade@bham.ac.uk
84. Evidence
Systematic reviews and meta
analyses
Randomised controlled trials
(experiments)
Cohort and longitudinal panel
designs
Coho
Cross-sectional surveys
Cro
Qualitative interviewsSee reviews:
(Kamerāde, 2015,
Jenkinson et al.
2013).
85. Evidence summarised
– The impact of regular volunteering on well-being is positive, more so
for a sustained regular volunteering and for people with lower levels
of wellbeing to start with ( Binder, M., & Freytag, A. ,2013).
86. How to measure impact of volunteering on
wellbeing and mental health?
• Subjective wellbeing (see Dolan, P., Layard, R., & Metcalfe, R. 2011) (Scale 0-10):
– Life satisfaction:
• Overall, how satisfied are you with your life nowadays?
• Overall, how satisfied are you with the area you live, relationships , employment
prospects etc
– Happiness:
• Overall, how happy did you feel yesterday?
– Anxiety:
• Overall, how worried did you feel yesterday?
• Overall, how worried are you about your financial situation, environmental problems etc.
– Worthwhileness
• Overall, how worthwhile are the things that you are doing in life? (e.g. Volunteering)
• Mental health (presence/absence of depression)
– The 5-item World Health Organization Well-Being Index (WHO-5) (see Topp, et al. 2015).
• Please indicate for each of the 5 statements which is closest to how you have been feeling over
the past 2 weeks
• Over the past 2 weeks ... I have felt active and vigorous, ... my daily life has been filled with
things that interest me etc (Scale 5 (all the time) to 0 (at no time)
87. A basic experimental design for estimating the effects
of volunteering on wellbeing.
Peoplewhohave
expressedan
interestto
volunteer
Experimental
group
(volunteers)
Control
group
(waitinglist)
Pre-test
Pre-test
Volunteering
Not
volunteering
Post-test
Post-test
are
randomly
assigned
to
=
88. Implications for policy and practice
• Volunteering as a public wellbeing and mental health
intervention?
– Not enough robust evidence
– Robust pilots and their evaluation is needed.
• Claims about volunteering:
– ‘Volunteering may improve your wellbeing and mental health’ [not ‘boosts,
improves, or increases’].
– ‘Nearly 60% of 70 volunteers in this volunteering programme reported
improved wellbeing’.
• Design&Evaluation:
– Lenght, frequency and type of volunteering: any realistic chances and mechanisms for
improving wellbeing?
– If wellbeing and mental health are the key expected outcomes&there is a relatively large
number of volunteers: consider pragmatic randomised controlled trials.
– If improved wellbeing and mental health are not the key expected outcomes& there are
very few volunteers: different research questions and variety of designs.
– Specific wellbeing measurements more informative than global/general ones.
89. More information
• Kamerāde, D. (2015) Third sector impact on human
resources and community: a critical review. TSRC
Working Paper Series No.134. Seventh Framework
Programme (grant agreement 613034), European
Union. Brussels: Third Sector Impact
• Jenkinson, C., Dickens, A., Jones, K., Thompson-Coon,
J., Taylor, R., Rogers, M., et al. (2013). Is volunteering
a public health intervention? A systematic review
and meta-analysis of the health and survival of
volunteers. BMC Public Health, 13(1), 773.
90. References
Binder, M., & Freytag, A. (2013). Volunteering, subjective well-being and
public policy. Journal of Economic Psychology, 34, 97-119.
Diener, E., Oishi, S., & Lucas, R. E. (2009). Subjective Well-Being: The Science
of Happiness and Life Satisfaction. In Oxford handbook of positive
psychology (pp. 187)
http://homepage.psy.utexas.edu/homepage/Class/Psy418/Josephs/Wynne
%20Folder/5-Subjective%20Well-Being.pdf
Dolan, P., Layard, R., & Metcalfe, R. (2011). Measuring subjective well-being
for public policy.http://eprints.lse.ac.uk/35420/1/measuring-subjective-
wellbeing-for-public-policy.pdf
Musick, M. A., & Wilson, J. (2003). Volunteering and Depression: the Role of
Psychological and Social Resources in Different Age Groups. Social Science
and Medicine, 56(2), 259-269.
Topp, C. W., Østergaard, S. D., Søndergaard, S., & Bech, P. (2015). The WHO-5
Well-Being Index: A Systematic Review of the Literature. Psychotherapy
and psychosomatics, 84(3), 167-
176.http://www.karger.com/Article/Pdf/376585
So, if we’re going to move forward the debate on what ‘gold standard’ transparency’ looks like then I think that we need to move the debate beyond the narrow confines of money and instead think about the broader way that organisations might function or work and think about what being an open, networked nonprofit might mean for them.
As a starter for 10, I think we need to debate what gold standard transparency might mean across these 7 domains.
And for the avoidance of doubt, lets be clear: gold standard is not a euphemism for more. I cant agree with those who seek to give transparency a bad name by simply dumping data or producing 200 page annual reports and accounts: obfuscation via increased disclosure is a classic strategy that ultimately has the opposite effect to what is needed.
Gold standard transparency is about better, not more; it is about data and narrative, about stories and explanations, so that data is not marooned and left to others to explain.
Gold standard transparency is about process and how we work, not just counting outputs (or even outcomes), so that our stakeholders can ‘puncture the membrane’, so that they can see inside – a real opportunity given our challenge of public understanding
So, if we’re going to move forward the debate on what ‘gold standard’ transparency’ looks like then I think that we need to move the debate beyond the narrow confines of money and instead think about the broader way that organisations might function or work and think about what being an open, networked nonprofit might mean for them.
As a starter for 10, I think we need to debate what gold standard transparency might mean across these 7 domains.
And for the avoidance of doubt, lets be clear: gold standard is not a euphemism for more. I cant agree with those who seek to give transparency a bad name by simply dumping data or producing 200 page annual reports and accounts: obfuscation via increased disclosure is a classic strategy that ultimately has the opposite effect to what is needed.
Gold standard transparency is about better, not more; it is about data and narrative, about stories and explanations, so that data is not marooned and left to others to explain.
Gold standard transparency is about process and how we work, not just counting outputs (or even outcomes), so that our stakeholders can ‘puncture the membrane’, so that they can see inside – a real opportunity given our challenge of public understanding
So, if we’re going to move forward the debate on what ‘gold standard’ transparency’ looks like then I think that we need to move the debate beyond the narrow confines of money and instead think about the broader way that organisations might function or work and think about what being an open, networked nonprofit might mean for them.
As a starter for 10, I think we need to debate what gold standard transparency might mean across these 7 domains.
And for the avoidance of doubt, lets be clear: gold standard is not a euphemism for more. I cant agree with those who seek to give transparency a bad name by simply dumping data or producing 200 page annual reports and accounts: obfuscation via increased disclosure is a classic strategy that ultimately has the opposite effect to what is needed.
Gold standard transparency is about better, not more; it is about data and narrative, about stories and explanations, so that data is not marooned and left to others to explain.
Gold standard transparency is about process and how we work, not just counting outputs (or even outcomes), so that our stakeholders can ‘puncture the membrane’, so that they can see inside – a real opportunity given our challenge of public understanding
Impact is the difference you make.
A way of thinking.
Embodied in strategy, embraced by leadership.
Achieving impact: why a voluntary organisation exists!
So, if we’re going to move forward the debate on what ‘gold standard’ transparency’ looks like then I think that we need to move the debate beyond the narrow confines of money and instead think about the broader way that organisations might function or work and think about what being an open, networked nonprofit might mean for them.
As a starter for 10, I think we need to debate what gold standard transparency might mean across these 7 domains.
And for the avoidance of doubt, lets be clear: gold standard is not a euphemism for more. I cant agree with those who seek to give transparency a bad name by simply dumping data or producing 200 page annual reports and accounts: obfuscation via increased disclosure is a classic strategy that ultimately has the opposite effect to what is needed.
Gold standard transparency is about better, not more; it is about data and narrative, about stories and explanations, so that data is not marooned and left to others to explain.
Gold standard transparency is about process and how we work, not just counting outputs (or even outcomes), so that our stakeholders can ‘puncture the membrane’, so that they can see inside – a real opportunity given our challenge of public understanding
People want to do good. They don’t care in which sector they do it.
So for the time precious, the cash poor, the outcome is the same: if they don’t think that we are using their resource to make the biggest impact, we wont be in the business of doing good.
Note this is a relative proposition, not an absolute. It’s no longer good enough to say we do good in the voluntary sector.
Money
Where does it come from and what do you spend it on?
Sorry, I know this is dull, but people want to know. We’re behind government on this.
And the likes of the NAO want to be able to ‘follow the money’. They’re firmly of the view that transparency reduces graft: sunlight is the best disinfectant
Wake up call: xBRL is coming. Then our financial data will be machine readable. So if we don’t sort the narrative out, someone else will do it for us. I think that this is incredibly important.
So, if we’re going to move forward the debate on what ‘gold standard’ transparency’ looks like then I think that we need to move the debate beyond the narrow confines of money and instead think about the broader way that organisations might function or work and think about what being an open, networked nonprofit might mean for them.
As a starter for 10, I think we need to debate what gold standard transparency might mean across these 7 domains.
And for the avoidance of doubt, lets be clear: gold standard is not a euphemism for more. I cant agree with those who seek to give transparency a bad name by simply dumping data or producing 200 page annual reports and accounts: obfuscation via increased disclosure is a classic strategy that ultimately has the opposite effect to what is needed.
Gold standard transparency is about better, not more; it is about data and narrative, about stories and explanations, so that data is not marooned and left to others to explain.
Gold standard transparency is about process and how we work, not just counting outputs (or even outcomes), so that our stakeholders can ‘puncture the membrane’, so that they can see inside – a real opportunity given our challenge of public understanding
Tory manifesto: unleash the power of PbR in charities
Expand PBR to ‘harness the talent and energy of charities’ and halve the disability employment gap
Target of one third of government spend on SMEs
Innovation in public services including through our sector
We need to start thinking about theories of change, and log frames: in other words, what’s the relationship between the changes we see in the world and the inputs that we are using…or asking for.
Input: Pie mix
Output: Pies made (100 pies made etc), pies sold (Number pies sold, year on year)
Outcome: More pies consumed (Customer surveys etc.), customer satisfied (repeat business, surveys, interviews etc.).
Impact: Obesity etc. (NHS Statistics – analysed pie eating Vs. non-pie eating, randomised testing, control groups etc.).
Impact
What difference do you make? How do you measure this? What can and can’t you measure? What you don’t measure and why? How effective are you compared to other interventions?
Intended outcome: gold standard transparency creates an environment where it is safe to fail, or fail forward.
So, if we’re going to move forward the debate on what ‘gold standard’ transparency’ looks like then I think that we need to move the debate beyond the narrow confines of money and instead think about the broader way that organisations might function or work and think about what being an open, networked nonprofit might mean for them.
As a starter for 10, I think we need to debate what gold standard transparency might mean across these 7 domains.
And for the avoidance of doubt, lets be clear: gold standard is not a euphemism for more. I cant agree with those who seek to give transparency a bad name by simply dumping data or producing 200 page annual reports and accounts: obfuscation via increased disclosure is a classic strategy that ultimately has the opposite effect to what is needed.
Gold standard transparency is about better, not more; it is about data and narrative, about stories and explanations, so that data is not marooned and left to others to explain.
Gold standard transparency is about process and how we work, not just counting outputs (or even outcomes), so that our stakeholders can ‘puncture the membrane’, so that they can see inside – a real opportunity given our challenge of public understanding
We exist to benefit the public
Intro me and NCVO CES.
Karl’s set the scene in talking about the importance of measuring impact. I’m now going to talk about how you might go about the first stage of focusing on your impact – planning it.
Often overlooked. People jump into measuring. Or get to the end of a project and wonder how to measure what achieved.
Equally applicable to impact of or on volunteers as paid staff.
I’ll explain why and how you can plan impact for 10 or 15 minutes, then we’ll try something out.
Put it in context.
Process broken down into cycle of 4 stages.
Impact practice – not impact measurement.
Plan - what want to achieve and how you’ll measure whether you have or not
Do - Then do the delivery and collect information
Assess – make sense of the information and draw conclusions
Review – learn from your findings, use them to improve and communicate to others.
Measuring is just one of the things you can do with your impact.
Broadly – you need to know what you want to achieve to work out whether or not you’ve achieved it. If you don’t know where you’re going, how will you know when you’ve got there?
Helps with:
Useful on it’s own to have planned impact. Everyone knowing where they’re going and the best ways to spend your time to make biggest diff
Tell people why you exist, why you do the work – external and internal people
Funders will want to know what you plan to achieve with their money
So you can work out how to measure it
Talk a bit about the terms you might use in planning.
Terms around impact used differently, but if all know what we mean in a given situation – fine.
1st two widely accepted/understood, last few less so.
Inputs – finances, staff
Process – projects, activities
Outputs – events, drop-in centre, befriending scheme
Outcomes – increased confidence or knowledge, better employability
Impact – reduction in youth unemployment locally, increased biodiversity, reduced loneliness in older people
Suggest you plan backwards – start with impact, ultimate purpose, then work out steps along the way and what you need to do, what money do you need?
Reverse it to tell your story.
Drawn in this format - called a logic model.
Some people use this to plan for straightforward projects or small orgs.
Another thing used is ToC – more complex.
Some may have heard of theory of change. Some may have one already – think about how yours is similar or dissimilar. find exercise easier.
Different definitions and interpretations, but one definition we use is:
A description of why a particular way of working will be effective, showing how change happens in the short, medium and long term to achieve the intended impact.
Often a map showing outcome chains, with an accompanying narrative (not shown).
Outputs are in purple – products or services produced. Chain of outcomes showing how change happens in short, medium and long term to achieve the impact.
All expressed using words of change: increased, greater, more, reduction.
Shows complexity of change. Good for showing progressive or related changes. Shows interrelationships between outputs and outcomes.
Here is A way of doing it:
1. Identify the problem, who is affected, underlying causes, and key contextual factors – high level of unemployment amongst young people in local area. Issues of skills, knowledge and confidence to get into work.
Agree long-term intended impact – at top. Work backwards like logic model.
Map outcomes backwards, thinking of them as necessary pre-conditions – what do clients need in order to…
Identify outputs necessary to achieve outcomes – what do we need to provide to create those changes? Might come in between levels
Identify assumptions – the ‘givens’ underlining your theory – for example, assumed the local economy stays fairly stable and there are jobs available. Assumed that peers will know about the job market and pass that on. Looking for plausible explanation of outcomes.
Tell it backwards – here’s what we do and what difference it makes
THEN can think about planning how you’ll measure things - consider what will show you whether or not each thing has been achieved – indicators. And the best way to measure that.
For appropriate behaviour at work an indicator might be level of punctuality. Increased knowledge of the job market – indicator could be number of jobs
We suggest you store in an M&E framework. Can check you’ve got everything covered and also create a plan for how you will collect the info etc.
Not going to cover in detail. Find examples of this on the NCVO CES website.
Developing ToC and M&E framework takes time and effort.
Most popular of CES’ blogs – 10 min theory of change challenge. Looking for a shortcut. Actually an introduction to the idea and way to briefly tell people about their intended impact. Even though it won’t give you a proper theory of change, jut articulating this quickly is useful to start you off.
For example – CES’ theory of change in brief.
Even slimmer than the 10 min theory of change, I challenge you to a 3 minute ToC.
How did it go? Easy? Hard?
2 volunteers to feed back. (write it down as they say it). Audience to comment.
mentioned activities throughout? were succinct enough? Did things blur or bleed into each other?
Keep that in mind today – what you want to achieve and what you do, when thinking about how you might measure it.
Want to know more?
We have a free guide on theory of change
As do NPC.
Looking to improve your impact practice generally – II is a national collaborative programme with lots of free resources. In particular there’s an online self-assessment tool to help you work out how good you are at all of the stages of impact practice, including planning.
Our contact details – if CES can help with any support around this area.
Discuss issues of impact through the lens of one particular project – the Evaluation of the Volunteering in care homes project.
So, I’ll briefly introduce the project;
Wont just present the findings (although you can read a lot more on the NCVO website – actually going to raise the issues around impact -
Raise some of the issues around impact;
How we attempted to overcome them – although I’m going to try to be quite critical of the approach
Then actually going to finish by showing a video that was produced out of the back of the project.
Impact on volunteers
Stories of survey M and S vouchers up front - even that hardly worked, had to fight for every response. Voluntary
One step removed – used to evaluating commissioners or on behalf of funder of projects - could compensate them
What we saying here…
Daiga will explore these
Insert results table for impacts - !!!!!!!
Meaningful social relationships reduce hospital admissions – well we can measure ability to deliver meaningful social relationships (not necessarily)
May also make us question the subjective impacts e.g. skills improvement is not the same as employment outcomes.
Problem is the evidence base isn’t there but can draw on it.
Asked staff and volunteers about a range in an impact survey –
Also asked them about impacts on themselves – much more reliable, and their perception of impact on residents. (very high)
Insert spread sheet – with findings!!!
Wayne Rooney –
Some labour is systematically undervalued
More normal – cost-effective
Not an implausibly high
Not a solution to the funding crisis
In some settings you can get a 1 to 10 ratio. Not here.
Overall, the project has not made a return on investment –
Value - £123,394
Investment - £147,639
Methodologically it is important but is about different audiences
Not about quotes – really understanding things
Defining what SWB and mental health is
Mental health is defined as a state of well-being in which every individual realizes his or her own potential, can cope with the normal stresses of life, can work productively and fruitfully, and is able to make a contribution to her or his community.
Both related to wide range of factors
Baseline of SWB (genetical predisposition), stability over time. Set ponit or equilibrium. Dynamic equilibrium model: set point plus change due to events, then retun to the set point.
Strongest predictors: health (physical and mental), life events.
Hedonic adaptation. Most people report SWB over average (slightly happy)
Nearly 10,000 papers in 2013.
Controlling systematic errors (bias) - Increasing strenght of evidence for causality
What questions can or cannot each of the types of evidence answer (what we or don't know from each type of studies)
Qualitative studies: what outcomes in specific context. //how typical are these volunteers and the volunteering effects?
Cross-sectional surveys: how does wellbeing and mental health different between volunteers and non/volunteers.
Longitudinal panel studies: can: what happens while volunteer, cannot determine causaltiy:
De Wit,Bekker et all 75% of the difference in well -being between volunteers and non -
volunteers is due to selection effects. When these selection -effects are removed On average, volunteering contributes to a
0.7% increase well-being.
Cohort studies
Randomised controlled trials: - some claim that by definition volunteering is voluntary therefore randomised controlled trials are not possible. In fact they are: randomise volunteers.
Systematic reviews and meta analyses
Waiting list- might undergo training, for example.