1. 1
Evidencing Impact and Value
Alison O’Hara
Chief Executive
Audiences North East

2. 2
Impact
Image: courtesy of Hales Gallery, London
and I-20 Gallery, New York

3. 3
Definitions
• Impact: the effect or impression of one thing against
another
• Economic impact: Radich(1987), the economic impact of
a given phenomenon can be defined as ‘the effect of that
phenomenon on such economic factors as the economic
behaviour of consumers, businesses, the market,
industry, the economy as a whole, national wealth or
income, employment and capital’
• Social impact: Landry et al (1993), ‘those effects that go
beyond the artefacts and the enactment of the event or
performance itself and have a continuing influence upon,
and directly touch, people’s lives’

4. 4
Value
Image: courtesy of BePak

5. 5
Definitions
• Value: worth in usefulness or importance, utility
or merit
• Matarasso (1996), which value systems are
used to provide benchmarks against which work
will be measured, and about whom defines
quality, value and meaning
• Social and economic rationales for the arts, with
their emphasis on the arts as a means to other
ends will serve to devalue arts for its own sake
eg John Tusa

6. 6
Current drivers for measuring
impact and value
• Economic downturn
– Local and national governments invest public
money in cultural opportunities. How do we
know if the money is well spent or wasted?
Does a £1 invested by government deliver
more than a £ in benefit?
• The Big Society

7. 7
Cuts in public investment
• DCMS – 25% cut for 2011/12 to 2014/15
• ACE cuts – 29% cut, 6.9% cut for majority
of RFOs for 2011/12 and reduction of
14.9% to overall budget available for
RFOs for 4-year settlement period
• National museums – 15% cut and remain
free to enter
• English Heritage – 32% cut
• Visit Britain – 34% cut

8. 8
The Big Society
• www.cabinetoffice.gov.uk Building the Big
Society
• Five priorities:
– Give communities more power
– Encourage people to take an active role in their
communities
– Transfer power from central to local government
– Support co-ops, mutuals, charities and social
enterprises
– Publish government data

9. 9
Measuring Intrinsic Value
• Measuring Intrinsic
Value: How to stop
worrying and love
economics (2009) by
Hasan Bakhshi, Alan
Freeman and Graham
Hitchen
Image: Jaipur Kawa Brass Band,
Durham BRASS Fest

10. 10
Measuring Intrinsic Value
• Recommends cultural sector embracing
the need to measure economic impact
• Cultural world will back itself into a corner
if it claims immunity from measuring
economic impact when money could be
spent on other things
• Even though it’s difficult to measure, it’s
not really optional – governments choose
between alternative expenditures

11. 11
Intrinsic v instrumental
• Intrinsic value (own merits) v instrumental value
(non artistic side effects eg social inclusion,
crime prevention and learning)
• Proper assessment of intrinsic value
• Public’s own valuation of the arts – contingent
value (CV) and willingness to pay (WTP)
• Ask the public what they would be prepared to
pay, faced with a choice of spending the money
on something else

12. 12
Culture and Sport Evidence
Programme (CASE)
• Vision is to influence culture and sports’
policy development and policy agenda
through the development of strategic,
policy-relevant, high-quality, cross-cutting,
social and economic evidence base for
culture and sport sectors
– Current research
– Research needs to provide
– Tools to collect research

13. 13
CASE
• Publication
– Understanding the drivers,
impact and value of
engagement in culture and
sport, July 2010
www.culture.gov.uk
• Database
– www.impact.arts.gla.ac.uk/
c5,800 individual studies or
reviews on the drivers,
impact and value of
engagement in culture and
sport, advanced search
terms, links to reports
Image: The Alnwick Garden

14. 14
Benefits of engaging in culture and sport
Individual engager
Achievement
Continuity with the past
Creativity
Diversion
Enjoyment
Escape
Expression
Health
Income
Inspiration
Knowledge of culture
Self-esteem
Self-identify
Skills/competency
Solace/consolation
Community
Bequest value
Community cohesion
Community identity
Creativity
Employment
Existence value
Innovation
Option to use
Productivity
Reduced crime
Shared experience
Social capital
National
Citizenship
International reputation
National pride

15. 15
CASE publications
• Measuring short-term private benefit of
engagement
– Subjective well-being measures, two-step approach
• First survey data is used to estimate how a person’s SWB
changes when they engage in culture and sport
• This change in SWB is valued monetarily using the ‘income
compensation approach’ ie the analysis estimates the
increase in SWB generated by an increase in income
– Deriving economic values from SWB
• The income compensation approach can be used to convert
estimates of the SWB effect of policy outcomes such as
engagement in culture and sport into estimates of monetary
value of these policy outcomes.

16. 16
CASE publications
• Measuring long-term public benefit of
engagement
– Due to data and evidence constraints work
was focused on the health gains associated
with doing sport
• However, CASE regional insights have
been developed to provide partners with
data and evidence on the position and role
of culture and sport within key local and
regional agendas

17. 17
CASE database

18. 18
Impact and value frameworks
• Arts Council England - self evaluation
framework
• MLA - Inspiring Learning
• Local Government Improvement and
Development (formerly IDeA)
• Audiences London – festivals and outdoor
events
• New Economics Foundation – theatre
• Film Council – impact of local cinemas

19. 19
Arts Council England – self
evaluation framework
• Online flexible
development tool to
support arts organisations
in evaluating their own
performance and to help
them inform their future
planning, July 2010
• www.artscouncil.org.uk/s
elfevaluation/background-
self-evaluation/
Image: Constellation by Kiki Smith,
National Glass Centre

20. 20
Arts Council England – self
evaluation framework
• Six key areas:
– Vision
– External environment
– Artistic aspirations and programme
– Participation and engagement
– Organisational capacity and capability
– Business model
• Each area has three strands:
– Topic – a breakdown of the key area into smaller areas of focus
– What success looks like – a high-level description of an organisation
that is delivering each topic effectively
– Questions you might ask yourselves – types of questions you might ask
to evaluate your strengths and weaknesses relating to that topic
– Resources – website links

21. 21
MLA - Inspiring Learning
• A self-help improvement framework for
museums, libraries and archives, 2008
• Inspiring Learning supports organisations to:
– Assess their strengths and plan improvements
– Provide evidence of the impact of their activities
through the generic learning and generic social
outcomes
– Improve their strategic and operational performance
• www.inspiringlearningforall.gov.uk/

22. 22
MLA - generic learning outcomes
• The generic learning
outcomes are
underpinned by a broad
definition of learning
which identifies benefits
that people gain
from interacting with
museums, libraries and
archives
• www.inspiringlearningfora
ll.gov.uk/toolstemplates/g
enericlearning/
Image: Killhope

23. 23
MLA - generic learning outcomes
• GLO checklist
• Recording and
analysing
qualitative and
quantitative data
• GLO coding
• Templates

24. 24
MLA - generic social outcomes
• Case studies, guidance and
tools to support museums,
libraries and archives in
planning how they deliver their
services and in measuring
their contribution to social
outcomes
• Step-by-step approach to
using the guidance, sample
questionnaires and case
studies
• www.inspiringlearningforall.gov
.uk/toolstemplates/genericsoci
al/index.html
Image: Sunderland Museum and
Winter Gardens

25. 25
MLA - generic social outcomes
• Three outcome areas that
relate to government policy
priorities
• Look at organisation’s high
level aims
• Identify which audiences are
priorities to target
• Consider how your
project/service contribute to
one or more of the GSOs
• Decide what evidence you
need to gather

26. 26
Local Government Improvement
and Development
• Formerly IDeA new web resource on how
to create a local outcomes framework for
culture and sport, September 2010
• Local outcomes framework will help you
measure and evidence the contribution
culture and sport provision makes to better
outcomes for your area
• www.idea.gov.uk/idk/core/page.do?
pageId=21649171

27. 27
Local outcomes framework -
process

28. 28
Local outcomes framework -
resources
• Guidance on what a local culture and sport
outcomes framework can help you do
• What should an outcomes framework include:
– Examples of outcomes triangles: children and young
people, economy, environment, health and well-being,
older people, safer communities, stronger
communities
– Logic model templates
– Evidence template
– Performance indicators template

29. 29
Audiences London – festivals and
outdoor events
• Information sheets
available to download,
September 2010
• Key points to consider
when trying to measure
and evidence success or
impact of a festival
• www.audienceslondon.or
g/1891/our-
resources/advice-for-
audience-research-at-
festivals-and-outdoor-
events.html
Image: Stockton International
Riverside Festival

30. 30
Audiences London – festivals and
outdoor events
• Information sheets:
– Evidencing success
– Methods
– Using questionnaires
– Sampling
– Working with volunteers
– Research guidelines and data protection
– Measuring economic impact

31. 31
New Economics Foundation -
theatre
• Capturing the audience
experience – a handbook for
the theatre
• Presents a new model for
describing the audience
experience with standard
survey templates and
guidance on how to use them
• Value – explores what it is
about a good theatre
experience that makes it ‘worth
coming out for’
• www.itc-
arts.org/uploaded/documents/
Theatre%20handbook.pdf Image: Theatre Royal

32. 32
The Audience Experience
Framework

33. 33
Film Council - impact of local
cinemas
• Impact of local cinema –
five case studies, 2005
• Suggested tools for use
in further cinema impact
studies
• Methodological notes
• www.ukfilmcouncil.org.uk/
10022?
page=1&step=10&viewby
=category&value=17004
Image: Tyneside Cinema

34. 34
Film Council - case studies
• Desk research
– No. of screens and seats
– Type of programme
– Annual ticket sales
– Revenue from food/drink/merchandising, advertising, project funding,
other income
– Annual expenditure
– Type of location and town
– Population of town
• Site visit
• Depth interviews with cinema manager and staff
• Focus group with cinema audience
• Phone interviews with cinema’s local suppliers, community groups
and educational organisations, local council and local press

35. 35
Film Council - case studies report
• The social, cultural and environmental
impact of local cinema
• The impact of local cinema on the local
economy
• Appendix – tables which analyse
information (quantitative and qualitative)
by impact eg economic, social, cultural,
environmental

36. 36
Film Council - suggested tools for
further research
• Pack of research materials:
– Templates for data collection
– Topic guides for interviews with stakeholders
– Briefing notes for cinema staff
– Monitoring form
– Draft letters
– Forms for measuring spending

37. 37
Impact and value frameworks – the
highlights
• Arts Council England - self evaluation
framework
• MLA - Inspiring Learning
• Local Government Improvement and
Development (formerly IDeA)
• Audiences London – festivals and outdoor
events
• New Economics Foundation – theatre
• Film Council – impact of local cinemas