Special session: evaluating design enabled innovations – towards a common impact evaluation methodology

1. Evaluating design enabled innovations.
Lessons from Designscapes for a
common impact evaluation
methodology
Dr Joe Cullen and Dr Kerstin Junge
Tavistock Institute of Human Relations
22.9.2021

2. Designscapes

3. • Funded by the European
Union’s research excellence
programme Horizon 2020
(2017-2021)
• 12 organisations, in 10
European countries –
academia, consulting, public
sector
• To encourage the uptake and
scaling of Design enabled
Innovations by enterprises,
start-up companies, public
authorities, and other urban
stakeholders
Design
Innovation
Urban
Designscapes

4. Desigscnapes
Evaluation
Designscapes
research
activities
Designscapes
funding
Policy
recommenda-
tions
Designscapes
capacity
building
Designgscapes
communication
Designscapes activities

5. Designscapes evaluation
• To support the
Designscapes’ overall
vision, mission, objectives
and outcomes
• To develop a scalable
impact evaluation
methodology
Developmental evaluation
Contributing to the
project design and
its development.
Process evaluation
Monitoring project
progress and
assessing the
effectiveness and
efficiency of the
project delivery.
Outcome evaluation
Assessing project
outcomes and
impacts.

6. Designscapes evaluation
1. Does Designscapes support inclusiveness,
and reduce inequalities in citizens’ access
to innovation?
2. Does Designscapes support co-creation,
and does co-creation in turn lead to
successful innovation?
3. Does design-led innovation improve
performance and efficiency in the
commercial and public sector - by
addressing the challenges cities face now
and in the future - and hence improve
competitiveness?
4. What additional contribution does the
urban context make?
• Theory based evaluation design
(theory of change and
contribution analysis)
• Mixed method design, e.g.:
• KPI design and monitoring
• Partner interviews, online
surveys and workshops
• Case studies (multiple case
study design)
• Longitudinal stakeholder
survey
• Cost consequence analysis
• Regression modelling
• Systems mapping

7. Emerging findings:
Does design-led innovation improve performance and efficiency in the
commercial and public sector - by addressing the challenges cities face
now and in the future - and hence improve competitiveness?

8. Problem / context Activites Outputs Outcomes Impacts
DEI is potentially an excluding process, so those
who might benefit or have something to contribute
don’t participate.
[ inclusion is not a key element of design generally.
Always the same people involved]
The urban environment presents a specific
manifestation of wicked problems and is at a scale
where you can develop solutions.
Top down perspective: “Many public sector
organisations (role of citizens) are missing out on
DEI.” Things like organisational structure, role and
purpose of public sector organisations and attitudes
towards new ways of working can get in the way of
using DEI. The public sector is not as agile as the
private sector and has different challenges and roles
in society. There is fear of experimentation and
therefore a lot of the more innovative or creative
methods are not taken up. Lack know how and
experience in certain areas of DEI. Bad experiences
with using DEI and challenges around adopting new
solutions on policy side. Currently not many lcoal
policy documents that include DEI.
Design capabilities are not diffused in the urban
context (ie many different actors in the urban
environment know about DEI and can use it) and
cannot capture the potential of urban contexts to
generate innovation.
Problem to not utilise design has a geographic
dimension: much more evolved in society and
business of northern Europe than in other parts
Businesses, especially SMEs, miss out on the
potential to utilise DEI in the urban context as a
source for improving efficiency and stimulating
growth”. Maybe business is less competent at
involving people with disadvantages.
Concepts of design agency and design enabled
innovation are used differently by different types of
organisations. Design capabilities are different in
different geographic and organisational contexts.
There is a lack of evidence on the effectiveness of
service or strategic design and innovation (with some
exceptions, e.g DK). There is little use of evaluation.
Policy forums and mid-term conference
Application form design / content and application
process for pilots
Development and offering of a training programme to
round 2 and round 3 pilots, online and face to face.
(ultimately opened to everybody at the end of the
project)
An increase in public sector effectiveness and
private sector competitiveness, and sustaining
innovation capabilities and processes
Integration of design and user-driven innovation
into innovation policies and support mechanisms,
including funding programme
Global problems as manifested at urban level
improve Improvements to global priorities at urban
level: Contribution to Design enabled Innovation
sustainable urban quality, economic and social
values; sustainable innovation capability at urban
level.
An improvement in evaluation tools, skills and
capacity to evaluate the effectiveness of design in
the innovation process
Top down: local or national policies are changing
to include DEI and / or DEI are introduced (more
widely) into policy at local level.
Bottom up: design capabilities more diffused in
urban environments: mabny different actors’
capabilities are raised.
An open-ended and tranferrable, global
ecosystem of design-enabled innovators in urban
environments that can be copied and leveraged by
other cities.
Cities have increased their design capabilities.
Bottom up design policy making and project
initiatition encouraged
Increasing political awareness and interest
on DEI at urban level, plus effectiveness -
how can you show use / relevance of DEI in
local contexts?
Emergence of a strategy / policy to support
the transformation of urban ecosystems into
DEI ecosystems in other European (not
Designscapes) cities.
Different facebook groups and communities
linked together, building a large community of
DEI practitioners.
1 open Designscapes facebook group
1 closed Designscapes facebook group
X number of participants
X number of facebook and other groups linked
together
Citizens engaged in policy making processes;
greater awareness among policy makers about
opportunities and challenges of doing this
Greater confidence among urban / regional
policy makers to experiment with innovate DEI
tools
Integration of Green and White paper contents
into programmes and / or policies by policy
makers at urban level
Pilot teams advance new solutions to alleviate
wicked problems in the urban space
Pilot teams improve and / or sustain their
innovation capacity /capability at urban level
Increase in awareness about diffuse design
skills as a result of completing the application
form especially among those that did not start
from a design process perspective.
Greater familiarity among designscapes
audiences with state of the art then hopefully
they will be more willing to adopt innovative
tools.
Increased awareness of benefits of ToC and
evaluation in the sector
Better evidence on benefits DEI in urban
environments
Applicants create their own ideas / solutions
specific to their urban environments.
Designscapes Green paper
Designscapes White paper
Designscapes funds 100 pilots: 50 feasibility
studies, 40 prototypes, 10 scalability proofed
projects
Designscapes developes a public Design
toolbox describing 30 tools, instruments and
methods to enhance DEI in cities
X number of Designscapes events
X number of participants
Evaluation framework (D2.1)
At the end: a tested and “universal” evaluation
approach
Community of DE innovators through facebook
group consisting of those funded by the project
initiatlly. This might expand after the project.
Solutions and policy recommendations.
Policy examples implementing DEI in practice at
city level
Evaluation reports
Designscapes mentors allocated to round 2+3
funded pilots
Co-creation and co-construction of events and
strategies with other player
Facebook group for funded pilots as first stage for
community building
Other activities needed to create ecosystem,
connecting projects and initiatives. Thinking ongoing
on how to do this, possibly by creating a network of
mentors.
Creating physical spaces for DEI practitioners /
Designscapes stakeholders to connect.
Designscapes develops 3 rounds of an open call
call, addressing everyone in Europe and stressing
user co-creation and social inclusion
Dissemination activities: toolbox, blog, journal articles
Implementing evaluation at Designscapes and pilot level
Blogs on evaluation and DEI
Evaluation / ToC training
Researching policy landscape feeding into agenda for
Valencia policy forum
Connecting Design innovators/projects (online - e.g.
facebook- and / or through networking events /
activities?); also providing opportunities to connect
and showcase their work
Invite pilots to policy fora
Capacity building activities and working collaboratively
(with policy makers and others) to understand gaps in
capacities
Iterative toolbox matching needs with tools (WP4),
making sure needs are identified
Match initiatives: bringing together organisations and
matching the ‘right’ Designscapes partner to support
‘skilling up’
Assumptions:
Public and private sector organisations use DEI
differently and have different capabilities.
The project focuses on the urban system not specific
(private) businesses
Are we able to provide enough funding or support or
resources to sustain a community or availability of
financial incentives ?
Are we able to build a community?
Are we able to fund diversity (e.g. geographic) and
involve disadvantaged people?
Are we able to exploit the innovation potential of urban
environments?
Are we able to develop consensus among policy
makers?
Assumptions:
Designscapes is able to fund geographically diverse projects.
We are funding the ‘right’ projects to provide data and
epxerience helping to achieve the outcomes.
Projects and stakeholders access the training, support and
tools provided
Risk: reality of funded pilots might not help us achieve
outcomes; capacities in peripheral regions limited.
Assumption: we are able to ‘attract’ or develop
interest among policy audiences.
Risk: challenge of communicating
Designscapes vision due to the multitude of
terminology used.
Webinars and coffee mornings

9. Global problems as manifested at
urban level improve.
Design enabled Innovation
creates economic and social
value
The urban environment presents a specific
manifestation of wicked problems and is at a scale
where you can develop solutions.
Design capabilities are not diffused in the urban
context (ie many different actors in the urban
environment know about DEI and can use it) and
cannot capture the potential of urban contexts to
generate innovation.
Pilot teams advance these solutions alleviate wicked
problems in the urban space
Applicants create their own solutions specific to their
urban environments.
Designscapes funds 100 pilots: 50 feasibility studies, 40
prototypes, 10 scalability proofed projects
Designscapes develops 3 rounds of an open call call,
addressing everyone in Europe and stressing user co-
creation and social inclusion
DEI creates sustainable
innovation capability at urban
level.
Effectiveness and
efficiency causal strand
Assumptions:
• Designscapes is able to fund geographically diverse
projects.
• We are funding the ‘right’ projects to provide data and
experience helping to achieve the outcomes.
• Projects have the potential to be further developed
Risks:
• Reality of funded pilots might not help us achieve
Outcomes;
• Capacities in peripheral regions limited
M1:
Designscapes
funding
instrument
M2:
Performance
and efficiency

10. Global problems as manifested at
urban level improve.
Design enabled Innovation
creates economic and social
value
The urban environment presents a specific
manifestation of wicked problems and is at a scale
where you can develop solutions.
Design capabilities are not diffused in the urban
context (ie many different actors in the urban
environment know about DEI and can use it) and
cannot capture the potential of urban contexts to
generate innovation.
Pilot teams advance these solutions alleviate wicked
problems in the urban space
Applicants create their own solutions specific to their
urban environments.
Designscapes funds 100 pilots: 50 feasibility studies, 40
prototypes, 10 scalability proofed projects
Designscapes develops 3 rounds of an open call call,
addressing everyone in Europe and stressing user co-
creation and social inclusion
DEI creates sustainable
innovation capability at urban
level.

11. 3 rounds of an open call
• Applicants to focus on:
• Urban dimension conducive to
use of design
• Using design to find more
effective responses to big
challenges
• DEI projects that can be
transferred to other contexts
(including to less innovative
contexts)
• Feasibility studies, prototypes,
scalability pilots
• Relatively modest funding:
• between EUR 5,000 (feasibility
studies) and
• EUR 25,000 (prototypes and
scalability pilots)
“We liked the approach in terms of design and
innovation and urban, and we think it was also
made for creative people (not super scientific and
bureaucratic as some other calls), we liked the
process approach where the outcome was not the
main focus – that fitted well in a prototytpe call; the
division of the call was intriguing – to have the
option to proceed further, feels understood for a
creative process” [project interview, round 2].
“When it comes to Swedish
funding, any calls so clearly
focused on using design for
urban sustainability is unique.
That is what we’re trying to do,
design something that fits into
what society today, (…).” [project
interview, round 3]

12. 99 pilots funded
0 10 20 30 40
South
West
North
East
Location of Designscapes pilots • 50 % of successful pilots would not
have gone ahead without
Designscapes funding at feasibility
stage
• 3% of successful applicants would
not have gone ahead without
Designscapes funding at prototype
stage
11 projects received funding in two rounds
3 projects received funding in all three rounds
Majority of survey respondents not applying to round 3 had
continued to develop their project in some form (as designed, small
changes, large changes)

13. Solutions to urban problems
46%
65%
40%
37%
62%
59%
0% 20% 40% 60% 80%
Climate Change
Social Exclusion
Economic Crisis
Crisis of Democracy
Low Quality of Life
Crisis of Values
Average priority areas across all
stages by success
Successful
•Lived experience
•Users as content experts
•Professional experience
/ existing product
•Shared knowledge
between users and project
leads
•Users as providers of
information
•Research:
•Users as providers of
information
36 % of successful applicants at
feasibility stage were looking for
funding for a new idea
Self-reported perceptions that
funded design enabled
innovations unique in their
country or in their focus (e.g.
social dimension)

14. Mechanism M1: Designscapes funding instrument. Designscapes provides a flexible funding
instrument to which applicants respond by applying design thinking and tools to generating and
developing new and innovative ideas to tackle problems linked to their urban environment.

15. Using design methods
40%
50%
60%
80%
90%
90%
66%
78%
56%
90%
63%
68%
78%
68%
46%
84%
74%
74%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Design methods to generate ideas
Prototyping methods
Usability Evaluation methods
Participatory design
Use of design thinking methodologies
Participatory process to validate ideas
Design methods proposed by successful projects by
stage
Stage 1 Stage 2 Stage 3
2578 citizens
54 public
administrations
103 not for profits
47 for profit
enterprises

16. • 81% ‘quite likely’ they will
take prototype to market
(round 2)
• 30% greater readiness to
introduce innovation to
new markets (round 3)
Benefits of co-creation
using design thinking
Bringing different people and perspectives
together enriches final output and
outcomes
Working through a problem with
stakeholders so the solution works in a
given context
Understanding user needs and market
interest in the innovation
Refining the innovation and creating
ownership by users
“It wasn’t, ‘Take this project and see what
you think of it and how you would
improve it’; it was, ‘Here you have the
square – what would you do to it, what
would you remove, what would you keep,
and what would you add?’ In this sense, it
was more enriching.” [citizen interview,
scalability pilot]

17. Value of Designscapes grant
“It has lowered our thresholds to incorporating
design methodology in our work significantly, so
we will now use it much more frequent and in a
more self-confident and relaxed manner”
[Case study interview, round 2]
We used design tools that allowed us to better
understand the calibration of the project according to
the different types of people/companies that might be
interested in using our technology and led us to
diversify the technology we developed according to
the needs of those who might use it. [survey, round 3]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Stimulating new ways of working
Ability to generate evidence on impact of the
innovation
Using design thinking and tools
To what extent has particiaption in Designscapes
enabled you to:
round 3 round 2

18. Sustainable innovation capacity –
behavioural additionality effects
Professional network gains
Round 1: 80%*
Round 2: 55%
Round 3:77%
* Statistically significant difference between funded and non-funded pilots
0 5 10 15 20
Other Desigsncapes project…
Potential funders
Potential customers
Experts in design thinking…
Higher education institutions
Public administrations
Maintaining links after end of
Designscapes funding
“It has lowered our thresholds to
incorporating design methodology in our
work significantly, so we will now use it
much more frequent and in a more self-
confident and relaxed manner”
[Case study interview, round 2]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Stimulating new ways of working
Ability to generate evidence on impact of the
innovation
Using design thinking and tools
To what extent has particiaption in Designscapes
enabled you to:
round 3 round 2

19. Economic value – business impact
Additional funding generated
by successful pilots
• 21 % of round 2 pilots:
• EUR 74,428 (average)
• higher than unsuccessful
applicants
• 23% of round 2 pilots
• EUR 16,170 (average)
• lower than unsuccessful
round 3 pilots (though fewer
of these were able to secure
this additional funding in
the first place)

20. Social impact
• Projects more confident
in achieving social
inclusion and quality of
life impacts.
• Less confident in
economic and climate
change impacts.
0
1
2
3
4
5
6
7
8
9
10
Climate impacts Social inclusion
impacts
Economic
impacts
Quality of life
impacts
Crisis of
democracy
impacts
Crisis of values
impacts
Not at all To a small extent To a moderate extent To a great extent Completely

21. Mechanism M2: Performance and efficiency.
By locating their pilots in the urban context and using design
thinking and tools, end-users creativity is mobilised and
combined with pilot teams’ technical and professional
expertise. This leads to new solutions or the adaptation of
existing ones which meet user needs better and hence get
adopted. This generates business and social value.

22. Towards a common impact methodology

23. The Co-Creation Call Theory of Change
user needs
responsiveness
design thinking
increased
performance &
efficiency
competitiveness
Presenting problem: Europe needs to become more competitive – by identifying and capitalising on
untapped sources of growth and employment
User involvement
central to
innovation
process
Design thinking
and co-creation
encourages user
involvement
Leads to service &
business model
innovation, then
increased
performance &
efficiency
Leads to
increased
competitiveness
of public, private
and third sectors

24. Application areas and evaluation
questions addressed by CIM
Application area Evaluation question Examples from the Designscapes
evaluation
1. User benefits and
business impacts
What user benefits and
business impacts are
associated with design-
related policies and
programmes?
Calculating the economic and social benefits
of the funding provided to projects by
Designscapes using Cost Consequence
Analysis (CCA)
2.Relationship between
design and innovation
What contribution does
design thinking make to
innovation?
Using Regression Analysis to predict the
likely effects of the application of design
thinking methods and tools to increase
innovation
3.Value-creating networks
and efficiency and
competition
How does DEI support
value-creating networks and
in what ways do they
increase efficiency and
competitiveness?
The contribution of the Designscapes
financial instrument to creating value-driven
networks and their impact on the efficiency
and competitiveness of funded projects

25. CIM Framework: Theory of Change

26. Why use Theory of Change in
DEI Evaluation?
Not only does Theory of Change
provide a holistic view of the vision
of the intervention and the
interconnections that make up this
vision, it situates that vision within
the wider social context.
This helps to better articulate the
contribution of the intervention to the
broader societal impacts of DEI.

27. Applicable at different scales
Scale Presenting Problem Activities Outputs Outcomes Impacts
Organisational (single
actor)
Small atelier making
customised designer-wear is
experiencing mounting
production costs and
decreasing profit margins
Owners join EU-wide textile
and clothing innovation
network
Installation of laser cutting
and 3D printing equipment
Production time per output
unit decreased by 15%
Profit increased by 12% after
1 year
Project (partnership) Across Europe, cities and
towns are losing their cultural
heritage artefacts.
10 partners in 6 EU countries
involved in using design
thinking to explore factors
affecting cultural heritage
loss
Cultural Academy set up
And delivers 10 international
workshops and 10 online
webinars, attended by 650
people in total
Over 75% of educational
programme participants
report increased awareness
of cultural heritage
Over 60% of city
administrations report
increased attendance at
heritage events in their cities
Community/regional Urban spaces in the region
have untapped potential to
develop innovative and
sustainable farming methods
Research on feasibility of
urban ‘vertical farms’,
including use of AI and
Industry 4.0 tools
Vertical Farm Prototype in 8
cities across the region
5,000 customers purchase
produce from vertical farms in
6 month trial period
5% of derelict urban
properties rehabilitated to
farming use in region
National Rates of unemployment of low
skilled young people are
increasing across the country
in all cities.
Develop a national jobs
database co-ordinated
nationwide
6,000 employment service
staff participate in Training
programme for employment
service staff
55% of users placed in jobs
after 1 year of programme
operation
Reduction in youth
unemployed rate of 15%
nationally
Trans-national The EU needs to become
more competitive by
capitalising untapped sources
of growth through co-creation
EC launches 5 year
programme to support
design-enabled innovation
through funding pilot projects
Design-enabled Innovation
funding instrument (DEIFI)
launched through ESF
Participating organisations
introduce innovations into
their practice
Behavioural additionality
analysis shows net increase
in efficiency of 12% across
EU

28. And for different actors
Actor Presenting
Problem
Activities Outputs Outcomes Impacts
SME A small specialist
furniture maker is
experiencing
falling demand
due to high costs
and changing
consumer
attitudes
Apply design
thinking to ‘outside
the box’ business
re-structuring
strategies
Prototype Circular
Economy furniture
service for
landlords and
tenants
Reduction in
volume of furniture
thrown away
Contribution to
reduction in use of
landfill sites
Public
Administratio
n
Public
administrations
have a duty to
support lonely but
they lack
resources
Design platform
and App to help
neighbours
connect with each
other
Anti-isolation
network set up
involving 80 public
administrations
Over 2,000 people
use the platform
and App in 6 month
period
Self-reported
loneliness and
isolation levels
have reduced by
23%

29. ToC
• Design Theory of Change
• Fit to scale and audience
Plan
• Implementation Plan
• Purposes, resources, actions
Measure
• Design indicators
• Select and apply MoV
Interpret
• Analyse and integrate results
• Triangulation, Mechanisms,
Contribution analysis
6
Presenting
problem
Inputsand
activities
Outputs Outcomes Impacts
STEP 7: What are the assumptionsthat link each step?
The things you do The changes you wish to see
Step 1 Step 2
Steps 4-5 Step 3
STEP 6: IFwe invest these resourcesand deliver these activities, THEN these outputs
will be produced, which will trigger these outcomesand THEN lead to our impact.
SMART SPICED
Methodology
Methods
Tools
2
BIPOLAR
DISORDER
REDUCED MOOD
SWINGS
GSK3B
BDNF
SEROTONIN
LITHIUMONLY
WORKS FOR ONE
THIRD OF PATIENTS
WE DON’T REALLY
KNOWWHY OR HOW

30. Application area 1: User benefits
and business impacts
SROI = (social impact
value – initial
investment amount) ÷
initial investment
amount x 100%
Stage What this involves
1. Mapping Produce an Impacts Map showing
the expected impacts of the changes
realised by the intervention
2. Measurable
financial
consequences
Select and quantify the impacts and
outcomes that have measurable
financial consequences attached
3. Non-financial
consequences
Select and evaluate the impacts and
outcomes that have measurable non-
financial consequences attached
4. Non-recurrent
costs
Identify and quantify non-recurrent
costs
5. Analysis Compare cost consequences of
alternatives and review results

31. Application area 2: Relationship
between design and innovation
 Providing an explanation of the causal
relationships between variables and how they
lead to observed outcomes
 Providing a measure of the relative strength of
the contribution each variable of interest makes
to the observed outcomes
 Making predictions about the likely effects on
outcomes of interest if the values of the
contributing variables are changed
 Producing simulation models based on ‘what-if’
scenarios – for example if you increase the level
of investment design-enabled innovation, what
is the likely effect on public sector
competitiveness?

32. Application Area 3: Value-creating
networks and their contribution to
efficiency and competition
• Social Network Analysis (SNA) maps and
evaluates the effects of interconnections
between actors in a network, including more
complex networks that make up eco-
systems
• Main focus is on the relationships between
network members – but also maps the
characteristics of the members
• Captures longitudinal evolution of
network/ecosystem over time

33. Questions and discussion