There is increasing recognition that the problems we and our communities face are complex, dynamic and not easily solved. Our solutions must be as equally complex and dynamic and social innovation (SI) has emerged as a promising method that can guide the development of these complex solutions. A key practice in social innovation is prototyping and the ability to learn quickly in order to make rapid, yet informed, decisions about the ongoing trajectory of our work. This workshop will introduce participants to the principles and functions of prototyping in a SI initiative and the Developmental Evaluation approach that guide learning, decision making and iteration. Leading Boldly Network: Calgary’s Network for Collaborative Social Innovation

1. Using Developmental Evaluation to
Support Prototyping:
A Workshop
Leading Boldly Network:
Calgary’s Network for Collaborative
Social Innovation

2. Workshop Agenda
• Getting Grounded:
– The Collaborative Social Innovation Process
– Your own experience with or questions about
prototyping
• What is Prototyping & How Is It Unique?
• What is Developmental Evaluation (DE)?
– Using DE to Support Prototyping

3. Collaborative Social Innovation is…
… the process of developing social innovations
that are both social in their ends and
collaborative in their means. Specifically,
collaborative social innovations include new
ideas (services, models and resource flows)
that simultaneously address complex social
issues and create new social relationships or
collaborations. In other words they are
innovations that are both good for society and
enhance society’s capacity to act.
Adapted from the Young Foundation

4. The LBN Process of Collaborative Social Innovation*
Secure resources
1. Co-Initiate – Set the right conditions and culture
2. Co-Sensing – Frame the issue or design question
 Secure resources
3. Co-Presence– Generate ideas
4. Prepare for Prototyping – Planning
5. Co-create / Prototype – Experiment to test ideas
 If prototype results prove to be promising, secure resources
6. Co-Evolve - Implement
7. Scale Out – Share innovations with a broader set of stakeholders
8. Scale Up – Systems Change
* This is a working model

5. What is Prototyping?
Prototype literally means first impression (proto
= first, typos = impression)
• A series of small-scale experiments, in
simulated or real environments, that test a
‘first impression’ or ‘hunch’ regarding how to
positively influence change within a defined
“problem space”

6. What is Prototyping?
• A core activity within the
collaborative social innovation
process
• Action– and learning–oriented
• Undertaken with the purpose
of designing, testing and
rapidly iterating new and
unverified processes, services
or programs in order to
discover how they truly
influence the complex
problem you are trying to
address
• A practice for situations that
are non-routine and
unpredictable, and therefore
different than what is required
for ongoing operations
• Best done collaboratively with
diverse stakeholders

7. What is Prototyping & How is it
Unique?
4 Prototyping Principles:
1. It is used at the beginning, or developmental
phase, of a new process, service or program,
where the way to the desired outcome is
unknown
2. It anticipatesfailure. Indeed a key objective of
prototyping is to create the conditions to be
able to ‘fail safe’, namely endure failure that is
survivable (hence small scale) and provides
learning to inform the next iteration.

8. What is Prototyping & How is it
Unique?
4 Prototyping Principles
3. It embraces rapid iteration informed by
feedback from the system (stakeholders, end-
users, etc.) it is trying to change
4. It includes reflection-in-action, the
intentional recording and documenting of
what is being learned as projects are
implemented

9. What is Prototyping & How is it
Unique?
The focus is on learning:
• Using evidence to move from “wild
guesses” to “reliable forecasts”
–The focus is not on results, but on
the overall trajectory of the
initiative

10. What is Prototyping & How is it
Unique?
Moving from Wild Guesses to Reliable Forecasts
Wild
guesses
Informed
estimates
Reliable
forecasts
Prediction
LearningTime

11. How is Prototyping a Unique Practice?
This practice enables us to:
• “Think with our hands” as a way to investigate the
merit of our proposed solution;
• collect user feedback; and
• observe how our actions influence (for better or
worse) change within the domain we are seeking to
transform.
“The key to dealing with complexity is to focus on
having good conversations about assumptions.”

12. Why Is Prototyping an Important
Practice?
• In a time of tight budgets, prototyping can be
used to test an unverified process, service or
program before considerable resources are
invested into a large-scale initiative.
– It helps to expose flaws and limitations in an
approach before significant investment has been
made
• Can help leaders/Boards manage the risks
involved with trying something different or new
• Can help us develop better products and
services

13. How to Prototype
1. Know when a prototype
project is the right course
of action
– Developmental phase /
uncertain outcomes
2. Develop a plan
– Write down what you plan
to do, what you expect to
happen and why
(Hypothesis of Record)
– “In an innovative initiative
the plan is a hypothesis”
3. Develop a clear learning
agenda (Developmental
Evaluation)
– Compare results to explicit
set of assumptions
4. Execute the Plan
– Analyze differences
between what you thought
would happen and what
actually happened
5. Based on what you
learned, revise the plan

14. How to Prototype

15. Small Group Conversation
What stood out for you?
What is becoming clearer?
What questions are you carrying?

16. Developmental evaluation is
a small but demanding niche
in the world of evaluation.

17. Five Niches
Ongoing Development
Model Replication
Crisis Situations
Pre-Formative (Innovation)
Cross-Scale Complexity

18. DE: 7 Things You Need to Know
1.
Essence
2.
Purpose
3.
Evaluator
4.
Methods
5.
Steps
6.
Reports
7.
Accountability

19. 1. The Essence of DE
• Developmental evaluation isn’t some particular methods of
recipe-like steps to follow. It doesn’t offer a template of
standard questions. It’s a mindset of inquiry into how to
bring data to bear on what’s unfolding so as to guide and
develop the unfolding. What that means and the timing of
the inquiry will depend on the situation, context, people
involved, and the fundamental principle of doing what
makes sense for program development (Patton 2010: pp.75-
6).

20. Development
is about creative
thinking.
Evaluation
is about critical
thinking.
Developmental
Evaluation
holds these
In balance.
The Essence of DE

21. 2. Purpose
• Developmental
evaluation focuses on
developmental
questions: What’s being
developed? How is
what’s being developed
and what’s emerging to
be judged? Given
what’s been developed
so far and what has
emerged, what’s next?

22. TRADITIONAL EVALUATIONS DEVELOPMENTAL EVALUATIONS
Render definitive judgments of success or
failure.
Provide feedback, generate learnings, support
direction or affirm changes in direction.
Measure success against pre-determined goals. Develop new measures and monitoring
mechanisms as goals emerge and evolve.
Position the evaluator outside to assure
independence and objectivity.
Position evaluation as an internal, team function
integrated into action and ongoing interpretive
processes.
Design the evaluation based on linear cause-
effect logic models.
Design the evaluation to capture system
dynamics, interdependencies, and emergent
interconnections.
Aim to produce generalizable findings across
time and space.
Aim to produce context-specific understandings
that inform ongoing innovation.
Accountability focused on and directed to
eternal authorities and funders.
Accountability centered on the innovators’ deep
sense of fundamental values and commitments.
Accountability to control and locate blame for
failures.
Learning to respond to lack of control and stay in
touch with what’s unfolding and thereby
respond to strategically.
Evaluator controls the evaluation and
determines the design based on the evaluator’s
perspective on what is important.
Evaluator collaborates in the change effort to
design a process that matches philosophically
and organizationally.
Evaluation engenders fear of failure. Evaluation supports hunger for learning.

23. 3.1 The Evaluator
• The developmental evaluator inquires into
developments, tracks developments, facilitates
interpretation of developments and their
significance, and engages with innovators, change
agents, program staff, participants in the process,
and funders around making judgments about what is
being developed, how it is being developed, the
consequences and impacts of what has been
developed, and the next stages of development
(Patton 2010, p. 227).

24. 3.2 Evaluator Roles
Surfacing Issues
Framing Concepts
Testing Quick Iterations
Tracking Developments

25. 3.2.1 Surfacing Issues
Raise critical questions,
tensions, debates that
emerge as part of the
developmental process.

26. 3.2.2 Framing Concepts
Flush out and flesh out
problem definitions,
lenses, frameworks and
‘working models’
underlying the
conception and design
of the intervention.

27. 3.2.3 Testing Quick Iterations
Surface –and interpret,
draw conclusions and
judge -- real time data
through techniques
such as rapid
reconnaissance,
secondary evidence,
modeling, outcome
mapping, simulations,
etc.

28. 3.2.4 Tracking Developments
Documenting concrete
developments – e.g.
learning, new
models, design
features, forks in the
road – in the evolution
of a group’s work.

29. 3.3 The Evaluator-User Relationship
Traditional Evaluation
• Positioned outside the
action to assure
independence and
objectivity.
Developmental Evaluation
• Positioned as team member
integrated into the action
and ongoing process of
gathering and interpreting
data, framing issues,
surfacing and testing model
developments. May be
situated externally or
internally.

30. 3.4 Competencies
Traditional Evaluator
Competencies
 Reflective Practice.
 Technical Practice
 Situational Practice.
 Management Practice.
 Interpersonal Practice.
Extra Competencies
for DE
 Comfort with adaptive
processes
 High tolerance for ambiguity
 Can react/adapt quickly
 Effective communication with
hyperactive, short-attention-
span, action-oriented
innovators
 Larger tool kit of evaluative
methods

31. 4. Methods
• [I]t’s worth emphasizing that no definitive list of developmental
evaluation inquiry approaches can or should be constructed.
Developmental evaluation creatively adapts whatever
approaches and methods fit the complexities of the situation
and are responsive, appropriate, and credible to social
innovators in opening up new understandings and guiding
further development.
• In being creative, the developmental evaluator is also practical
and pragmatic, doing the best job possible within available
resources and other constraints. Constraints always exist and do
what constraints do -- constrain. Our ability to think of
alternatives is limited. Resources are always limited. Time is of
the essence. We do what we can. Part of what we can do is
adapt other inquiry traditions to the purposes of developmental
evaluation.

32. Methods continued
Key Features
• Contingency
• Real Time
• Flexible
Adaptive/Evolutionary
• Evaluation methods
need to be continually
adapted to reflect the
changes in the
emerging innovation
and the learnings and
questions of the
innovators.

33. 5. Steps

34. 6. Reports
• Dynamic complexities don’t slow down or wait for evaluators
to write their reports, get them carefully edited, and then
approved by higher authorities. Any method can be used but
will have to be adapted to the necessities of speed, real-time
reporting and just-in-time, in-the-moment decision-making.
• That is a major reason the developmental evaluator is part of
the innovation team, to be present in real time as issues arise
and decisions have to be made […] Contrary to the usual
practice in evaluation of fixed designs that are implemented
as planned, developmental evaluation designs can change as
the innovation unfolds and changes (Patton 2010, p. 335-6).

35. Emerging Practices on Reports
Ongoing Real Time Feedback
• A variety of formats that
capture key evaluation
questions, results and
learnings, and new
developments (e.g. memos,
PPT, visual diagrams, etc.) in
a way that is fits the pattern
of information flow and
decision-makers of the
evaluation users?
Periodic “Developmental
Accounts”
• A record of the major
developments that have
emerged in a
developmental process so
far (interim accounts)
and/or at the end of
significant developmental
period (e.g. a shift into
formative evaluation).

36. The Tracking the Evolution
Of the Intervention

37. 7. Accountability
To Whom
• People responsible
for developing
and/or adapting an
intervention.
For What
• Developing a better
understanding of the
issue being addressed,
leverage points for
change, and developing
an intervention.
• This may – or may not –
develop into a workable,
stable or ultimately
successful intervention.
For How
• A rigorous, data-
based, user-friendly
and responsive,
process of reality
testing that balances
creative and critical
thinking.
• Guided by standard
evaluation principles
and ethics.

38. Tamarack Interview with
Michael Quinn Patton on
DE
http://tamarackcommunity.
ca/g3s61_CC4I5.html
Utilization-Focused
Evaluation: 4th Ed.
Michael Quinn Patton
The DE Primer.
Jamie Gamble.
http://www.imprintinc.ca/
Getting to Maybe.
Frances Westley
Brenda Zimmerman
Michael Quinn Patton
Resources
DE 201: Practitioners Guide
Dozois, Langlois,
Blanchet-Cohen
JW McConnell Site
Developmental Evaluation
Michael Quinn Patton

39. Extras for Prototyping

40. types of prototyping
•exploratory prototyping – testing the demand, feasibility
& viability of an idea, quickly and inexpensively.
•developmental prototyping – testing the
demand, feasibility & viability of an idea more detail, to see
if its works in practice.
•slow prototyping – a concurrent focus on developing an
idea or practice AND testing the capacity of the group to
carry out the idea.
•fast prototyping – an exclusive focus on testing an idea
or practice.

41. fast prototyping
slow prototyping
exploratory
prototyping
developmental
prototyping
testing a collaborative funding
model (simulation)
exploring a new mission and
design for employment program
(scenarios)
testing the efficacy of different
campaign messages before roll out
(random controlled trials)
testing the buy-in, viability
feasibility of a funding
strategy (role playing)
developing & adapting practices
for inter-agency collaboration
(after action review)

42. Questions
• What questions – if any – emerge for you about DE?
• What is alive for you about DE might contribute to
your work?

43. Key Resources
• Nesta (UK)
• The Other Side of Innovation: Solving the
Execution Challenge
• The Open Book of Social Innovation
• The Austin Centre for Design
• IDEO
• Human Centred Design