1. A Case Study on the Use of
Developmental Evaluation
for Navigating Uncertainty
during Social Innovation
Chi Yan Lam, MEd
AEA 2012 @chiyanlam
October 25, 2012
Assessment and Evaluation Group, Queen’s University
Slides available now at www.chiyanlam.com

2. “The significant
problems we have
cannot be solved at the
same level of thinking
with which we created
them.”
http://yareah.com/wp-content/uploads/2012/04/einstein.jpg

3. Developmental Evaluation
in 1994
• collaborative, long-term
partnership
• purpose: program
development
• observation: clients who
eschew clear,specific,
measurable goals

4. Developmental Evaluation
in 2011
• takes on a responsive,
collaborative, adaptive
orientation to evaluation
• complexity concepts
• systems thinking
• social innovation

5. Developmental Evaluation (Patton, 1994, 2011)
DE supports innovation development Evaluator works collaboratively with
to guide adaptation to emergent and social innovators to conceptualize,
dynamic realities in complex design, and test new approaches in
environments long-term, ongoing process of
adaptation, intentional change and
DE brings to innovation and adaptation development.
the processes of:
Primary functions of evaluator:
• asking evaluative questions
• elucidate the innovation and
• applying evaluation logic adaptation processes
• gathering and reporting eval data • track their implications and results
to inform support project/
program/product, and/or • facilitate ongoing, real-time data-
organizational development in real based decision-making in the
time. Thus, feedback is rapid. developmental process.

6. Developmental Evaluation (Patton, 1994, 2011)
DE supports innovation development to Evaluator works collaboratively with
guide adaptation to emergent and social innovators to conceptualize,
dynamic realities in complex design, and test new approaches in
environments long-term, ongoing process of
adaptation, intentional change and
DE brings to innovation and adaptation development.
the processes of:
Primary functions of evaluator:
• asking evaluative questions
• elucidate the innovation and
• applying evaluation logic adaptation processes
• gathering and reporting eval • track their implications and results
data to inform support project/
program/product, and/or • facilitate ongoing, real-time
organizational development in real data-based decision-making
time. Thus, feedback is rapid. in the developmental process.

7. Developmental Evaluation (Patton, 1994, 2011)
Improvement
DE supports innovation development
to guide adaptation to emergent and
dynamic realities in complex
Evaluator works collaboratively with
social innovators to conceptualize,
design, and test new approaches in
environments long-term, ongoing process of
adaptation, intentional change and
DE brings to innovation and adaptation development.
the processes of:
• asking evaluative questions vs Primary functions of evaluator:
• elucidate the innovation and
• applying evaluation logic adaptation processes
• gathering and reporting eval data • track their implications and results
to inform support project/
Development • facilitate ongoing, real-time data-
program/product, and/or
organizational development in real based decision-making in the
time. Thus, feedback is rapid. developmental process. .

8. Developmental
Evaluation
is
reality
testing.

10. • DE: novel, yet-
to-be developed
empirical &
practical basis
• Research on
Evaluation:
• scope and
limitations
• utility &
suitability
diff. context
• Legitimize DE

11. Overview
• Theoretical Overview (only briefly)
• Case Context
• Case Study

12. Research Purpose
to learn about the capacity of developmental
evaluation to support innovation
development.
(from nothing to something)
12

13. Research Questions
1.
To what extent does Assessment Pilot Initiative qualify as a
developmental evaluation?
2.
What contribution does developmental evaluation make to
enable and promote program development?
3.
To what extent does developmental evaluation address the
needs of the developers in ways that inform program development?
4.
What insights, if any, can be drawn from this development about the
roles and the responsibilities of the developmental evaluator?
13

14. Social Innovation
• SI aspire to change and transform social
realities (Westley, Zimmerman, & Patton,
2006)
• generating “novel solutions to social
problems that are more effective, efficient,
sustainable, or just than existing solutions and
for which the value created accrues primarily
to society as a whole rather than private
individuals” (Phills, Deiglmeier, & Miller, 2008)
14

15. Complexity Thinking
Situational Analysis Complexity Concepts
“sensemaking” frameworks
that attunes the evaluators
to certain things
15

16. Simple Complicated Complex C
• predictable
• replicable
•
•
predictable
replicable
• unpredictable
•difficult to replicate
h
•
a
• known known • unknown
• many variables/parts • many interacting
• causal if-then working in tandem in variables/parts
models
•
sequence
requires expertise/training • systems thinking? o
• complex dynamics?
• causal if-then models (Westley, Zimmerman, Patton, 2008) s

17. http://s3.frank.itlab.us/photo-essays/small/apr_05_2474_plane_moon.jpg

18. Complexity Concepts
• understanding dynamical behaviour of
systems
• description of behaviour over time
• metaphors for describing change
• how things change
• NOT predictive, not explanatory
• (existence of some underlying principles; rules-
driven behaviour)
18

19. Complexity Concepts
• Nonlinearity (butterfly flaps its wings, black swan); cause and
effect
• Emergence: new behaviour emerge from interaction... can’t really
predetermine indicators
• Adaptation: systems respond and adapt to each other, to
environments
• Uncertainty: processes and outcomes are unpredictable,
uncontrollable, and unknowable in advance.
• Dynamical: interactions within, between, among subsystems change
in an unpredictable way.
• Co-evolution: change in response to adaptation. (growing old
together)
19

20. Systems Thinking
• Pays attention to the influences and
relationships between systems in reference to
the whole
• a system is a dynamic, complex, structured
functional unit
• there is flow and exchanges between systems
• systems are situated within a particular
context
20

21. Complex Adaptive Dynamic Systems

22. Practicing DE
• Adapative to context, agile in methods,
responsive to needs
• evaluative thinking - critical thinking
• bricoleur
• “purpose-and-relationship-driven not
[research] method driven”(Patton, 2011, p.
288)

23. Five Uses of DE (Patton, 2011, p. 194)
Five Purposes and Uses
1. Ongoing development in adapting program, strategy,
policy, etc.
2. Adapting effective principles to a local context
3. Developing a rapid response
4. Preformative development of a potentially broad-
impact, scalable innovation
5. Major systems change and cross-scale developmental
evaluation
23

24. Method & Methodology
• Questions drive method (Greene, 2007; Teddlie and Tashakkori,
2009)
• Qualitative Case Study
• understanding the intricacies into the phenomenon and
the context
• Case is a “specific, unique, bounded system” (Stake,
2005, p. 436).
• Understanding the system’s activity, and its function and
interactions.
• Qualitative research to describe, understand, and infer
meaning.
24

25. Data Sources
• Three pillars of data
1. Program development records
2. Development Artifacts
3. Interviews with clients on the significance
of various DE episodes
25

26. Data Analysis
1. Reconstructing evidentiary base
2. Identifying developmental episodes
3. Coding for developmental moments
4. Time-series analysis
26

27. !
27

28. Assessment Pilot Initiative
• Describes the innovative efforts of a team of 3
teacher educators promoting contemporary
notions of classroom assessment
• Teaching and Learning Constraints ($, time, space)
• Interested in integrating Social Media into Teacher
Education (classroom assessment)
• The thinking was that assessment learning
requires learners to actively engage with peers
and challenge their own experiences and
conceptions of assessment.
28

29. 29

30. Book-ending: Concluding
Conditions
• 22 teacher candidates participated in a hybrid, blended
learning pilot. They tweeted about their own experiences
around trying to put into practice contemporary notions of
assessment
• Guided by the script: “Think Tweet Share” - grounded in e-
learning and learning theories
• Developmental evaluation guided this exploration, between
the instructors, evaluator, and teacher candidates as a
collective in this participatory learning experience.
• DE became integrated; Program became agile and
responsive by design
30

31. 31

32. How the innovation
came to be...

33. Key Developmental Episodes
• Ep 1: Evolving understanding in using social media
for professional learning.
• Ep 2: Explicating values through Appreciative
Inquiry for program development.
• Ep 3: Enhancing collaboration through structured
communication
• Ep 4: Program development through the use of
evaluative data
Again, you can't connect the dots looking forward; you can only connect them looking backwards. - Steve Jobs
33

34. (Wicked) Uncertainty
• uncertain about how to proceed
• uncertain about in what direction to proceed (given many choices)
• uncertain how teacher candidates would respond to the intervention
• the more questions we answered , the more questions we raised.
• Typical of DE:
• Clear, Measurable, and Specific Outcomes
• Use of planning frameworks.
• Traditional evaluation cycles wouldn’t work.
34

35. How the innovation came to
be...
• Reframing what constituted “data”
• not intentional, but an adaptive response
• informational needs concerning
development; collected, analyzed,
interpreted
• relevant theories, concepts, ideas;
introduced to catalyze thinking. Led to
learning and un-learning.
35

36. Major Findings
RQ1: To what extent does API qualify as a
developmental evaluation?
1. Preformative development of a potentially
broad-impact, scalable innovation
2. Patton: Did something get developed?
✗
(Improvement vs development vs innovation)
✔ ✔
36

37. RQ2: What contribution does DE make to
enable and promote program development?
1. Lent a data-informed process to innovation
2. Implication: responsiveness
• program-in-action became adaptive to
the emergent needs of users
3. Consequence: resolving uncertainty
37

38. RQ3: To what extent does DE address the
needs of developers in ways that inform
program development?
1. Definition - defining the “problem”
2. Delineation - narrowing down the problem space
3. Collaboration - collaboration processes; drawing on
collective strength and contributions
4. Prototyping - integration and synthesis of ideas to ready
a program for implementation
5. Illumination - iterative learning and adaptive development
6. Evaluation - formal evaluation processes to reality-test
38

39. Implications to
Evaluation
• One of the first documented case study into
developmental evaluation
• Contributions into understanding, analyzing
and reporting development as a process
• Delineating the kinds of roles and
responsibilities that promote development
• The notion of design emerges from this
study
39

40. Implication to Theory

41. • Program as co-created
• Attending to the “theory” of the program
• DE as a way to drive the innovating process
• Six foci of development
• Designing programs?

42. Design and
Design Thinking
42

43. Design+Design Thinking
“Design is the systematic exploration into the complexity of options (in
program values, assumptions, output, impact, and technologies) and
decision-making processes that results in purposeful decisions about the
features and components of a program-in-development that is informed by
the best conception of the complexity surrounding a social need.
Design is dependent on the existence and validity of highly situated and
contextualized knowledge about the realities of stakeholders at a site of
innovation. The design process fits potential technologies, ideas, and
concepts to reconfigure the social realities. This results in the emergence of
a program that is adaptive and responsive to the needs of program users.”
(Lam, 2011, p. 137-138)
43

44. Implications to
Evaluation Practice
1. Manager
2. Facilitator of learning
3. Evaluator
4. Innovation thinker
44

45. Limitations
• Contextually bound, so not generalizable
• but it does add knowledge to the field
• Data of the study is only as good as the data collected from
the evaluation
• better if I had captured the program-in-action
• Analysis of the outcome of API could help strength the case
study
• but not necessary to achieving the research foci
• Cross-case analysis would be a better method for generating
understanding.
45

46. Thank You!
Let’s Connect!
@chiyanlam
chi.lam@QueensU.ca