Presentation given at #CESToronto2013. Abstract: Developmental evaluation (DE) supports social innovation and program development by guiding program adaptation to emergent and dynamic social realities (Patton, 2011; Preskill & Beer, 2012). To that end, the developmental evaluator is expected to draw on a multitude of skills and to take on multiple roles depending on the situational demands of the DE. Beyond that little guidance is provided in the literature to guide evaluation practice amidst the decidedly complex and turbulent space of social innovation. This presentation explores the contextual demands made of the evaluator and the emergence of the different roles taken on by the developmental evaluator to enable social innovation in a case of DE. The ‘preformative development’ of this social innovation saw the integration of microblogging into teacher education. Analysis revealed four roles assumed by the developmental evaluator when innovating: a) evaluator, b) facilitator of learning, c) project manager, and d) innovation thinker.

1. ChiYan Lam, MEd
CES 2013
Insights on Developmental
Evaluator Roles during
Innovation: A Case Study
@chiyanlam
June 11, 2013
Assessment and Evaluation Group, Queen’s University
Slides available now at www.chiyanlam.com
1Monday, 10 June, 13

2. Acknowledgements
2Monday, 10 June, 13

3. 3Monday, 10 June, 13

4. Impersonal
Hard to Focus
Passive
4Monday, 10 June, 13

5. ✓Learn from peers
✓Reflect on prior
experiences
✓Meaning-making
✓Active construction
of knowledge
5Monday, 10 June, 13

6. Assessment Pilot Initiative
• Barriers to Teaching and Learning: $, time,
space
• Interested in integrating Web technologies
to Teacher Education to open up possibilities
• The thinking was that assessment learning
requires learners to actively engage with
peers and challenge their own
experiences and conceptions of
assessment.
6
6Monday, 10 June, 13

7. Dilemma
• Barriers to Teaching and Learning: $, time, space.
• PRACTICUM = out of sight, out of touch.
• Instructors became interested in integrating Web
technologies to Teacher Education to open up
possibilities
• The thinking was that assessment learning
requires learners to actively engage with peers
and challenge their own experiences and
conceptions of assessment.
7
7Monday, 10 June, 13

8. So what happened...?
• 22 teacher candidates participated in a hybrid, blended
learning pilot. They tweeted about their own
experiences around trying to put into practice
contemporary notions of assessment.
• Guided by the script:“Think Tweet Share”
• Developmental evaluation guided this exploration,
between the instructors, evaluator, and teacher candidates
as a collective in this participatory learning experience.
• DE became integrated; Program became agile and
responsive by design
8
8Monday, 10 June, 13

9. 9
9Monday, 10 June, 13

10. Research Purpose
to learn about the capacity of developmental
evaluation to support innovation
development.
(from nothing to something)
10
10Monday, 10 June, 13

11. Research Question
What insights, if any, can be drawn from this
development about the roles and the
responsibilities of the developmental
evaluator?
11
11Monday, 10 June, 13

12. Developmental Evaluation
in 2011
• takes on a responsive,
collaborative, adaptive
orientation to evaluation
• complexity concepts
• systems thinking
• social innovation
12Monday, 10 June, 13

13. Developmental Evaluation
DE supports innovation development to
guide adaptation to emergent and
dynamic realities in complex
environments
DE brings to innovation and adaptation
the processes of:
• asking evaluative questions
• applying evaluation logic
• gathering and reporting eval
data to inform support project/
program/product, and/or
organizational development in real
time.Thus, feedback is rapid.
Evaluator works collaboratively with
social innovators to conceptualize,
design, and test new approaches in
long-term, ongoing process of
adaptation, intentional change and
development.
Primary functions of evaluator:
• elucidate the innovation and
adaptation processes
• track their implications and results
• facilitate ongoing, real-time
data-based decision-making
in the developmental process.
(Patton, 1994, 2011)
13Monday, 10 June, 13

14. Developmental Evaluation
DE supports innovation development
to guide adaptation to emergent and
dynamic realities in complex
environments
DE brings to innovation and adaptation
the processes of:
• asking evaluative questions
• applying evaluation logic
• gathering and reporting eval data
to inform support project/
program/product, and/or
organizational development in real
time.Thus, feedback is rapid.
Evaluator works collaboratively with
social innovators to conceptualize,
design, and test new approaches in
long-term, ongoing process of
adaptation, intentional change and
development.
Primary functions of evaluator:
• elucidate the innovation and
adaptation processes
• track their implications and results
• facilitate ongoing, real-time data-
based decision-making in the
developmental process.
(Patton, 1994, 2011)
Improvement
vs
Development .
14Monday, 10 June, 13

15. Developmental
Evaluation
is
reality
testing.
15Monday, 10 June, 13

16. Practicing DE
• Adapative to context, agile in methods,
responsive to needs
• evaluative thinking - critical thinking
• bricoleur
• “purpose-and-relationship-driven not
[research] method driven”(Patton, 2011, p.
288)
16Monday, 10 June, 13

17. Five Purposes and Uses
1. Ongoing development in adapting program, strategy,
policy, etc.
2. Adapting effective principles to a local context
3. Developing a rapid response
4. Preformative development of a potentially broad-
impact, scalable innovation
5. Major systems change and cross-scale developmental
evaluation
17
(Patton, 2011, p. 194)
Five Uses of DE
17Monday, 10 June, 13

18. Evaluator Roles
18
Neutral, detached social scientist
Judge
Program Advocate
Consultant
Change agent
Technical expert
Facilitator of organizational inquiry
Support Decision Making
Collaborator
Empower othersText
Auditor
18Monday, 10 June, 13

19. Role Conceptualization
• Depends on...
• Evaluation methods
• Evaluation models
• evaluator relationship with
stakeholders
• activities!? (Skolits, Morrow, & Burr,
2009)
19
19Monday, 10 June, 13

20. Why Roles Matter
• Roles we adopt determine behaviours and
degree of contribution within an eval.
• Has implications for:
• how we conduct evaluation (processes)
• what is considered acceptable behaviour
(practices)
• training
20
20Monday, 10 June, 13

21. DE Roles - Bricoleur
• Patton, 1994, 2011
• Collaborator
• Evaluator
• Support decision-making
• Documentarian
• Elucidate innovation and adaptation...
• .....?
21
21Monday, 10 June, 13

22. DE Roles - DE Primer
• Coach to the organization, and to the
innovation process
• Collaborator
22
22Monday, 10 June, 13

23. DE Roles - DE 201
• Analyst (strategic thinking, patter
recognition)
• Relationship Builder
• Servant Leader
• Facilitator of Learning
23
(Dozois, Langlois & Blanchet-Cohen, 2010)
23Monday, 10 June, 13

24. Method & Methodology
• Qualitative Case Study
• understanding the intricacies into the
phenomenon and the context
• Case is a “specific, unique, bounded
system” (Stake, 2005, p. 436).
• Understanding the system’s activity, and its
function and interactions.
• Qualitative research to describe, understand, and
infer meaning.
24
24Monday, 10 June, 13

25. Data Sources
• Three pillars of data
1. Program development records
2. Development Artifacts
3. Interviews with clients on the significance
of various DE episodes
25
25Monday, 10 June, 13

26. Data Analysis
1. Reconstructing evidentiary base
2. Identifying developmental episodes
3. Coding for developmental moments
4. Time-series analysis
26
26Monday, 10 June, 13

27. 27
!
27Monday, 10 June, 13

28. Findings - DE Roles
during Innovation
1. Manager
2. Facilitator of learning
3. Evaluator
4. Innovation thinker
28
28Monday, 10 June, 13

29. DE Role - Manager
• Manage the project
• resources, personnel
• Manage the progress
• timeline, deadlines,
milestones
• Rally the troop
• keep on task
http://civilengineerlink.com/wp-content/uploads/2011/02/project-managers.jpg
29Monday, 10 June, 13

30. DE Role - Evaluator
• Report on usage
• shortened reporting cycle
• monitor the conversation
and report on substantive
themes
• facilitates data-based
decision-making
• Systematic inquiry - render
information (e.g. via lit review)
to inform program decisions
http://freshspectrum.com/freshspectrum-in-the-wild-12-illustrations/
30Monday, 10 June, 13

31. DE Role - Facilitate
Learning
• Meaning-making of data
• made difficult due to no
prior framework
• had to be appropriate given
context
• To evaluate requires deep
familiarization with a particular
subject --> makes you a quasi-
expert.
• creates opportunity to help
develop capacity in others
31Monday, 10 June, 13

32. DE Role - Innovation
Thinker
• Innovation is like
entrepreneurship. Good
entrepreneur has good
business acumen. Knows
the ‘business’
• Evaluators working in
social innovation needs
to know innovation .
• Process of innovation is
often non-linear.
http://seec.schulich.yorku.ca/ContentResources/LibraryFiles/innovations.jpg
32Monday, 10 June, 13

33. ThankYou!
Let’s Connect!
@chiyanlam
chi.lam@QueensU.ca
33Monday, 10 June, 13