Introducing the AI ShillText Generator A New Era for Cryptocurrency Marketing...
Ces 2013 - Doing Developmental Evaluation at the System Level
1. +Doing Developmental
Evaluation at the
System Level:
Lessons learned from
an addiction system
change initiative
Tuesday, June 11th, 2013
Jenna van Draanen, Kaileah McKellar, Sanjeev Sridharan
2. +
Agenda
An overview of the DTFP Evaluation
DTFP Evaluation Methods
DTFP Evaluation Results
Developmental Evaluation at the System Level
Lessons learned
Lingering questions for the field of evaluation
3. +
Drug Treatment Funding Program
Addiction treatment system change
Health Canada-funded project, delivered through the Ontario
Ministry of Health
11 system change projects (3 streams)
Multiple project leads and organizations
5 year timeline compressed to 18 months
4. +
DTFP
Funding
Selection of
Projects &
Organization
s
Project Implementation
Testing &
validatio
n
Development
of Tools and
resources
Development of
Inter-organizational
Networks
System-level evaluation
& Evaluation capacity
building
Uptake
&Spread
System Change
and practice
improvement
• Increased access
and use of
evidence
• Enhanced
Collaborative
Problem-Solving
Capacity
• Enhanced
Evaluative Culture
Planning for
sustaining
networks
Collaboration & Co-Development
Knowledge Exchange
Sphere of Direct
Control
Collaborative
Activities
Evaluation
Activities
4
Project-level
activities
Sphere of Direct
Influence
Sphere of Indirect
Influence
5. +
Developmental Evaluation of DTFP
Incomplete knowledge
1) Spread processes within the system need to be developed
2) Relationships between different stakeholder groups need to be
strengthened.
6. +
Developmental Evaluation of DTFP
Purpose
To bring evaluative thinking to DTFP and to help DTFP to evolve
and create change
Multiple and flexible methods
Key-informant interviews, case studies, surveys, capacity-building
workshops, demand mapping, observation, videography,
document review, realist synthesis, and timeline of impact
exercises
Heavy engagement
What needs to be developed?
7. +
Results
Enhancements seen in evaluation and performance
measurement, evidence-informed practice, knowledge
exchange projects
At the project level
Case studies giving insight into the system performance
8. +
Results Themes
Define the system
Pay attention to dynamics
Lack of ownership
Tools themselves will not create change
Planning for spread
8
9. +
Developmental Evaluation Challenges
Multiple simultaneous impacts
Projects having limited sphere of direct control
Timeline of impact
Providing feedback in an environment of uncertainty
10. +
Lessons Learned
Focus on learning
Developing
relationships
Rapid feedback
Redundancy in a
good thing
Clarifying the nature of the addiction treatment systemAchieving change in a climate of fragmentationBranding DTFP as Quality Improvement InitiativeChanging the system from an output to outcomes culture Needing to better understand incentives and clarify capacitiesPaying attention to supply and demandNeed to clarify ownership of the projectUncertainty around key partner roles and expectationsReluctance to change roles over timePlanning for use - tools in themselves will not bring about change.Need to embed spread processes into Accountability agreementsUtilize or develop bottom-up models of spreadNeed to clarify the role of various actors in spread efforts Need to fund spread and sustain activities for long enough to see change
How can evaluators assess readiness for developmental evaluation in a system of actors?Comfort with nonlinearity, emergence, unpredictability, and atypical evaluationWhat is the role of evaluation in disrupting systems and encouraging change?How can evaluators foster co-evolution with partners who are not ready for developmental approaches?