Diálogo (im)probable con Joshua Fisher llevado a cabo en el itdUPM el 30 de mayo de 2019

1. Developmental Evaluation: Tips, Tools,
& Lessons Learned
Diálogo Improbable
30 May 2019
Joshua Fisher

2. Our conversation
• What is Developmental Evaluation?
• Why do we need it?
• What are useful tools?
• What are good practices?

3. Our conversation
What is Developmental Evaluation?

4. What is Developmental Evaluation?
Analytical system of information collection, sense-making, evidence-
based decision support
1. Process monitoring
2. Outcome evaluation
3. Adaptive management
(Davies & Dart 2005)
(Gamble 2008)

5. (Reproduced from Gamble 2008)

6. Our conversation
Why do we need it?

7. What types of problems do we encounter in development and social innovation?
Complex:
Cause & Effect: Understandable
in retrospect, but not repeatable
Complicated:
Cause & Effect: Detectable but not
separate over time and space
Chaotic:
Cause & Effect: Not identifiable
Simple:
Cause & Effect: Repeatable, perceivable
and predictable
(Snowden & Boone 2005)
Characteristics
• Boundaries and limits of problems are clear
• Problems are well-understood by stakeholders
• Problem & Solution behave in predictable ways
• Finite set of solutions

8. (Reproduced from Gamble 2008)
http://www.migration4development.org/docs/logframe.pdf

9. Complex:
Cause & Effect: Understandable
in retrospect, but not repeatable
Complicated:
Cause & Effect: Detectable but not
separate over time and space
Chaotic:
Cause & Effect: Not identifiable
Simple:
Cause & Effect: Repeatable, perceivable
and predictable
(Snowden & Boone 2005)
Characteristics
• Boundaries and limits of problems are unclear
• Problems defined uniquely by each stakeholder
• Dynamics & interactions are unpredictable
• Infinite solution set

10. Development challenges demonstrate BOTH complicated & complex dynamics
(Reproduced from Gamble 2008)(Reproduced from Gamble 2008)

11. Today’s development challenges occur in complex systems
Interconnectivity
Non-linear cause
& effect
Multi-causality
Constant change

12. Complex Systems are multi-scalar
Complexity is replicated at each scale
Interactions across scales
Solutions have to address this fractal pattern
(Fisher & Coleman Forthcoming)

13. Not solved, but re-solved over & over
Poorly defined
Socially complex
Evolving
Instigating
Non-ending
(Balint et al 2011)
Our ‘development challenges’ are actually ‘wicked problems’

14. (Balint et al 2011 – Wicked Environmental Problems)
Collectively define the problem
Discover drivers & dynamics
Discuss & implement
Learn:
• What happens
• What’s next
• What else
• Who else
• Why
’Re-solving’ wicked problems requires deliberative & adaptive intervention
Adapt & iterate

15. Wicked problems require changing the way we think about
relationships
Individual participant/partner in programming
Entrepreneurs, Market linkages, etc.
Network of convening & coordinating partners
Platform members, design team
Partners in the operating ecosystem
Governments, families, social networks, sectors
Convening partners
Investors, capital providers
Beneficiary
Grantee / Implementing body
Secondary / Indirect beneficiaries
For donor, sponsor
’Partnerships’ require that everyone contribute to
monitoring, evaluation, and learning

16. The ‘Blindspots’ of Traditional M&E/Impact Assessment (USAID, 2013)
1) Broad range of action & outcomes in system (non-linear, cross-scale, unintended)
2) Alternative causes and influence from other actors & factors (attribution dilemma)
3) Full-range of processes & pathways that contribute to observed change (multi-causality)
4) Some outcomes are unobservable (non-occurrence, inference)
http://www.migration4development.org/docs/logframe.pdf

17. Performance-oriented Methods
Logical frameworks (equivalent)
• Objectives
• Measured against preset targets
• Quarterly data collection and ex-
post data assessment and
reporting to sponsor
Process-oriented Methods
Observe & reflect on processes
• Allow for emergent
outcomes
• Continuous data collection;
periodic assessment &
discussion with partners
• Situate results analysis &
evaluation in system context
Deliberative Learning Ecosystem

18. Our conversation
What are useful tools?

19. GOPC 2014 – Spaces MERL

20. Long-term development investment to encourage entrepreneurship &
job creation in India, Mozambique, & Peru
Platform approach: partnership with local communities and
implementing partners
Phases of intervention
1. Listening
2. Co-creation
3. Prototyping
4. Acceleration & Deepening Impact
5. Continued Support

21. How can each platform understand their own ‘wicked’ system?
How can we compare across different cases?

22. Developmental Evaluation Approach
Evaluation Question
1) Does participation in an innovation-based development initiative lead to direct (tangible goods, asset access and ownership, income, etc.) and/or
indirect economic benefits (access to services, employability, additive market linkages, etc.) for participating households, communities, and enterprises?
2) Are the prototype experiments developed through the co-creation process in each W4P intervention qualitatively and/or technically similar, and do
they address the issues raised during the ‘listening’ phase?
3) What strategies are employed by innovation networks/platforms to adapt to structural and contextual changes?
4) How does participation in an innovation network/platform affect interrelationships among implementing partners, and what are the costs and
benefits of this approach when compared with more traditional development projects?
5) How does the presence of an innovation-based initiative affect the wider communities and members of the development ecosystem?
6) What additional design considerations are required to more effectively monitor, evaluate, measure and learn from innovation platforms?

23. Dashboards
(Mayo 2016)
Needs /
Opportunities
Matrix
Most Significant
Change
Barriers /
Enablers Matrix
Sentinel
Indicators
Developmental Evaluation Approach

25. Overview of Tools
Most Significant Change
(Davies & Dart 2005)

26. Most Significant Change
Prototype
participant
Prototype
participant
Prototype
participant
Prototype
participant
Prototype
participant
Prototype
participant
Prototype
participant
Prototype
participant
Prototype
participant
W4P Support
Team
W4P Support
Team
W4P Support
Team
W4P Support
Team
W4P Support
Team
W4P Support
Team
W4P
Platform
Member /
Organization
W4P
Platform
Member /
Organization
W4P
Platform
Member /
Organization
W4P Country Platform
W4P Regional Platform W4P Regional Platform
W4P Program (LCBF, 3 Country Teams, Other Partners)
SignificantChangeNarratives
Feedbacksentbackthroughdecisionchain
• Significant
Change
narratives
collected
• Support teams
select narratives
to send up
• Narratives discussed and
responses and lessons
learned documented
• Platform member
organizations select
narratives to send up
• Narratives discussed and
responses and lessons
learned documented
• Country platform selects
narratives to send up
• Country teams and LCBF
select narratives to share
with other platforms

29. May / June ‘19 Oct / Nov ‘19 Feb / Mar ‘20
2019 2020 2021 2022
Develop a deeper understanding over time & track changes

30. Overview of Tools
Sentinel Indicators
Source:
(Mayo 2016)

31. Overview of Tools
Examples
Source:
(Mayo 2016)

32. Our conversation
What are good practices?

33. Multiple methods, multiple scales, with intentional discussions to inform programming
Evaluation Partners need investment/ownership & distance/objectivity
Cost-efficient in terms of ALL partners’ time, energy, economies, incentives, ROI & SROI
Balance between scientific purity/rigor & useful data and information
Partnerships in learning & adaptation rather than transactional programs & evaluations
Good Practices

34. References
• Balint, P.J., R.E. Stewart, A. Desai, and L.C. Walters. 2011. Wicked Environmental Problems: Managing Uncertainty and Conflict. Washington, D.C.: Island Press.
• Davies, R., and J. Dart. (2005) The Most Significant Change (MSC) Technique. Available online at:
https://www.researchgate.net/publication/275409002_The_'Most_Significant_Change'_MSC_Technique_A_Guide_to_Its_Use/download
• Fisher, J., & Coleman, P. (Forthcoming). The fractal nature of intractable conflict: Implications for sustainable transformation. In L. Kriesberg (Ed.) Transforming
Intractable Conflicts. Rowman and Littlefield. Boulder.
• Gamble, J. (2008). A Developmental Evaluation Primer. Montreal. The J.W. McConnell Family Foundation.
• GOPC (2016) Spaces MERL – Systems and Complexity White Paper. Washington DC. USAID.
• Mayo, T. (2016) AEA 2015 - Sentinel Indicators: A Systems-Based Approach to Monitoring and Evaluation. USAID.
• Snowden, D. J. & M.E. Boone (2007). A Leaders Framework for Decision Making, Harvard Business Review
• USAID, 2013. Discussion Note: Complexity Aware Monitoring. V 2.0. Washington DC. Available online at:
http://www.dmeforpeace.org/sites/default/files/0107_Complexity%20Aware%20Monitoring%202013-12-11%20FINAL.pdf
• USAID, 2018. Discussion Note: Complexity Aware Monitoring. Washington DC. Available online at: http://usaidlearninglab.org/library/complexity-aware-
monitoring-discussion-note-brief
• http://www.migration4development.org/docs/logframe.pdf

35. Discussion