2. Session Overview
• Interactive - audience participation
• Global Giving’s storytelling work
• Evaluation of networks
• Collaborative learning activity
• Wrap up / Q&A
3. Introduction
• The Rockefeller Foundation Evaluation
Office is supporting innovative evaluation
approaches, new methods and tools and
evaluation capacity development
• This session presents the work of two of
RF’s grantees who are key contributors to
advances in development evaluation
4. Tell us about something that surprised you at InterAction yesterday.
SKILLS
OTHER
THE
IDEAS
EXPERIENCE
What is your story about?
Place it somewhere inside the triangle, or in the “other” box.
7. GlobalGiving Impact Framework
Catalyze Marketplace of $, Info, Ideas
Accessible to Growing # of More Effective Orgs
Democratizes Aid and Philanthropy
9. Closing the Broken Feedback Loop
Feedback makes it safe (and satisfying) to give to
riskier (but maybe better performing) orgs.
Feedback identifies great NGOs and helps them learn.
Resulting data is useful for other funders, NGOs, government
agencies, and researchers.
10.
11. Methodological Logic
Story Capture
Many
Emergence Self-Signification
Act on Insights Narratives
Contextualized
Statistics
Visualization
Discussion (Theme, Geo, Org, Tim
e)
12. What’s working?
• Real-Time Decision-Making Aid for GlobalGiving
Working! • Story Patterns Illuminate Needs
• Actionable Constituent Voice
• Performance Dashboard for NGOs
Promising! • Methodology Recognized as Useful
• Interoperability with other Monitoring Tools
• Globally Scalable Collection Infrastructure
Working on it! • Market-Driven Incentives for Story Collection
• Open-Source Analysis Tools
13. Evaluating
Coalitions and Networks
Jared Raynor
Director of Evaluation
InterAction Conference
May 2012
strategies to achieve social impact
14.
15. Logic Model: Advocacy Initiatives
Acting Project Inputs/ Strategies: Outcomes: Impact:
Organization: Resources:
Advocacy Staff: Issue analysis/
•Time, experience and research
Improvement in the Quality of Living for the Community
expertise Better defined and
Change in Social Structure Reflecting Positive Social Justice Shift
(administrative, legislativ
Nonprofit or Advocacy
framed problems
coalition as a strategy)
Media advocacy/
Organization (uses
e, election-related and
Public awareness
legal)
raising
•Core skills
(analytical, communicatio Issues more clearly set
n, research, etc.) Grass roots on the public agenda
organizing and gain prominence
Funding: (momentum, interest, a
•Monetary resources Coalition building/ wareness, etc.)
•Non-monetary/ non-staff networking
resources (e.g.
volunteers, in-kind Policy makers adopt
donations) Policy analysis/ new policies based on
Coalition/Network
research the agenda / do not
(as an Actor as
compared to a
Organizational adopt harmful policies
strategy)
Reputation: Legal action
•Advocacy related
•General perception as a Lobbying and direct Policy decisions
quality organization policy-maker effectively
•Representative for base influence implemented
constituency
Assure good policy
Network: implementation
•Policy makers (Administrative/ Impact of new policies
•Partner organizations Regulatory is evaluated
•Media oversight, TA, monit
•Mobilized base oring, etc.)
(C) TCC Group 2010
16. Continuum of Inter-Organizational
Relationships (IORs)
Mutual Merger
Accountability
Joint Venture
Strategic Alliance
Collaboration Formal
Informal
Organization Organization
Coalition
Formal Network
Committee
Separate
Accountability
Informal Network
Adapted from: Wolf, T. (2003). A Practical Approach to Evaluation of Collaborations. In T.E. Backer (Ed.). Evaluating Community
Collaborations. New York, NY. Springer Publishing Co. with additional acknowledgement of K. Hobson and M. Hightower King at AEA 2009.
17. Relationships for What
Purpose?
• Respond to complexity
• Cope with turbulence/complexity
• Acquire resources (while maintaining autonomy)
• Decrease transaction costs
• Gain legitimacy/power
Positive Negative
Externalities Externalities
(Value) (Congestion)
18. What Are We Measuring?
Coalition
Organization
Individual
19. An Entanglement of Relations
Coalition
Organization
Organization Number of
Individual Relationships
Organization 2 Person Coalition: 8
Individual
Organization 3 Person Coalition: 18
4 Person Coalition: 32
Individual
Individual
20. A Simplified Approach for
Evaluation
Network/
Member
Coalition Outcomes
Capacity
Capacity
Image credit: http://www.cugelman.com/research/united-nations-web-network.htm
22. Member Capacity
• What do you want to get out of the network/coalition?
• How do you justify your involvement?
• How much time can you devote?
• What can you offer?
Individual Capacity
Network Coalition
Boundary spanners Skill/knowledge to work collaboratively
Time/space to engage in network Time and Commitment to working in
coalition
Relevant skills and interest Relevant decision-making authority
Desire to be part of something Strategic use of coalitions to fill critical
bigger gaps and leverage resources toward
achieving your mission
24. The Critical Organizational Capacities for Advocacy Initiatives: The Logic Model “Inputs” Initiative Logic Model
Leadership:
Management: Technical:
Issue analysis/
research
Improvement in the Quality of Living for the Community
Media advocacy/
Public awareness
raising
Grass roots
organizing
Coalition building/
networking
Policy analysis/
research
Adaptability:
Legal action
Lobbying and
direct policy-
maker influence
Access Administrative/
Regulatory
window of implementation
opportunity/ influence
threat:
•Offense
•Defense
Org. Culture
(C) TCC Group 2008
25. A Few Key IOR Capacities
Value Goal
Proposition Destination
(Why is this IOR the (What is the IOR
right approach?) trying to achieve?)
• Rules, procedures and decision-making
• Action/Exchange-oriented
• Leadership
Entropy Bureaucracy
(disorder) (rigidity)
26. Multiple Levels of Outcome
Community
Community Impact/
Outcomes
Development
of IOR
Coalition/
Network
Organization
Benefits to
Organizations
Individual
Benefits to
Individuals
27. Capturing ‘Externalities’
• Sustained networks/relationships
(Relationships hurt)
• Reduced sense of isolation (Personality
conflicts)
• Better implementation/coordination of actual programs
addressing the issue (More entrenched programming)
• Cross-fertilization and Innovative ideas (Group think)
• Other specific project goals met
28. Outcome questions for IORs in
Emergency Settings
• Has the IOR increased the ability to coordinate during varying
stages of the emergency, including preparation?
• Did the IOR reduce political complexity?
• Did the IOR reduce or increase time and resource constraints?
• Has the IOR increased the reliability of information available?
• Has the IOR facilitated overcoming infrastructure challenges?
• Has the IOR enhanced the well-being of personnel?
• Has the IOR facilitated better relationships with affected
populations and donors?
• Has the IOR increased ability to engage with the media?
30. Contact Information
Jared Raynor
Director of Evaluation
TCC Group
www.tccgrp.com
31. Leveraging the Evaluator
• Navigate power differentials
• Use to:
• Build (inform development)
• Sustain (strengthen existing
performance; identify
inefficiencies)
• Reflect (accountability for
resources and prep for future)
• Understand how to position
yourself in a network
• Help articulate goal destination
and value proposition
32. Mobile Phone Principles of Good
IOR Behavior
1 Getting the Right
Plan
• What do you need to
accomplish?
• Do you have
appropriate
bandwidth (value vs.
congestion)?
• Right amount of
minutes/airtime?
33. Mobile Phone Principles of Good
IOR Behavior
2 Aware of Surroundings
• How should others contact
you? You contact them?
• Are you talking too loudly?
Too quietly?
• How much background
noise is there?
Photo source: whatisacellphonejammer.com
34. Mobile Phone Principles of Good
IOR Behavior
3
•
Strategic Use
Do you have the right
connections?
• Are there new apps?
• Are you playing too much Angry
Birds?
• Rate/assess your experience
35. Mobile Phone Principles of Good
IOR Behavior
1 Getting the Right
2 Aware of Surroundings
• How should others contact
Plan you? You contact them?
• What do you need to • Are you talking too loudly?
accomplish? Too quietly?
• Do you have • How much background
appropriate
bandwidth (value vs.
3
•
Strategic Use
Do you have the right
noise is there?
congestion)?
connections?
• Right amount of
• Are there new apps?
minutes/airtime?
• Are you playing too much Angry
Birds?
• Rate/assess your experience
Hinweis der Redaktion
How we think about our impactSustainable marketplaceGrowing #s of participating orgsHelping those orgs be more effectiveUsing data to inform the sector as a whole
Michael Luca – Harvard Business School researcherhttp://www.hbs.edu/research/pdf/12-016.pdf+1 star = + 5%-9% RevenueOnly independent restaurants – no effect on chains.Chain restaurant market share declines with Yelp penetration.Yelp covers 70% of restaurants. Seattle Times covers 5% Michelin does not cover Seattle.Individual feedback influences the market.Smart store owners seek feedback and react to it.New markers of quality and reliability.More value created benefitting larger, more local group of restaurants.
This is what we’re trying to achieve.But how to we realize these Yelp-like effects within our marketplace?
Prompting question or image triggers a lived experienceStoryteller self-indexes story, gives meaning (can be done by others)Software detects visual patterns among storiesSoftware enables deeper dive into individual storiesPeople discuss patterns and story clustersAct on patterns, weak signals, phase shiftsStimulate beneficial patterns; dampen undesirable ones
Working! Library of Lived ExperiencesVisualizing Shifting Story PatternsDissemination to Official Aid SectorPossibility of Actionable Constituent VoiceUseful to GlobalGivingPromising!Rolling BaselinesUseful to DonorsUseful for OrganizationsUseful for Beneficiary GroupsInformation Challenges Existing FrameworksCross-Organizational ThinkingWorking on it!Stories facilitate decision-makingStories are basis for evidence-based policy changesFraming the ImpactPeer-to-Peer Knowledge Management
This is for an advocacy initiative. Implementing program work is not significantly different.
Concept of interorganiztaional relationships comes from Heather Creech and UniversaliaFrequently a conflation in the literature between coalition and networkCoalitions are almost always organized around member organizations, whereas networks are split between member networks and individual networks; however, in both cases, it is individuals that do the activitiesNatural gravitation toward increased structure or chaos leads to fluidity among the types of IOR that a single entity will go through.
Important to name these things.
So, up to this point you see a notion of convergence—by and large we are talking about nuance on a theme. Thus, the conclusion is one of nuance as opposed to distinct identity. To test whether this holds up, we must dive into the individual indicators and see if at the end they still lend us to this conclusion.
Not radically different questions for members to consider—they go through a similar decision-making process.
Community: It’s impact on; Increased visibility/knowledge of issue/public willBetter/less fragmented relationships with policy-makers and allies and reduced enemiesNew knowledge/development of good research (increased data) Overcoming important “sticking” points in moving an agenda/policyIncreased coordination of activities and better implementationDevelopment of IOR: Its position (power, centrality, capacity)Increased IOR capacity (e.g., clarity of vision; ability to manage/raise resources; visibility, etc.)Increased quality/prestige/ engagement of membershipIncreased collaboration between IOR members outside the IORMore rapid and organized ability to respondMore important to measure cohesion with network because there is a presupposed cohesion through formalization with coalitionsAbility to promote pair-wise relationshipsDensity as a measure (depth, strength and durability)Organization: Either direct or through individuals enhancing the organization based on experience in the networkNew relationship partnersIndividual: Personal development of skills, sense of belonging/identity, motivation, etc.Increased skills/ capacityIncreased legitimacy/ powerIncreased access to resources (while maintaining autonomy)Increased informationSense of belongingDecreased transaction costsUnit of analysis more likely to center on the individual/organization than the network as a whole.