1. Evaluating community Evaluating collaborationEvaluating capacity Tom Kelly Annie E. Casey Foundation Tanja Kubas-Meyer Making Connections Providence www.aecf.org public.me.com/tomkellyjr tkelly@aecf.org slideshare.net/tomkaecf (410) 223-2932
2. Annie E. Casey Foundation Established in 1948 by Jim Casey, founder of UPS Casey “cousins” with Casey Family Program, Marguerite Casey Foundation, Jim Casey Youth Opportunity Initiative Mission: To improve the lives of America’s most vulnerable children Operates foster care as Casey Family Services
3. Annie E. Casey Foundation Began grantmaking in 1980’s State child welfare reform Juvenile detention reform New Futures – community/schools Grantmaking in the 1990’s Urban workforce – Jobs Initiative Urban children’s mental health Teen pregnancy prevention – Plain Talk Rebuilding Communities Initiative Grantmaking in the 2000’s Community change initiative – Making Connections PRI’s and responsible redevelopment – East Baltimore
4. Overview of Comprehensive Community Initiatives (CCIs) Goals focus on multiple levels, including individual/family, neighborhood, organizations, community, region Focus on building capacity of neighborhood stakeholders and local institutions Attend to interrelated influences Multiple players, layers, systems
5. Challenges in Evaluating Comprehensive Community Initiatives: Outcomes are either too distal (reduced poverty rates) or too abstract (improved community capacity) Difficulty developing a counterfactual (e.g., Community wide intervention, other programs tackling same issue, lack of control group or comparable community) Outcomes and theories of change shift, as community needs fluctuate and learning occurs Unexpected outside influences
6. Range of Tools for a Continuum of Data Needs/Questions Outcome Evaluation Surveys Anecdote Observations Random Control Comparison T1 vs T2 Sampling Focus Groups Gut Guess (no info) Strongest evidence Good Enough to Decide/Act Now
7. Measuring Success Change in PLACE Change in PEOPLE Change in COMMUNITYCAPACITIES Change in SYSTEMS OF SUPPORT & SERVICES Change in ENVIRONMENT & CONTEXT Contribution of Weed & Seed - Explain WHAT, HOWand WHEREthe initiative is making a difference
8. RBA Model (in one slide) Who are we concerned about? Population What conditions or change do we want for this population? Result How can we measure these results? Indicators How would we know if we are making progress? Performance measures Who has a role to play? Partners What works? Best practices What do we propose to do? Our theory and strategies
9.
10. Results…what difference did we make in terms of: IMPACT on children and families INFLUENCE on policies, practice, systems, and services LEVERAGE of resources
11. Impact, Influence, Leverage We seek to make a positive difference in the lives of children and families through three types of RESULTS: Impact: Changes in a condition of well being for the children, adults, families or communities directly served by grants, programs, agencies, or service systems. Influence: Changes in capacity (including cultural competence), policies, regulations, systems, practice or public opinion. Leverage: Changes in investments by other public or private funders in our strategies to improve outcomes for children and families.
12. Place-based community accountability Program-level AND population-level (whole neighborhood) accountability Beyond traditional contractual obligations to a funder Strengthening shared and collaborative accountability to the whole population across multiple partners and organizations Increased partnership of all partners, players, and funders working in a place
13. Accountability requires… Funders and decisionmakers (i.e., those with power and resources) are held accountable for the decisions they make and they must publicly affirm and report their targets and results All stakeholders must have publicly available data with which to hold all accountable Community residents must be prepared to be organized and participate and act in decisionmaking and accountability processes Transparent and open decisionmaking processes and structures need to exist to hold all stakeholders accountable Incentives for positive progress and sanctions for negative change
17. Measuring Leverage A Practical Guide to Documenting Influence and Leverage In Making Connections Communities Reisman, Jane; Langley, Kasey; Stachowiak, Sarah; Gienapp, Anne www.organizationalresearch.com www.aecf.org
18. Making Connections Poor outcomes for children are geographically clustered, requiring a specific neighborhood-based strategy 10+ year investment in 10 communities (2000 to 2010) Hartford Indianapolis Louisville Milwaukee Oakland Providence Denver Des Moines San Antonio Seattle/White Center
19. Results/Outcomes Adults have work and earnings Families save, build and protect assets Families access quality services Families are connected to social supports Children are healthy and prepared to succeed in school (grade-level reading)
20. Local Learning Partnerships (LLPs) Established to support local Making Connections sites (for 10-years and beyond) with data and information to help strategy development, and self-evaluation Collaborative group of data holders, researchers, evaluators, and data users (including nonprofits and residents) Multiple roles: Building/consolidating local data warehouses Performance and results measurement and feedback to work teams Quantitative and qualitative data collection on families and neighborhoods, including evaluation of process and outcomes Promoting local capacity (by residents and systems) to use data strategically
21. Community capacities Program implementation and management Data, self-evaluation, and accountability Communication, public policy, and will-building Resident leadership and engagement (Authentic Demand) Financial sustainability and infrastructure
22. Community capacities SCOPE – coordinated and integrated services and service systems SCALE – policy and system changes that reach large numbers of families SUSTAINABILITY – financing and infrastructure Authentic demand/resident engagement Data, learning, and accountability
25. Building a scale Developed common (general) indicators across sites Worked with sites to revise and better define concepts and stages Tested in individual sites Added a “score”
26.
27. Using the scale Implemented collaborative and participatory process of measuring capacities in each site Evaluators facilitated a review and discussion of specific areas of work (eg, workforce, assets, school readiness) What is the evidence? Site specific score
28. Contributing to evaluation Annual assessments produce a site specific T1T2T3 Use additional data to answer questions like: Did technical assistance help sites advance? What patterns exist across sites? Does higher capacity lead to more/bigger/better results for families?
29. Lessons Collaborative and participatory process was powerful way to reinforce shared understanding of all partners of concepts and theory of change (decision) More important to focus on change within sites than comparing sites given different environments/context
30. Lessons/Challenges Important to anchor discussion of capacities in terms of results/work Capacity for WHAT? Iterative process Still difficult to summarize across sites Need additional verification of evidence and perspectives Tool/scale less important than the process
31. Measuring… Resident engagement Community engagement Participation Social networks/social capital Civic engagement Community organizing
32. Authentic Demand “Honeycomb” Residents advocate for results in strategy work groups and play key decision making roles Membership networks continue to grow in size and effectiveness Resident leadership training institutionalized in community colleges and other local partners Organizing leads to policy changes at local and state level Resident leaders provide impetus for forming service provider networks
33. What Does Authentic Demand Look Like?(The Forms) Leadership strategies that offer learning opportunities and training to residents and other partners focused on various skills and capacities including strategic planning, data analysis, policy advocacy and others; Strong social networks that build relationships and reciprocal exchange among and between members of the community, and link residents to new resources and opportunities;
34. What Does Authentic Demand Look Like?(The Forms) Community organizing efforts that mobilize community members in the collective and in strategic alliances with other stakeholders seeking better outcomes for children, families and neighborhoods The kind ofcivic participation that enables residents to establish community priorities and then hold elected officials and service providers accountable through voting, local forums, community research, and consumer and policy advocacy.
35. How do we know that it’s meaningful?(Interim Outcomes) Voice. There are ample opportunities for community members to not only make their needs, desires and opinions known, but also to participate with dignity in environments where decisions affecting the community are being made. Accountability. Those who represent community interests at decision-making tables are representative of and accountable to well-defined and legitimate constituencies, including voters, members of a community or faith-based organization, or agency leadership. As important, there is evidence of a “demand environment” services, resources and opportunities are determined more by the needs and desires of those in the community rather than what organizations have available or what service providers “decide” families and the neighborhood need.
36. 3. Learning, skills and capacity. There are consistent, reliable and trustable opportunities for leadership training, mentoring, or other skill building opportunities that enhance the competence and confidence of community members at decision-making tables and in civic life. 4. Identity. People believe they hold a stake in the larger community. In addition, community identity is inclusive and represents its diversity -- different cultural groups within the community are genuinely represented and embraced as members of the community.
37. 5. Reciprocity. Community members are actively encouraged to contribute their time and talent, because there is a clear understanding that everyone has something to offer the change agenda. Service to the community is rewarded through public recognition, ceremonies, and events. A social “norm” of participation and of action is nurtured. 6. Choice. Not only are there many options to engage in and contribute to change efforts, but those options connect to the needs and desires of the community – what has value for them – not the agendas of someone else.
38.
39.
40.
41. Tools & ways of measuring Software for tracking program, process, and contextual data in a central place (Backpack- Backpackit.com) Tracking resident engagement (Constant Contact and YS.net) Mapping participation (GIS)
42. Tools & ways of measuring pre/post...or often better...post-then assessments of learning charting timelines learning meetings core capacity assessment individual and group interviews social network analysis
43. Periphery Cluster Link Node Social Network Analysis: A Few Helpful Definitions Core Hub Source: Monitor Institute
45. Other resources Race Matters Toolkit http://www.aecf.org/racematters.aspx Guide to Evaluating Advocacy & Policy Change http://www.organizationalresearch.com/publications/a_guide_to_measuring_advocacy_and_policy.pdf When and How to Use External Evaluators http://www.aecf.org/publications/data/using_external_evaluators.pdf Tools and Resources for Assessing Social Impact (TRASI) (beta) foundationcenter.org/trasi
