Presented at national Weed & Seed CCDO Conference in Baltimore. Feb 25, 2010

1. Evaluating community Evaluating collaborationEvaluating capacity Tom Kelly Annie E. Casey Foundation Tanja Kubas-Meyer Making Connections Providence www.aecf.org public.me.com/tomkellyjr tkelly@aecf.org slideshare.net/tomkaecf (410) 223-2932

2. Annie E. Casey Foundation Established in 1948 by Jim Casey, founder of UPS Casey “cousins” with Casey Family Program, Marguerite Casey Foundation, Jim Casey Youth Opportunity Initiative Mission: To improve the lives of America’s most vulnerable children Operates foster care as Casey Family Services

3. Annie E. Casey Foundation Began grantmaking in 1980’s State child welfare reform Juvenile detention reform New Futures – community/schools Grantmaking in the 1990’s Urban workforce – Jobs Initiative Urban children’s mental health Teen pregnancy prevention – Plain Talk Rebuilding Communities Initiative Grantmaking in the 2000’s Community change initiative – Making Connections PRI’s and responsible redevelopment – East Baltimore

4. Overview of Comprehensive Community Initiatives (CCIs) Goals focus on multiple levels, including individual/family, neighborhood, organizations, community, region Focus on building capacity of neighborhood stakeholders and local institutions Attend to interrelated influences Multiple players, layers, systems

5. Challenges in Evaluating Comprehensive Community Initiatives: Outcomes are either too distal (reduced poverty rates) or too abstract (improved community capacity) Difficulty developing a counterfactual (e.g., Community wide intervention, other programs tackling same issue, lack of control group or comparable community) Outcomes and theories of change shift, as community needs fluctuate and learning occurs Unexpected outside influences

6. Range of Tools for a Continuum of Data Needs/Questions Outcome Evaluation Surveys Anecdote Observations Random Control Comparison T1 vs T2 Sampling Focus Groups Gut Guess (no info) Strongest evidence Good Enough to Decide/Act Now

7. Measuring Success Change in PLACE Change in PEOPLE Change in COMMUNITYCAPACITIES Change in SYSTEMS OF SUPPORT & SERVICES Change in ENVIRONMENT & CONTEXT Contribution of Weed & Seed - Explain WHAT, HOWand WHEREthe initiative is making a difference

8. RBA Model (in one slide) Who are we concerned about? Population What conditions or change do we want for this population? Result How can we measure these results? Indicators How would we know if we are making progress? Performance measures Who has a role to play? Partners What works? Best practices What do we propose to do? Our theory and strategies

10. Results…what difference did we make in terms of: IMPACT on children and families INFLUENCE on policies, practice, systems, and services LEVERAGE of resources

11. Impact, Influence, Leverage We seek to make a positive difference in the lives of children and families through three types of RESULTS: Impact: Changes in a condition of well being for the children, adults, families or communities directly served by grants, programs, agencies, or service systems. Influence: Changes in capacity (including cultural competence), policies, regulations, systems, practice or public opinion. Leverage: Changes in investments by other public or private funders in our strategies to improve outcomes for children and families.

12. Place-based community accountability Program-level AND population-level (whole neighborhood) accountability Beyond traditional contractual obligations to a funder Strengthening shared and collaborative accountability to the whole population across multiple partners and organizations Increased partnership of all partners, players, and funders working in a place

13. Accountability requires… Funders and decisionmakers (i.e., those with power and resources) are held accountable for the decisions they make and they must publicly affirm and report their targets and results All stakeholders must have publicly available data with which to hold all accountable Community residents must be prepared to be organized and participate and act in decisionmaking and accountability processes Transparent and open decisionmaking processes and structures need to exist to hold all stakeholders accountable Incentives for positive progress and sanctions for negative change

14. Results Accountability www.resultsaccountability.com http://raguide.org/ www.aecf.org “Casey ResultsNet”

15. Weed & Seed Collaboration Coordination Community participation Leverage

16. Weed & Seed Collaboration Coordination Community participation AUTHENTIC DEMAND Leverage CORE CAPACITY ASSESSMENT

17. Measuring Leverage A Practical Guide to Documenting Influence and Leverage In Making Connections Communities Reisman, Jane; Langley, Kasey; Stachowiak, Sarah; Gienapp, Anne www.organizationalresearch.com www.aecf.org

18. Making Connections Poor outcomes for children are geographically clustered, requiring a specific neighborhood-based strategy 10+ year investment in 10 communities (2000 to 2010) Hartford Indianapolis Louisville Milwaukee Oakland Providence Denver Des Moines San Antonio Seattle/White Center

19. Results/Outcomes Adults have work and earnings Families save, build and protect assets Families access quality services Families are connected to social supports Children are healthy and prepared to succeed in school (grade-level reading)

20. Local Learning Partnerships (LLPs) Established to support local Making Connections sites (for 10-years and beyond) with data and information to help strategy development, and self-evaluation Collaborative group of data holders, researchers, evaluators, and data users (including nonprofits and residents) Multiple roles: Building/consolidating local data warehouses Performance and results measurement and feedback to work teams Quantitative and qualitative data collection on families and neighborhoods, including evaluation of process and outcomes Promoting local capacity (by residents and systems) to use data strategically

21. Community capacities Program implementation and management Data, self-evaluation, and accountability Communication, public policy, and will-building Resident leadership and engagement (Authentic Demand) Financial sustainability and infrastructure

22. Community capacities SCOPE – coordinated and integrated services and service systems SCALE – policy and system changes that reach large numbers of families SUSTAINABILITY – financing and infrastructure Authentic demand/resident engagement Data, learning, and accountability

23. Measuring Community Capacities Began with stages of change framework (INSITES, Bev Parson)

25. Building a scale Developed common (general) indicators across sites Worked with sites to revise and better define concepts and stages Tested in individual sites Added a “score”

27. Using the scale Implemented collaborative and participatory process of measuring capacities in each site Evaluators facilitated a review and discussion of specific areas of work (eg, workforce, assets, school readiness) What is the evidence? Site specific score

28. Contributing to evaluation Annual assessments produce a site specific T1T2T3 Use additional data to answer questions like: Did technical assistance help sites advance? What patterns exist across sites? Does higher capacity lead to more/bigger/better results for families?

29. Lessons Collaborative and participatory process was powerful way to reinforce shared understanding of all partners of concepts and theory of change (decision) More important to focus on change within sites than comparing sites given different environments/context

30. Lessons/Challenges Important to anchor discussion of capacities in terms of results/work Capacity for WHAT? Iterative process Still difficult to summarize across sites Need additional verification of evidence and perspectives Tool/scale less important than the process

31. Measuring… Resident engagement Community engagement Participation Social networks/social capital Civic engagement Community organizing

32. Authentic Demand “Honeycomb” Residents advocate for results in strategy work groups and play key decision making roles Membership networks continue to grow in size and effectiveness Resident leadership training institutionalized in community colleges and other local partners Organizing leads to policy changes at local and state level Resident leaders provide impetus for forming service provider networks

33. What Does Authentic Demand Look Like?(The Forms) Leadership strategies that offer learning opportunities and training to residents and other partners focused on various skills and capacities including strategic planning, data analysis, policy advocacy and others; Strong social networks that build relationships and reciprocal exchange among and between members of the community, and link residents to new resources and opportunities;

34. What Does Authentic Demand Look Like?(The Forms) Community organizing efforts that mobilize community members in the collective and in strategic alliances with other stakeholders seeking better outcomes for children, families and neighborhoods The kind ofcivic participation that enables residents to establish community priorities and then hold elected officials and service providers accountable through voting, local forums, community research, and consumer and policy advocacy.

35. How do we know that it’s meaningful?(Interim Outcomes) Voice. There are ample opportunities for community members to not only make their needs, desires and opinions known, but also to participate with dignity in environments where decisions affecting the community are being made. Accountability. Those who represent community interests at decision-making tables are representative of and accountable to well-defined and legitimate constituencies, including voters, members of a community or faith-based organization, or agency leadership. As important, there is evidence of a “demand environment” services, resources and opportunities are determined more by the needs and desires of those in the community rather than what organizations have available or what service providers “decide” families and the neighborhood need.

36. 3. Learning, skills and capacity. There are consistent, reliable and trustable opportunities for leadership training, mentoring, or other skill building opportunities that enhance the competence and confidence of community members at decision-making tables and in civic life. 4. Identity. People believe they hold a stake in the larger community. In addition, community identity is inclusive and represents its diversity -- different cultural groups within the community are genuinely represented and embraced as members of the community.

37. 5. Reciprocity. Community members are actively encouraged to contribute their time and talent, because there is a clear understanding that everyone has something to offer the change agenda. Service to the community is rewarded through public recognition, ceremonies, and events. A social “norm” of participation and of action is nurtured. 6. Choice. Not only are there many options to engage in and contribute to change efforts, but those options connect to the needs and desires of the community – what has value for them – not the agendas of someone else.

41. Tools & ways of measuring Software for tracking program, process, and contextual data in a central place (Backpack- Backpackit.com) Tracking resident engagement (Constant Contact and YS.net) Mapping participation (GIS)

42. Tools & ways of measuring pre/post...or often better...post-then assessments of learning charting timelines learning meetings core capacity assessment individual and group interviews social network analysis

43. Periphery Cluster Link Node Social Network Analysis: A Few Helpful Definitions Core Hub Source: Monitor Institute

44. Lessons learned

45. Other resources Race Matters Toolkit http://www.aecf.org/racematters.aspx Guide to Evaluating Advocacy & Policy Change http://www.organizationalresearch.com/publications/a_guide_to_measuring_advocacy_and_policy.pdf When and How to Use External Evaluators http://www.aecf.org/publications/data/using_external_evaluators.pdf Tools and Resources for Assessing Social Impact (TRASI) (beta) foundationcenter.org/trasi