Research, Policy & Evaluation: If I Knew Then What I Know Now: Building Successful Evaluation
1. If I Knew Then What I Know Now: Building a Successful Evaluation Roblyn Brigham, Brigham Nahas Research Associates Andy Hoge, New Jersey SEEDS Janet Smith, The Steppingstone Foundation Danielle Stein Eisenberg, KIPP Foundation April 8, 2010
16. The Lie Factor ( The Visual Display of Quantitative Information, 2 nd Ed. by E.R. Tufte, 2001) Los Angeles Times Aug. 5, 1979, p. 3
Hinweis der Redaktion
Evaluation helps organizations succeed in gaining funding, delivering services, and improving internal processes. Yet conducting rigorous evaluation is a challenge when resources are limited, or you are in charge of evaluation without having had evaluation training. In this workshop, four evaluators of college-access organizations with different levels of experience identify key realworld stumbling blocks such as: 1) confusing evaluation for external use with evaluation for internal use; 2) finding that too much data paralyzes organizational decisions; 3) prioritizing data collection over data analysis; 4) finding your audience’s eyes glazing over when you talk about evaluation; and 5) over or underestimating the resources you will need for your project. This session will focus on evaluation tips and tools, lessons learned, and most importantly, mistakes to avoid. It is designed for those charged with leading evaluation for their organizations (even if they have little evaluation experience); it is not for those completely new to evaluation. Introduction of Panelists: Roblyn Brigham, Brigham Nahas Research Associates Andy Hoge, New Jersey SEEDS Janet Smith, The Steppingstone Foundation Danielle Stein Eisenberg, KIPP Foundation
Overview slide for Janet/Danielle’s section: Describe our orgs re: mission, size Describe our orgs briefly: Two things very important to how we do evaluation: KIPP – national, multiple site, autonomous model (meaning the Foundation does not run or operate the KIPP schools, rather it provides support and training and economies of scale. Currently 82 schools in 20 states and DC, serving 21,000 students. We do internal and external evaluations – I’m going to focus on our internal program evaluation work today.
Evaluation - not a one-size-fits-all approach • Size & Structure of Organization – Who on your staff does program evaluation? No one, everyone or one person? - Evaluator wears many hats? • Culture of Org – Is evaluation part of your organization’s culture already or will this work be entirely new? What systems and processes need to be put in place to create a data-driven culture? What time and resources are dedicated to all phases? Age of org affects what can/should be evaluated Nature of the program offering(s) – direct service, info only, prepping now for future outcomes, multiple sites? Implementing a model vs. responsive to specific program context Through experience we’ve learned that the following matters greatly: - • Size & Structure of Organization – Who on your staff does program evaluation? No one, everyone or one person? - Evaluator wears many hats? • Culture of Org – Is evaluation part of your organization’s culture already or will this work be entirely new? What systems and processes need to be put in place to create a data-driven culture? What time and resources are dedicated to all phases? Age of org affects what can/should be evaluated Nature of the program offering(s) – direct service, info only, prepping now for future outcomes, multiple sites? Implementing a model vs. responsive to specific program context Describe our orgs briefly: Two things very important to how we do evaluation: KIPP – national, multiple site, autonomous model (meaning the Foundation does not run or operate the KIPP schools, rather it provides support and training and economies of scale. Currently 82 schools in 20 states and DC, serving 21,000 students. We do internal and external evaluations – I’m going to focus on our internal program evaluation work today. Two important things about how my org has decided to do eval: DSE: 1. Recently, we built a culture around making program evaluation a critical component to a program’s lifecycle; and engaged program managers in managing their own evaluations 2. Work hard to first define program goals, participant outcomes, and even process goals; and then connect them to the right evaluation tools and processes in order to ensure that the information is useful and actionable.l (show slide with tool to define eval goals, outcomes, tools) JS 1. Learning org – make decisions based on research and data (but easy to leave evaluation thinking until the end) - moving beyond “satisfaction surveys” that ask “rate your level of satisfaction” 2. Interconnected Teams – shared database; own evals and linked evals; mixed methods; Showcase: why does this matter to you?
Evaluable questions: Examples of using proxies, be realistic in making claims Lesson learned: If you are not going to act on evaluation findings, do not collect the data –yet (painful decision-making but necessary)
Data Collection activities: • Articulating “Evaluable” questions - What does Success Look Like? (Theory of change) • (Deciding What Data to Collect) Data Collection – An iterative process – Identifying what data needs to be collected; Ways of collecting data; getting people on board; Identifying who’s involved in data collection • Commitment to Using Results – Being clear upfront how you will USE the results from data collection and analysis: process for making sure the data is utilized for decision-making and program improvement How do data link back to mission and theory of change? What can you take on now? Decide this as part of larger eval over time • What to evaluate (what data to collect)? – Guidelines: Incremental steps based on age of org: New orgs – beginning eval: implementation & staff training After first year – focus on knowledge and behavior outcomes of those you serve After a few years – focus on measuring program impact (with external evaluator?) at theory-of-change level • 1-2 things we’ve learned from experience: DES: 1. When we were a younger organization, no systems in place for doing real program evaluation – program managers were each responsible for doing it on their own, but with no support, no guidelines, and no expertise in this area. Lots of survey monkeying – we learned there needed to be some centralization of efforts to ensure quality evaluation was happening, that we were actually learning from our experiences, that we were retaining information during staff transitions, and that the data we collected was actually utilized. 2. Now that we’re a bit older and have some processes in place, the big question becomes “how much is too much” – survey/interview request fatigue – what data is really necessary – what needs to be collected every year and what doesn’t, etc. 3. Who’s involved is also important – important to have analyst involved in survey development – critical to knowing what data is actually going to be useable. JS: 1. 2. Steppingstone: consider how to send “message in a bottle” - Survey Punch- card, team planning tools, comments in Excel
• Data Analysis - Who’s involved in analyzing the data? Best is team, inclusion of non-eval folks included in later stages to help interpret - Ask yourself: what’s surprising and why? what’s worrisome and why? what’s missing? Share Results of Analysis: org parts are linked via data-driven cycle, activity: each Team presents its data Prioritize action to be taken in response to analysis – formative? Summative? • 1-2 things we’ve learned from experience: DES: 1. KIPP – Articulate the link to your vision/mission: Delivering against vision slide – demonstrate that we set vision first, used variety of data collection methods to track progress towards vision (or goals), and then presented information in ways that were appropriate for audience. 2. Example: Take action based on results; KSLP team utilizes nightly surveys to make immediate adjustments for courses the following day. Others have more subtle changes – but bottom line is – data should be used. JS: 1. Steppingstone – Quarterly Showcase: Teams share their own data analysis, other teams point out how those data affect them, where data-sharing will be key (attention to cross-Team needs is key – JS role, establishing data calendar) 2. Example: Transition Study – exploratory research to include point of view of all broad range of stakeholders. Analyzed themes using x-team analysis group, categorized themes per Team, left each team to do final interpretion and take action: informed support services activities, informed socio-emo curriculum
Knowing your audience: Something we will all respond to who wants headlines? who will want to test your interpretation of the data? analyst not always best person to assess what is best for audience Examples of what we’ve learned through experience: JS 1. visuals count: double-check automated processes - e.g., Excel slide -------------- chart with wrong Y axis for percentile and correct Y axis -------------- slide ------------- USA today graphic or overly busy graphic or data heavy graphic -------- DSE 1. KIPP: Variation in slide decks on various benchmarks – differentiated according to audience 2. What I learned: started to write lots of “reports” or “white papers” at the beginning – truth is – power point often works best to make succinct, easy to digest points, and it’s easily shared.
Ensure participants feel the collective power and impact of our broader Team and Family and the overall movement by providing a powerful introduction and reconnection to KIPP Provide an opportunity for big KIPPsters to network and make connections with others by bringing communities together to share, reflect, and learn Provide an opportunity for personal learning and growth Kick off the 2009-2010 school year with high energy and building momentum towards the belief of what is possible for our kids and renewing our collective commitment to realizing these possibilities in our KIPP communities across the country.
% of doctors has decreased by 15% - two-dimensional data But size of doctor (change in area of image) has decreased by over 75% - Problem using area to represent two-dimensional data
![Overview and Introduction ,[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)