Workshop for NetworkPeninsula - Nonprofit EXcellence in Training (NEXT) Workshop series
Most nonprofits assume they are doing good work and changing the lives of their service recipients, but how do you really know? In this workshop, participants will learn some important concepts, approaches, and techniques to utilize when attempting to measure nonprofit effectiveness. Common mistakes and challenges will also be discussed.
Program Evaluation: Can You Prove You Are Making a Difference?
1. Program Evaluation and
Measurement Outcomes:
Can You Prove You’re Making a Difference?
DCPNI – Isaac Castillo - @Isaac_outcomes 1
Isaac D. Castillo
Deputy Director
DC Promise Neighborhood Initiative
@Isaac_outcomes
Isaac.Castillo@dcpni.org
December 4, 2015
2. Why Should You Care About
Outcomes?
December 2015 Isaac Castillo - @isaac_outcomes 2
• LAYC domestic violence story
– LeapOfReason.org
– First Do No Harm…Then Do More Good
• New domestic violence program
component designed to teach three
things:
– Partner violence is not an OK expression of
love
– Partner violence is not OK in Latino culture
– There are safe ways to get out of violent
relationships
3. Learning Objectives
• Ability to define success for your organization
and its programs/interventions
• Identify ways to measure progress and
outcomes (using multiple approaches)
• Understand how to begin creating an
organizational culture that embraces
information and uses data to make
programmatic improvements
December 2015 Isaac Castillo - @isaac_outcomes 3
4. Who Counts?
• May seem like a
simple thing, but
even defining who
‘counts’ as one of
your participants is
a very important
decision.
December 2015 Isaac Castillo - @isaac_outcomes 4
5. Duplicated vs. Unduplicated Counts
• An unduplicated count is one that counts human
beings
– A person / human being would only be counted ONCE,
regardless of how many activities or sessions they have
attended.
– Example: consider this class – what is the unduplicated
count of this class?
• A duplicated count is one that counts ‘service slots’
– Not a count of human beings – rather it is a count of how
many service slots are occupied.
– Example – if we all came back tomorrow – what would the
duplicated count be? Would the unduplicated count
change?
December 2015 Isaac Castillo - @isaac_outcomes 5
6. Talking and thinking about your
counts
• What do you present publically? A duplicated
count or an unduplicated count?
• Do you ‘count’ someone who only comes to a
single event at your location?
• Do you collect different things for duplicated
or unduplicated populations?
December 2015 Isaac Castillo - @isaac_outcomes 6
7. Who Would You Fund?
• You have $1 million to provide funding to a
tutoring program for “at-risk” youth.
• You need to pick one of two programs to fund
– but you can only pick one!
• Assume everything else is equal aside from
the information provided on the next page.
– Same service population
– Same areas of service
– Same tutoring approach
December 2015 Isaac Castillo - @isaac_outcomes 7
8. Which Program Would You Fund?
December 2015 Isaac Castillo - @isaac_outcomes 8
Program # 1 Program # 2
Served 500 “at-risk” youth Served 50 “at-risk” youth
Provided 2,500 total hours
of tutoring
Provided 2,500 total hours
of tutoring
Each youth received
average of 5 hours of
tutoring
Each youth received
average of 50 hours of
tutoring
5 % of youth showed
improved math grades on
report cards
90 % of youth showed
improved math grades on
report cards
9. Does this Change Things?
December 2015 Isaac Castillo - @isaac_outcomes 9
MathProficiency
Time
Program 1
Comparison Group
Program 1
MathProficiency
Program 2
Comparison Group
Program 2
Time
10. If You Understand Dieting…You
Understand Measurement
• What are some things you want to measure if
you are dieting (how do you know your diet is
‘working’)?
• Why are you measuring these things?
• How do you measure them (and who
measures them)?
• How often (when) do you measure them?
December 2015 Isaac Castillo - @isaac_outcomes 10
11. Same Concept: Different Words
What do you call this sandwich?
December 2015 Isaac Castillo - @isaac_outcomes 11
12. Outputs vs Outcomes
• Output measures assess what you do and who you serve
(Did you actually deliver programming and to whom?)
• Served 100 youth during summer camp
• Provided 2,250 hours of tutoring during the academic year
• 9 out of 10 youth attended at least 75 % of available art instruction
classes offered
Outcome measures assess changes in knowledge, attitudes,
behaviors or conditions for your service population.
• 75 % of youth increased their knowledge of local history during the
summer camp
• 50% of youth increased math grades by one grade level during the
academic year
• 25% fewer youth reported being involved in bullying over the last year
December 2015 Isaac Castillo - @isaac_outcomes 12
13. Outputs
• Outputs DO:
– Tell you about whether your program was
implemented well. For example, they indicate
whether a program:
• delivered the intended number of sessions
• reached its intended population
• resulted in adequate participation levels
• Outputs DO NOT:
– Tell you if participants benefited from your program
– Serve as indicators of program success or
effectiveness
December 2015 Isaac Castillo - @isaac_outcomes 13
14. Outcomes
• Outcomes DO:
– Tell you if participants benefited from your program
– Serve as indicators of program success or effectiveness
• Outcomes DO NOT:
– Tell you about whether your program was implemented
well (or provide clues about how your program improved
participant outcomes)
December 2015 Isaac Castillo - @isaac_outcomes 14
15. Do You Need To Measure Outcomes
for Every Program?
• No.
• You should measure outcomes for your
activities that provide a consistent set of
programming or services for participants.
• Measuring just outputs is OK for some
activities or programmatic approaches.
• Activities or services that are only used once
by people likely will NOT produce outcomes.
December 2015 Isaac Castillo - @isaac_outcomes 15
16. Need to Collect Both Outputs and
Outcomes
• Determine if more services lead to
more/better outcomes
• Determine if there is a ‘minimum’ amount of
service needed to improve outcomes
• Determine if too many services actually lead
to decreased or negative outcomes
December 2015 Isaac Castillo - @isaac_outcomes 16
17. Should You Measure Outcomes For
These Activities?
• An organization provides a health fair for community
members
– No – this is a service that a person would access once. Unlikely
this will lead to any lasting change.
– But you should still track outputs – how many attended?
• An organization provides a series of classes lasting 10
weeks that teaches how to lower cholesterol.
– Yes – should measure changes in knowledge, attitudes, and
possibly behavior.
– Outputs should also be tracked (attendance)
• You could potentially also track:
– How many health fair attendees ended up enrolling in and
completing the classes.
December 2015 Isaac Castillo - @isaac_outcomes 17
18. Confirmatory Outcomes
• Confirmatory outcomes: outcomes you
expect will change as a result of program
participation
– You will hold yourself responsible for confirmatory
outcomes for all (or the majority) of your regular
participants.
– You will devote adequate resources and time to achieving
confirmatory outcomes .
– You will measure your confirmatory outcomes
December 2015 Isaac Castillo - @isaac_outcomes 18
19. Exploratory Outcomes
• Exploratory outcomes : outcomes potentially, but
not definitely, improved by your program
– You think (have a theory) that you may have an effect
on exploratory outcomes.
– But you will not hold yourself responsible for
exploratory outcomes.
– You may or may not measure exploratory outcomes.
– If you do not measure your exploratory outcomes, you
should have a theory (or prior research) that supports
why you think the exploratory outcomes are likely to
happen.
December 2015 Isaac Castillo - @isaac_outcomes 19
20. Confirmatory vs Exploratory
Outcomes Example
• A 24-week program designed to teach teens about the
dangers of tobacco use and decrease (or prevent)
smoking of cigarettes
• Confirmatory outcome: decrease in the self-reported
number of cigarettes smoked during the past 7 days
• Exploratory outcome: decrease in the self-reported
use of alcohol, marijuana, or other drugs during the
past 7 days
The program SHOULD decrease use of cigarettes. And if
the program decreases cigarette use, then participants
MIGHT decrease use of other substances as well.
December 2015 Isaac Castillo - @isaac_outcomes 20
21. Some Examples of Outcomes
December 2015 Isaac Castillo - @isaac_outcomes 21
Healthy eating
habits
Program Outcome(s)
Sex education
for teens
Parent / child
communication
skills
Consumption of
more fruits and
vegetables
Healthier
weight
Decreased
yelling / anger
during conflicts
Increased
knowledge of
dangers of
unprotected /
early sex
Decreased
risky
sexual
activity
Fewer
unplanned
teenage
pregnancies
22. Timing of Outcomes
• Best to think about sequence of outcomes – and
not focus just on when outcomes may/should
occur.
• Early / initial outcomes should lead to more
outcomes, which in turn will lead to more
outcomes.
• Important question: how far down the chain of
outcomes will you measure?
December 2015 Isaac Castillo - @isaac_outcomes 22
23. Short Term Outcomes
• Expect to be affected in a short period of time
• Closely related to/influenced by the program’s
outputs
• Necessary steps
– Ensure program is on track
– Realistic measure of success
December 2015 Isaac Castillo - @isaac_outcomes 23
24. Intermediate and Long-Term
Outcomes
• Intermediate Outcomes
– Link between short and longer-term outcomes
– Most will be confirmatory outcomes
• Long-Term Outcomes
– Change after longer period of time
– Ultimate outcomes
– Mix of confirmatory and exploratory outcomes
December 2015 Isaac Castillo - @isaac_outcomes 24
25. How Long Will Each Type of Outcome
Take?
• Will depend on the length of your
intervention.
December 2015 Isaac Castillo - @isaac_outcomes 25
Short Term Intermediate Long Term
Six MonthsOne Month Twelve Months
6 Years1 Year 12 Years
26. Performance Management and
Evaluation
December 2015 Isaac Castillo - @isaac_outcomes 26
Performance
Management
Regular measurement,
Improve outcomes
Evaluation
Occasional measurement,
Improve effectiveness
27. PM and Eval: Speed
Performance Management
is rapid and frequent
December 2015 Isaac Castillo - @isaac_outcomes 27
Evaluation is methodical
and delayed
28. PM and Eval: Data Collection and
Analysis
December 2015 Isaac Castillo - @isaac_outcomes 28
Performance Management
is simple and requires little
expertise
Evaluation can be complex
and often requires
specialized training
29. PM and Eval: Completeness
December 2015 Isaac Castillo - @isaac_outcomes 29
Performance Management
gives you enough to guess at
whole puzzle
Evaluation lays out as much
of the puzzle as possible
30. Why Bother With All of This?
December 2015 Isaac Castillo - @isaac_outcomes 30
Ultimately, you should be measuring
outcomes or effectiveness for a
single reason:
To better serve your clients / population.
31. Isaac’s Contact Information
31December 2015 @Isaac_outcomes
Isaac D. Castillo
Deputy Director
DC Promise Neighborhood Initiative
On Twitter: @Isaac_outcomes
Email: Isaac.Castillo@dcpni.org