The document provides an overview of Lauren Decker's presentation on going beyond evaluation buzzwords. It discusses the importance of logic models and theories of change in evaluation. A logic model graphically depicts a program's key components, including resources, activities, participants, and short-, medium-, and long-term outcomes. It also shows how these components are hypothesized to produce intended changes. Measuring implementation fidelity, or the extent to which a program's components are delivered as intended, is important for evaluation. High-quality evaluations clearly use sound practices, focus on use and usefulness, align with program logic models, and consider implementation fidelity. Evaluators should ask the right questions and have clearly defined outcomes.

1. Going Beyond
Evaluation
Buzzwords
Lauren Decker, Ph.D.
Edvance Research
Presented at Texas ACE Annual
Conference, Austin, TX.
August 7, 2013

2. One word or phrase
expressing how you feel
about evaluation

3. “If you don’t know where you are going, how are you gonna’
know when you get there?”
 Yogi Berra

4. Where are we going and how
will we get there?
 What’s a logic model and how does it help?
 Defining seemingly inter-changeable terms
 Time and tool to begin a logic model (or refine one)
 What is actually happening and how does it help to
know?
 Program and evaluation importance of understanding
implementation fidelity
 Bringing thoughtful program lessons into evaluation
plan and use
 What makes a great evaluation?
 Planning and asking the right question(s)

5. Logic Models and
Theories of
Change

6. Logic Model vs. Theory of
Change
 Logic Model = graphically depicts what your program
is intended to do
 Key program components
 Theory of Change = why your program
operates as intended and how
components and activities are hypothesized
to move outcomes
 Assumptions underlying
expected change

7. Simple Logic Model
Inputs Outputs Outcomes
Resources Activities Participation Short
Medium
Long
What
we do
Who we
reach
What results
we expect
What we
invest

8. Logic Model Example

9. How does having a logic
model help my program?
Program phase
 New program
 Existing program
 Redesigning
existing program
Logic model use
 Creation & planning
 Documentation &
communication
 Reinvention,
improvement, &
expansion

10.  Use the paper
provided to begin
a logic model for
your program
 Remember to
include resources,
intended activities,
participants, short-,
mid-, and long-
term outcomes
 If you can also
represent the
theory of change
(arrows)
What does our
program logic
model look like?
10 minutes

11. What should we include in our
program Logic Model?
 Identify:
 All key components of program
 Resources, intended activities, participation, short-,
mid-, and long-term outcomes
 Components and pathways(mediators) through
which program is expected to produce intended
outcomes (theory)
 Student outcome(s) program is designed to
improve
 Short-, mid-, long-term outcomes
 Other outcomes as well if relevant

12. I have a Logic
Model, now
what?
– What is fidelity of
implementation?

13. Fidelity of Implementation
 The extent to which key components of
the program are delivered as originally
intended by the developer
 Key components = strategies, practices,
activities and behaviors that are critical to
defining your program

14. Logic Model Example

15. Fidelity of Implementation
 the extent to which key components of
the program are delivered as originally
intended by the developer
 Key components = strategies, practices,
activities and behaviors that are critical to
defining your program
 When is your program not your program?

16. How do I measure fidelity of
my program?
Fidelity
Structural
key
components
Interactional
key
components
Structure Process

17. How does measuring fidelity
help my program?
 Understanding fidelity helps you know about:
 Participation
 Activity delivery
 Content
 Example questions you can answer:
 Were activities implemented by program staff
according to design?
 Did staff receive program content as planned?
 What were the adaptations that were made to the
program?
 What were the factors that may have affected program
fidelity?

18. “I think you should
be more explicit
here in Step Two.”
Evaluations without measuring
fidelity…

19.  Visit someone
else’s logic
model in process
 For the activities
listed suggest:
 What type of
fidelity could be
measured
 Ways of
measuring that
type of fidelity
for the activity
How to measure
fidelity in your
program?
10 minutes

20. What do we need to do to
understand fidelity of our program?
 Periodically measure implementation
fidelity
 Measured separately for each key
component
 Thresholds specified for determining
whether key components of program
were implemented with fidelity
 Assess and Report whether each key
component was implemented with fidelity

21. What makes a
great evaluation?

22. High-quality evaluations include:
 Evaluators who clearly use sound evaluation
principles and practices (transparency,
consideration of context, etc.)
 Complex programs require thoughtful, directed
evaluations
 Focus from the start on use of evaluation
processes and products to provide information
to, and about, the program being evaluated
 Alignment to the program’s logic model and
theory of change
 Include provision for some level of
implementation fidelity

23. High-quality evaluations include:
 Preparation and planning for variation within multi-
site evaluations
 Consideration of clear deadlines for completion
and deliverables set by program
 Upfront acknowledgement of the limitations of the
proposed evaluation
 Minimizing jargon to increase clarity of
communication of results
 Striking a balance between consultancy/
recommendation support and maintaining
unbiased distance as an independent evaluator

24. Before meeting with an
evaluator:
 Complete your program logic model and
theory of change
 Or have discussions and ideas of key
components for development assistance from
evaluator
 Consider what you really want to learn
 What outcomes do you want an evaluation to
focus on for your program?

25. What do you really want to know?
Make sure you ask the right question!

26. Before meeting with an
evaluator:
 Complete your program logic model and
theory of change
 Consider what you really want to learn
 What data is already available or will need to
be collected to answer your right question(s)?
 If you believe data will need to be collected do
you want to collect them or have the evaluator
conduct the data collection?
 Think about whether you have identified SMART
outcomes!

27. SMART Outcomes are:
 Specific
 Reflect simple information that is communicable and
easily understood
 Measurable
 Can changes by measured in reliable and valid ways?
 Achievable
 Able to be collected and sensitive to change during the
allotted time
 Relevant
 Reflect information that is important and likely to be used
 Time bound
 Progress can be tracked at desired frequency within
allotted time
Source: Adapted from
World Bank Group.

28. Questions for an evaluator
 How does your proposed evaluation align with our
program logic model and theory of change?
 What data do you need from us to conduct the
evaluation? What data to you propose to collect,
when, and in what format?
 How intrusive will the data collection be for our
program?
 Will your evaluation be able to tell us if the program
was in place as intended?
 What we really want to know from an evaluation is
_________. How will your proposed evaluation
answer this question?

29. Questions to ask yourself about an
evaluation proposed to you
 Does the evaluation design reflect my logic model
and theory of change?
 Do the proposed outcomes follow my logic model
and theory of change?
 Is it feasible to collect outcome data at the time I
expect to see changes (short- medium- and long-
term outcomes) in those selected?
 Are the selected outcomes and measures
relevant for my program and my stakeholders?
 Does the evaluation include SMART outcomes that
will be evaluated?

30. Logic Model Example

31. Questions?
Thank you.
Lauren Decker, Ph.D.
ldecker@edvanceresearch.com