1. 3/23/10
Empowering Youth To Be Evaluators:
Involving young people in evaluating
informal education programs
Amy Grack Nelson, Evaluation & Research Associate
Science Museum of Minnesota
Overview
 Overview of participatory evaluation
 Participatory evaluation examples
 Sampling of interactive techniques
What is participatory evaluation?
1

2. 3/23/10
It’s All About Utility
 Utility - one of the four essential features of all
evaluations
(Joint Committee on Standards for Educational Evaluation, 1994)
 A way to help ensure use is to increase the primary
intended users’ level of participation in the
evaluation. (Cousins & Earl, 1995; Patton, 2008)
Participatory Evaluation
“Applied social research that involves trained evaluation
personnel and practice-based decision makers working
in partnership.” (Cousins & Earl, 1995, pg. 8)
Core purpose  increasing use
Characteristics of Participatory Evaluation
 Balanced control of evaluation process
 Involvement of primary users
 Extensive participation throughout the evaluation
(Cousins & Earl, 1995; Cousins & Whitmore, 1998)
Interactive Evaluation Practice Continuum
(King & Stevahn, 2002)
2

3. 3/23/10
Benefits of Participatory Evaluation
Increases use of evaluation results by:
 Enhancing relevance of the evaluation
 Increasing understanding of the data
 Increasing ownership of the findings
(Cousins & Whitmore, 1998; King & Stevahn, 2002; Patton, 2008)
Evaluation capacity building
 Develop analytic and evaluative skills
 Stakeholders develop a more “critical eye”
(Cousins & Earl, 1992, 1995)
Participatory Evaluation Examples
Science Museum of Minnesota’s
Kitty Andersen Youth Science Center
Kitty Andersen Youth Science Center’s Park Crew
Facilitate earth science and environmental education
activities in the Big Back Yard and on outreaches
3

4. 3/23/10
Example 1: Summative Evaluation
Youth will…
 Learn about water related earth surface processes
 Develop teaching skills
 Learn about related science, technology, engineering, and
math (STEM) careers
Evaluation design
 Observations and interviews of youth staff at the
beginning and end of summer
 Evaluation workshop to engage youth in results
Reviewing the Work
Keep/Change
Discussion
4

5. 3/23/10
Incorporating
Evaluation Data
How often youth talked about why something is considered a pollutant (n=27)
Keep/Change Discussion
Benefits Limitations
 Engages users with data to  Takes considerable amount
think about successes and of time
areas of improvement
 Can be used to generate
recommendations
5

6. 3/23/10
Youth Benefits
Meaningful involvement in evaluation can help youth
develop higher order thinking skills, specifically
analytic and evaluative skills. (London et al., 2003)
 Youth became more reflective of their work.
 Youth comments reflected increased knowledge of the
activities and confidence in sharing that knowledge with
visitors.
 Youth had a stronger sense of ownership and control.
Adult Staff Benefits
 Provided important feedback about the crew’s work.
 Gained deeper understanding of the participants’
experience and could proactively identify and respond
to their needs.
 Increased understanding of evaluation and ability to
interpret data and generate recommendations.
 Developed capacity to include participatory evaluation
in future work.
6

7. 3/23/10
Outcomes of the Process
Participants experience a sense of empowerment and
pride when they have an influence on the way
programs are run and see their ideas acted upon.
(Checkoway et al., 2003; Horsch et al., 2002; London et al., 2003)
 Youth used suggestions to develop their own training.
 They created a visitor survey and collected data.
 Youth shared their ideas with a museum operations staff
member.
Example 2:
Formative Evaluation of Outreaches
Evaluation Process
Identify daily Craft survey
Pilot surveys
objectives questions
Discuss pilot data
Enter and code Administer
and revise
data surveys
surveys
Analyze and Generate Improve outreach
discuss data recommendations activities
7

8. 3/23/10
Organizational Requirements for PE
 Evaluation must be valued
 Sufficient time and resources
 Commitment to organizational learning as a means to
improvement
 Motivated individuals
 Interest and ability to learn evaluative skills
(Cousins & Earl, 1992)
Evaluator Requirements for PE
 Sufficient technical and facilitator skills
 Accessible for participatory activities and support
 Necessary resources and time
 Serve an instructional role
 Motivation and commitment to participate
 A tolerance for imperfection
 Flexibility
(Burke, 1998; Cousins & Earl, 1992; King, 1998)
8

9. 3/23/10
Interactive Techniques
A sampling of interactive methods to gather and
discuss evaluation data
What youth do when they get stuck on a project (n=8)
9

10. 3/23/10
Interactive Graphs
Benefits Limitations
 Quick data collection  People may be
 Everyone can see the influenced by others
process and results  People may be hesitant
 Can be used as a
to place a rating where
starting point for no one else has
deeper conversations
 Can see data by
various characteristics
(King, 2009)
Carousel Sheets
Benefits Limitations
 Alternative to traditional  Participants may influence
brainstorming each others’ responses
 Lots of information in a  Tend to get first responses
short timeframe and gut reaction; not
 Quick way to see patterns deep and thoughtful
 Promotes high involvement  Responses may be too
brief
 Involves users in analysis
 May need to reanalyze
some of the data
(King, 2009)
10

11. 3/23/10
Questions?
Amy Grack Nelson: agnelson@smm.org
To see a sampling of evaluation reports visit:
www.smm.org/researchandeval
11