Why youth mentoring as an intervention strategy? Why be systematic/rigorous about developing (and improving) mentoring intervention strategies and evaluating their effectiveness? What is “best practice” when developing mentoring intervention strategies? What are the most rigorous and informative methods for evaluating youth mentoring intervention strategies?

1. Development
and Evaluation
of Mentoring
Programs for
Youth

2. Strategies for Fostering
Success in the Next
Generation
David L. DuBois
University of Illinois at Chicago
Colloquium Presentation, University
of Girona, Barcelona, Spain
June 20, 2013

3.  Why youth mentoring as an intervention strategy?
 Why be systematic/rigorous about developing (and
improving) mentoring intervention strategies and
evaluating their effectiveness?
 What is “best practice” when developing mentoring
intervention strategies?
 What are the most rigorous and informative methods
for evaluating youth mentoring intervention
strategies?
 Concluding thoughts

4.  Fundamental role of mentoring in human development
 Evolutionary basis
• Mechanism for intergenerational knowledge transfer (Wong, 2009)
• Adaptive advantage of altruism toward other group members (Bowles, 2006)
• Value of shared responsibilities for caregiving: allo-parenting (Diamond, 2012) and
age-mixing (Ellis et al., 2012)
 Development and learning powered by dyadic relationships with 4 key
features (Bronfenbrenner, 2005; see also Li & Julian, 2012)
• Emotional attachment
• Reciprocity
• Progressively more complex activities
• Gradual shifting of power in favor of developing person

5.  Lack of Access to Mentoring in Contemporary Society
 ~1 in 5 youth in the U.S. estimated be lacking supportive relationships
with caring adults
• Least access among youth experiencing economic disadvantage
 Evidence suggests proportion of youth without any “go to” adult may be
increasing
Source: DuBois (2013).

6.  Empirical Support for Youth Mentoring Supports
 Capacity of mentoring programs to promote youth outcomes
demonstrated through several meta-analyses of rigorous (quasi-
experimental and experimental) evaluation studies (DuBois et
al., 2002, 2011; Tolan et al., 2008; Wheeler et al., 2010)
 Evidence of:
• Benefits in multiple areas: social, behavioral, emotional, academic

7. 0
0.2
0.4
EffectSize
Youth Mentoring
(Pre-1999)
Youth Mentoring
(1999-2010)
Source: DuBois et al. (2002, 2011)

8.  Evidence of:
• Benefits in multiple areas: social, behavioral, emotional, academic
• Magnitude of impacts that are comparable to those achieved by other forms of
youth intervention, but may be more holistic/multi-faceted
0
0.1
0.2
0.3
EffectSize
Youth Mentoring
(1999-2010)
Other Youth
Interventions
Source: DuBois et al. (2011)

9. -0.5
-0.3
-0.1
0.1
0.3
0.5
Pre-Test Post-Test
YouthOutcomes
Non-Mentored Youth
Mentored Youth
Promotion
Prevention
 Evidence of:
• Benefits in multiple areas: social, behavioral, emotional, academic
• Magnitude of impacts that are comparable to those achieved by other forms of youth
intervention, but may be more holistic/multi-faceted
• Both preventive and promotive effects (DuBois et al., 2011)

10.  Evidence that systematic intervention development
strategies (e.g., theory- and research-basis) tend to
result in more effective programs
 Systematic development likely to produce more
replicable and scalable programs

11.  Limited magnitude of current program effects indicates
need to search for stronger programs/strategies
 Not safe to assume effectiveness of a
strategy/program based on prior evaluations due to
potential differences in implementation
quality, targeted youth population, local
conditions, etc.
 Need to grow knowledge base for the field and fill in
gaps (e.g., long-term effects)

12.  Consider drawing on one or more existing program
planning frameworks (e.g., PRECEDE-PROCEED;
Greene & Kreuter, 1999)
1. Assess Needs &
Capacities
of Population
2. Assess Causes,
Set Priorities &
Objectives
3. Design &
Implement
Program
4. Evaluate Program
Reassess causes
Redesign
Source: Green & Kreuter (2005)

13.  Assess needs, assets, and priorities of the community
to be served and design strategies accordingly.
 Potential sources of information:
• Existing: Research, public data bases
• New: Key informant interviews, focus groups, surveys
 Potential questions:
• Who has least access to mentoring?
• Who stands to benefit most from mentoring?
• What kinds of mentoring are most needed?
• What types of mentoring are already occurring?
• What assets (human, organizational, etc.) could be leveraged to
provide effective mentoring?

14.  Search for “leads” on promising intervention strategies
 Review scholarly literature:
• Theoretical models
• Observational research
• Evidence-supported strategies – SPECIAL EMPHASIS
– Priority on high quality experimental and quasi-experimental evaluations
– Draw on findings of systematic reviews of such evaluations, especially
meta-analyses
DuBois & Karcher (2014) Herrera, DuBois, & Grossman (2013)

15. • A Few Recent Examples (among many!)
– Youth-nominated mentoring associated with improved educational and
employment outcomes for marginalized older youth (Schwartz et al., in press)
– Beneficial effects of community-based mentoring on depressive symptomatology
among higher-risk youth (Herrera et al.. 2013)
– Emphasis on similarity of interests when matching mentors and youth predictive
of greater program effectiveness (DuBois et al., 2011)
– Support for mentor assuming teaching and advocacy roles also linked to
enhanced effectiveness (DuBois et al., 2011)

16. Empirically-Supported Best Practices
-0.1
0
0.1
0.2
0.3
0.4
0.5
0 1 2 3 4 5 6 7
Number of Practices
SizeofEffectonYouth
Small
Effect
Medium
Effect
(Based on DuBois et al., 2002)

17.  Outreach to practice community
• Program models and intervention strategies already in use
• Seek input and ideas on directions for program/strategy design
 Obtain input from stakeholders
• Acceptability, usability, feasibility, utility
• Youth, mentors, parents, staff
 Develop and pilot strategies/program components
 Build program logic model / theory of change
 Iteratively refine through multiple cycles of implementation
 Collect and analyze process evaluation data to guide
process
 Carefully document and “manualize” all
components/activities

18.  Example: GirlPOWER! (DuBois et al., 2008)
 Mentoring program for ethnic minority girls ages 10-13
• Designed as enhancement of Big Brothers Big Sisters (BBBS)
community-based mentoring (CBM) program
• 12 monthly, 3-hour group sessions in 10-15 mentor-youth pairs
 Assessment
• Informed by:
– Interviews with youth development professionals, mentors, parents, and youth
– Review of research
• Findings:
– Needs -- Lack of evidence-based mentoring programs for girls
– Assets -- Girls group-based programming model already in use by local BBBS
agency
– Priorities – Girls’ need for support particularly great during transition to
adolescence

19.  Research and theory basis
• BBBS CBM program model is empirically supported
• Incorporated practices linked to greater effectiveness in meta-
analysis of mentoring program effectiveness (DuBois et al., 2002)
– Structured activities for mentors and youth
– Ongoing mentor training
– Parent support and involvement
• Built in theory-/research-based
gender-sensitive program
elements
– Relational focused group format
– Counter limiting stereotypes of girls
– Focus on salient concerns for girls
during transition to adolescence

20.  Program design process
• Informed by interviews/focus groups with stakeholders
(youth, mentors, parents, program staff)
– Example: Girls interested in competition, but did not want to resemble school
• Iterative piloting and refinement of group sessions and other program
components
– Example: Build in more time for informal socializing based on input of mentors
and youth
• Ongoing manualization of all aspects of program
– Examples: Mentor training, group sessions, out-of-session activity guides, parent
communications

22.  Example: Step-It-Up-2-Thrive (research in progress)
 Mentoring framework for youth ages 10-16
• Designed as enhancement of BBBS CBM program
• Based on “Road Map” for thriving: Sparks, Growth Mindset, Thriving
Indicators, Goal Management/Pursuit Skills
• Resources available on-line (http://www.stepitup2thrive.org/)
 Assessment
• Informed by:
– Dialogue with BBBS national program staff
– Review of research
• Findings:
– Needs – Lack of evidence-based programming to promote youth thriving through
mentoring
– Assets -- BBBS program model includes regularly occurring support contacts with
mentors = important mechanism for program delivery
– Priorities – Growing interest in moving beyond “prevention” model to “promotion”
model focused on helping youth to reach their full potential

24.  Research and theory basis
• Recent research findings point toward contribution of abilities and
experiences in each of the four areas of the “Road Map” to outcomes
important for youth (e.g., academic success, avoiding involvement in
problem/delinquent behavior) Informed by:
– Associations for most part relatively modest, suggesting that combination of
strengths in all areas may be needed for optimal outcomes
– Theoretically, effects of factors in one area also may be dependent on those in
others (e.g., need growth mindset to effectively pursue sparks)
• Emphasis on teaching/guidance role for mentor supported by findings
of recent meta-analysis of mentoring program effectiveness (DuBois
et al., 2011)

25. Positive effects of intervention teaching incremental
theory (growth mindset) to 7th graders
(Blackwell, Trzesniewski, & Dweck, 2007)2

26.  Program design process
• Drew heavily on existing resource materials from Thrive Foundation
for Youth
• Driven by national BBBS staff and input from staff of 3 “lead design”
agencies (BBBS affiliates)
• Piloting of mentor training and group activities
• Manualization of all aspects of program

27.  Planning
 Decide on primary purposes of evaluation
 Ensure adequate resources and technical expertise are
available
 Consult an accepted set of standards for program evaluation
(e.g., Program Evaluation Standards of Joint Committee on
Standards for Education Evaluation)
 Obtain approval from research review board
*See DuBois (2014) for a more detailed treatment of issues involved with evaluation of
mentoring programs.

28.  Process Evaluation
 Major purposes
• Determine whether program happens as planned
• Obtain information to guide further development/refinement of
program
• Accountability to stakeholders
• Support good program delivery
– Monitoring procedures likely to improve implementation – “what gets
measured, gets done” – and linked to better mentoring program effects (DuBois
et al., 2002)

29.  Should assess both fidelity of implementation and dosage
• Fidelity = extent to which program/strategy was implemented as
planned
– Assess for both staff- and mentor-implemented activities
– Key components to assess:
» Adherence: extent to which services/activities are carried out according to
design
» Exposure: level of intended services/activities provided
» Quality of Delivery: how well services/activities are implemented
» Responsiveness: how participants experience services/activities
» Differentiation: extent to which the services/activities differ from those
associated with other similar programs
• Dosage = levels of exposure/participation for individual participants
– Assess with respect to services provided to both youth and their mentor
 Important to gather both quantitative and qualitative data
• Quantitative: May be particularly helpful for gauging whether
implementation benchmarks are met and whether these improve over
multiple implementations
• Qualitative: May be particularly helpful for identifying ways to
improve program content and for detecting unexpected experiences
of participants

30.  Outcome Evaluation
 Major purposes
• Determine whether program has intended effects on participants
• Test program’s theory of change
• Obtain information needed to support economic evaluation of
program’s “return on investment”

31.  Keys
• Need credible comparison group of non-participants to obtain
accurate and unbiased estimate of program effects
– Random assignment to program vs. comparison group = “gold standard”
• Need sufficiently large sample to detect effects
• Need reliable and valid measures of program outcomes
– Use program logic model / theory of change as guide
– Ideally measure key outcomes through multiple methods – “triangulate”
• Incorporate follow-up assessment of outcomes beyond the period of
program involvement
• Analyze data for both overall program effects and effects on different
subgroups of youth

32.  Example: GirlPOWER!
 Small-scale (n = 40) randomized controlled trial comparing
GirlPOWER! to standard BBBS community-based mentoring
 Process Evaluation
• Agency records, mentor reports
(Adherence, Exposure, Differentiation, Dosage)
• Direct observation (Quality of delivery)
• Youth, mentor, parent feedback surveys (Responsiveness)
 Outcome Evaluation
• Youth (and their mentors) randomly assigned to participate in
GirlPOWER! or BBBS standard programming
• Pre-test and one-year (end of program)
• Youth, parent, and mentor surveys
• School and juvenile system records

33.  Selected Findings
• Process Evaluation
– Strengthening mentor-youth relationships
» Mutual sharing
» “we got to learn more about each other” [mentor]
» “we communicated more” [youth]
» “Seeding” discussions and activities
» “… it will be easier for us to discuss peer pressure & drugs” [mentor]
» “… it gave me some ideas on how we can stay fit together” [youth]
– Facilitating group support and cohesion
» Supportive group experience for youth
» “she really looks forward to … hanging with the Littles” [mentor]
» “[performing in talent show helped me] break out of my shell” [youth];
» Mentor sharing and support
» “meeting similar Matches makes the entire BBBS experience better”
[mentor]
» “… the Bigs find it helpful to share info, concerns, etc.” [mentor]
» Match connections
» “Bigs can share ideas & make plans together …” [mentor]

34. – Program improvements guided by findings
» POWERSessions
» make activities more interactive, less didactic; free up time for casual
socializing
» greater utilization of mentors as teachers and role models
» POWERBuilders
» Encourage matches to design their own activities; be liberal in
“crediting” activities as addressing different program topics
» Resource materials
» Reduce length and organize in program binder

35. – Improvements evident in youth and mentor experiences of the program sessions
Spring 2005 Fall 2005
AverageRatingsfor"FUN"
1
2
3
4
5
Little(Self) Big(Little) Big(Self)
Respondent
Rating
AverageRatingsfor"HELPFUL"
1
2
3
4
5
Little(Self) Big(Little) Big(Self)
Respondent
Rating

36. • Outcome Evaluation
– Improvements in mentoring relationship quality
» More frequent mentor-youth contact
» Stronger instrumental/goal-oriented dimension of relationships
» Mentor seen by youth as more disapproving of problem behaviors
(e.g., unhealthy eating, substance use)
– Improved youth outcomes (greater improvement than youth in standard
community-based mentoring)
» Health behavior
» Health knowledge & beliefs
» Exercise
» Emotional well-being and resilience
» Perceived coping efficacy
» Domain-specific and global self-esteem
» Anxiety/depression
» Academic
» Intrinsic motivation for learning
» Educational aspirations

39.  Example: Step-It-Up-2-Thrive
 Randomized controlled trial comparing BBBS community-
based mentoring with and without integration of Step-It-Up-
2-Thrive framework
 10 BBBS agencies, ~800 youth ages 10-16 with one or more
risk factors for engaging in delinquent behavior
 Process Evaluation
• Staff reports, agency records (Adherence, Exposure, Differentiation)
• Implementation checklist completed by staff (Dosage)
• Staff self-assessment, participant feedback (Quality of delivery)
• Youth, mentor, parent feedback surveys (Responsiveness)
 Outcome Evaluation
• Youth (and their mentors) randomly assigned to participate in BBBS
CBM with addition of “Thrive” integration or standard programming
• Pre-test and 15 months
• Youth, parent, and mentor surveys

40.  Benefits of “real time” analysis of process evaluation data
• Unexpectedly low rates of participation in group activities for matches
(mentor-youth pairs)
– Adjustment:
» Development of alternative formats
» Activity guides for mentor-youth pairs to work through on their own
» On-line format
• Challenges with having matches move through different stages of
intervention activity in planned sequence
– Adjustment:
» Allow flexibility in sequencing where fits with needs and interests of youth
(and mentor)
• Lack of case manager comfort with supporting
intervention content
– Adjustments:
» Increase their role as “trainers” of mentors
» Provide greater opportunities for technical assistance from
national staff

41.  Promise of mentoring is arguably as well-established
as any form of youth intervention
 It would appear, however, that only a surprisingly small
portion of this potential has been realized
 More systematic and rigorous approaches to development
and evaluation of mentoring intervention strategies could
help to narrow this gap
 Important not to overlook need for complementary
focus on improving extent & equity of youth access to
effective forms of mentoring

42.  Impact + Reach = Healthier Future Generations and
Greater Global Prosperity

43. Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict
achievement across an adolescent transition: A longitudinal study and an intervention. Child
Development, 78, 246-263.
Bowles, J. (2006). Group competition, reproductive leveling, and the evolution of human altruism.
Science, 314(5805), 1569–1572.
Bronfenbrenner, U. (2005). Making human beings human: Bioecological perspectives on human
development. Thousand Oaks, CA: Sage.
Diamond, J. (2012). The world until yesterday. New York: Viking Press.
DuBois, D. L. (2013). Trends and correlates of youth reports of not having a parent or other adult
available to provide support with personal problems. Manuscript in prepartion
DuBois, D. L. (2014). Program evaluation. In D. L. DuBois & M. J. Karcher (Eds.), Handbook of
youth mentoring (2nd ed., pp. 481-498). Thousand Oaks, CA: Sage.
DuBois, D. L., Holloway, B. E., Valentine, J. C., & Cooper, H. (2002). Effectiveness of mentoring
programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157–
197.
DuBois, D. L., & Karcher, M. J. (Eds.). (2014). Handbook of youth mentoring (2nd ed.). Thousand
Oaks, CA: Sage.
DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective
are mentoring programs for youth? A systematic assessment of the evidence. Psychological
Science in the Public Interest, 12, 57–91.

44. DuBois, D. L., Silverthorn, N., Pryce, J., Reeves, E., Sanchez, B., Silva, A., . . . Takehara, J.
(2008). Mentorship: The GirlPOWER! program. In C. LeCroy, Craig Winston, & J. E. Mann
(Eds.), Handbook of prevention and intervention programs for adolescent girls (pp. 326–336).
Hoboken, NJ: Wiley.
Ellis, B. J., Del Giudice, M., Dishion, T. J., Figueredo, A. J., Gray, P., Griskevicius, V., . . .
Wilson, D. S. (2012). The evolutionary basis of risky adolescent behavior: Implications for
science, policy, and practice. Developmental Psychology, 48, 598-623.
Green, L.W.,&Kreuter, M.W. (2005). Health promotion planning:An educational and ecological
approach (4th ed.). New York: McGraw-Hill.
Herrera, C., DuBois, D. L., & Grossman, J. B. (2013). The role of risk: Mentoring experiences and
outcomes for youth with varying risk profiles. New York: A Public/Private Ventures project published
by MDRC.
Li, J., & Julian, M. (2012). Developmental relationship as the active ingredient: A unifying working
hypothesis of ―what works‖ across intervention settings. American Journal of
Orthopsychiatry, 82, 157-166.
Schwartz, S., Rhodes, J., Spencer, R., & Grossman, J. (in press). American Journal of Community
Psychology. Youth initiated mentoring: Investigating a new approach to working with vulnerable
adolescents. American Journal of Community Psychology.
Tolan, P., Henry, D., Schoeny, M., & Bass, A. (2008). Mentoring interventions to affect juvenile
delinquency and associated problems. Campbell Systematic Reviews, 16. Retrieved from
http://www.campbellcollaboration.org/lib/download/238/
Wheeler, M. E., Keller, T. E., & DuBois, D. L. (2010). Review of three recent randomized trials of
school-based mentoring: Making sense of mixed findings. Social Policy Report, 24(3).
Wong, K. (2009). Twilight of the Neandertals. Scientific American, 301(2), 32–37.