The document discusses conducting a rigorous random assignment evaluation of Youth Villages' Transitional Living program. It notes that while random assignment is the gold standard for evaluation, it also presents ethical issues in denying some youth access to the program. It describes the recruitment process, challenges in meeting enrollment goals, efforts to monitor program fidelity, costs to the provider, and the lengthy timeline from beginning the study to receiving preliminary outcome results. The evaluation aims to determine the program's impacts on outcomes like housing stability, education, employment, and crime reduction.

1. The Need for Rigorous Evaluation of
Interventions to Improve the
Transition to Adulthood for Youth in
State Care
Mark E. Courtney
School of Social Service Administration and Chapin Hall
University of Chicago

2. My Purpose Today
Present recent research on foster youths’
transitions to adulthood
Describe the focus of social policy on this
population
Summarize the weak evidence regarding the
effectiveness of interventions for this population

3. How do foster youth fare during the
transition?

4. Midwest Study Design and Sample
Largest prospective study of foster youth making the transition to
adulthood since the Foster Care Independence Act of 1999
Collaboration between state child welfare agencies and the research
team
Foster youth in Iowa, Wisconsin and Illinois who:
 Were still in care at age 17
 Had entered care before their 16th birthday
 Had been placed in care because they were abused, neglected or
dependent
 Not originally placed because of delinquency
Data from in-person interviews (structured and in-depth qualitative) and
government program administrative data

5. Study Design and Sample (continued)
Wave Year Number Response Age at
Interviewed Rate interview
1 ’02 – ’03 732 96% 17 – 18
2 ‘04 603 82% 19
3 ‘06 591 81% 21
4 ‘08 602 82% 23-24
5 ’10 – ’11 596 83% 26

6. Young Women’s Educational Attainment

7. Young Men’s Educational Attainment

8. Young Women’s Educational Enrollment

9. Young Men’s Educational Enrollment

10. Young Men’s and Young Women’s Employment
72% employed during year; mean earnings among employed = $13,989

11. Family Formation Among Young Women
19% of women with children have a nonresident child

12. Family Formation Among Young Men
66% of men with children have a nonresident child

13. Young Women’s Criminal Justice System
Involvement

14. Young Men’s Criminal Justice System
Involvement

15. Summary of What We Know About Early
Adult Outcomes Post Chafee
Outcomes are relatively poor across a variety of domains
Trends are generally problematic:
 Declining engagement in education, though some are still in school
 Gradually increasing but poor engagement in the workforce
 Many non-resident children
 Troubling levels of justice system involvement continuing through mid
20s
 Functioning in other domains (e.g., mental and behavioral health, risk
behaviors, victimization) is also poor
Outcomes vary by gender; males fare worse
Despite a sobering picture overall, many young people leaving
the care of the state do well

16. 16
US Social Policy and the Transition to
Adulthood for Foster Youth

17. U.S. Demographic, Developmental, and
Policy Context
The transition to adulthood in the U.S. is taking
longer
 Markers of the transition are happening later; half of
young people between 18-24 live with a parent
 $38k in direct support between 18-34
Developmental psychologists describe a new
period of “emerging adulthood”
Yet, U.S. policy provides little support for young
adults

18. U.S. Policy on Foster Youth in Transition
Research from 1990s continues to show poor
outcomes
1999 Foster Care Independence Act
 $140 million per year allocated to states
 Funds a broad range of services
 Up to 30% of funds can be used for room and board
 Allows states to extend Medicaid to foster youth through age 21
 Amendment to law allows appropriation up to $60 million per
year to fund education/training vouchers for up to $5000 per year
through age 23
 Creates outcome reporting requirements and devotes 1.5% of
funds to rigorous evaluation of promising programs

19. A Brave New World: The Fostering Connections
to Success Act of 2008
Among its provisions, the law:
 Extends Title IV-E funding (including guardianship and
adoption subsidies), at state option, to age 21
 Youth must be 1) completing high school or an
equivalency program; 2) enrolled in post-secondary or
vocational school; 3) participating in a program or
activity designed to promote, or remove barriers to,
employment; 4) employed for at least 80 hours per
month; or 5) incapable of doing any of these activities
due to a medical condition
 Existing IV-E protections remain, including ongoing
court oversight of state foster care provision

20. But…Evidence of What Works is Lacking
Cochrane collaboration review of evaluation research on IL programs
(Montgomery et al, 2006) found no rigorous studies: “Further
research incorporating randomized designs is both feasible and
necessary”
Recent ACF-funded randomized evaluations:
 No impact of life skills training, tutoring/mentoring, and employment support
 Massachusetts Adolescent Outreach had some positive effects, but those
appear to be mediated by the program’s impact on youth remaining in care past
age 18.
The bottom line: Too many programs are poorly targeted, have poorly
developed logic models, and are not intensive enough to influence
outcomes for youth making the transition to adulthood from foster
care.
WE NEED RIGOROUS EVALUATION RESEARCH!!!

21. Evaluation of Youth Villages’
Transitional Living Program
John Martinez
Deputy Director, Health and Barriers to Employment Policy Area

22. Overview of Session
 Introduction to MDRC and evaluation
 Who is in the study
 Evaluation Status
2

23. Who is conducting the evaluation?
 MDRC, a non-profit, non-partisan education
and social policy research organization and
intermediary
 Based in New York City with a regional office in
Oakland, CA
 Dedicated to learning what works best to
improve the lives of low-income families
 Nearly 40 years of experience evaluating social
policy programs
3

24. Who is funding the evaluation?
 The Edna McConnell Clark Foundation
 The Annie E Casey Foundation
 The Bill and Melinda Gates Foundation
4

25. Components of the Evaluation (I)
 Impacts: To what extent does TL improve outcomes for
youth? Which approaches are most effective for whom?
 On what outcomes (e.g. housing stability, educational
outcomes, labor market success, reduced crime)
 Over what period of time (e.g. length of follow-up)
 To what magnitude?
 For what subgroups?
Use Random Assignment Research Design
Data sources: public records data, baseline forms, and surveys
5

26. Components of the Evaluation (II)
 Program implementation: What services are provided?
How are they delivered? What challenges are encountered?
Are the challenges related to serving youth aging out of state
care?
 Measuring enrollment and participation rates
 Measuring implementation fidelity
 Measuring the contrast
Data sources: surveys, field research, YV MIS, in-depth
interviews with youth
 Costs and Benefits: What are the costs? Do benefits
outweigh the costs?
6

27. Impact Analysis Design
 Evaluation uses random assignment, the most
reliable way to assess what difference a program
makes
 Youth eligible for TL were assigned, by chance, to
one of two groups:
 TL Program Group: receives TL services
 Community Services(CS) Group (Control Group): not eligible
to receive TL services for up to five years
 MDRC will follow both groups for at least three
years (and possibly longer, depending on funding)
7

28. Random Assignment
Participants meet program criteria
Participants give consent
Baseline data collected
Random
Assignment
Program group Control group
Enroll in program Receive other services
in the community
8

29. Why Random Assignment?
 Gold standard of research as it is the most reliable way to
measure impacts:
 Ensures motivation levels and personal characteristics of youth in
program and control groups are same on average at beginning of
program
 Youth in the control group illustrate what would have happened if
not for the program
 Any subsequent difference in outcomes can be attributed to the
program with the highest confidence
 Widely used in public service settings
 Endorsed by OMB, DOE and other federal agencies
 Fair and equitable way to determine who receives the program
9

30. Who is in the study?

31. What is the baseline information form?
 Youth completed a baseline survey prior to random
assignment
 Baseline data provides a “picture” of these youth at
study entry
 Multiple uses
 Allows researchers to assess whether random assignment
worked
 Could be used in impact analysis models
 Can also be used to determine whether TL works differently
for different types of youth
 Presenting data today on 1,225 study participants

32. Did RA work?
 Goal: two research groups in which the only
difference is that one group was eligible to receive TL
and one was not
 Baseline data can help support that the two groups
were equivalent
 Data suggests this was the case: of 22 variables, only
2 had a SS difference between the 2 groups (and we
would expect that to happen by chance)

33. Demographics
Characteristic Full Sample (%)
Gender
Male 52
Female 48
Race
Hispanic 5
White/non-Hispanic 51
Black/non-Hispanic 38
Other/non-Hispanic 6

34. Age at random assignment
Age at RA
9%
20% 18
19
20-24
71%

35. Age at 1st custody placement
1%
6%
6%
LT 1
1-5
23% 6-10
11-14
64% 15-18

36. Contact with biological parent
Contact with biological Contact with biological
mother father
Every day Every day
At least 16% At least
25% 1X/wk 1X/wk
43% At least At least
12%
8% 1X/mo 53% 1X/mo
8%
LT 1X/mo LT 1X/mo
11%
8% 16%
Never Never

37. Other characteristics
Characteristic Full Sample (%)
Contact with any other relative at least 1X/mo 88
Pregnant at baseline 4
Has any children 17
Enrolled in school 54
Ever repeated a grade or held back 43
Ever suspended from school 81
Ever arrested 64

38. Current Status
 Random assignment complete
 1322 youth randomly assigned within two years
 Most field work associated with the implementation
study is complete
 Survey fielding is ongoing
 Very high response rates (about 85 percent)

39. What’s Next
 Survey fielding will continue (survey firm should
wrap up winter/spring 2014)
 Implementation report slated for publication in
January 2014
 Impact report slated for publication in
spring/summer 2015

40. Thank You
20
John Martinez
john.martinez@mdrc.org
212-340-8690
www.mdrc.org

41. The promises and perils of
random assignment
evaluations – a provider’s
perspective
Presentation to First Focus/SPARC
March 19, 2013
All contents ©2011 by Youth Villages, Inc. with all rights reserved

42. Discussion Topics
• Embarking on a random assignment evaluation
• Confronting the ethical issues
• Recruiting study subjects
• Monitoring program fidelity
• Controlling study costs
• Awaiting results

43. Why do a random assignment evaluation?
PRO CON
• Provides a benchmark • Frightening
• Value in having • Time consuming
independent evaluation
• Referral sources/ service
• Exciting purchasers don’t require it
• Opportunity for program • Youth Villages already has
improvement an on-going outcome
evaluation process
• Push toward evidence-
based practices from govt.
and foundation funders
• Meets organizational goal
of increasing use of
evidence-based services

44. Confronting the Ethical Issues of Random Assignment
Forty percent of youth in the
study are denied entry into the
TL Program.
How do we justify that?

45. Recruiting Study Subjects
The expectation:
The reality:

46. It wasn’t as easy as we thought!
1,300
1,200
1,100
Original goal – 1,600
1,000
Adjusted goal – 1,300
900
800
700
600
500
400
300
200
100
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Month
Goal Actual

47. Monitoring Program Fidelity
In some ways, this has been the easy one!
• Program Model Adherence Reviews
• Balanced Scorecard
Is that enough?

48. Controlling Study Costs
The evaluation is funded by the Edna
McConnell Clark Foundation and the
Bill and Melinda Gates Foundation
What are the provider’s costs?
• Study coordinator (full time)
• Leadership staff time
• Assessors (to determine eligibility and
appropriateness for program)
• Training and travel

49. Awaiting Study Results
Planning Begins – July 2008
Study Recruitment Begins – October 2010
One Year Follow-up Begins – November
2011
Study Recruitment Ends – October 2012
One Year Follow-up Completed – January 2014
Six
Years! Preliminary Report on One Year Outcomes –
July 2014

50. Lessons learned?
• Go in with eyes wide open
• Take time to prepare
• Be ready for the unexpected