Washington Evaluators Brown Bag by Ladel Lewis August 28, 2012 Evaluating one site of a federally funded, longitudinal, multi-site initiative to improve services for children with mental health issues and their families presents numerous challenges. Many individuals, particularly racial minorities, are understandably reluctant to participate or remain in an evaluation concerning such sensitive issues. Further, not all the sites fit neatly into the same “one size fits all” evaluation protocol that must be used at all the sites. Cultural competence is crucial regarding: (1) breaking the barriers to participation; (2) balancing the traditional perspectives of “informed consent” and “confidentiality” with those of the participants; (3) balancing the need for consistent measures in our national study with the local realities of our participants; (4) interpreting and reporting the results. Seeking input from stakeholders at each step of the evaluation helped us recognize and overcome these barriers, and attain equitable recruitment and retention rates among Caucasian and African-American participants. Ladel Lewis received a B.A. in Criminal Justice from the University of Michigan in 2001 and a M.A. in Sociology in 2005 from Western Michigan University. Studying evaluation research under Dr. Chris Coryn at the Evaluation Center, she earned her Ph.D. in Sociology in 2012 at Western Michigan University. She has published journal articles across disciplines such as “User Perceptions of Accessible GPS as a Wayfinding Tool for Travelers with Visual Impairments” published in the AER Journal: Research and Practice in Visual Impairment and Blindness, “White Thugs & Black Bodies: A Comparison of the portrayal of African American Women in Hip-Hop Videos” published in the Hilltop Journal and “Lights, Camera Action: The Portrayal of African American Women In Hip Hop Videos” in the Call & Response Journal.

1. Ladel Lewis, Ph.D.
Carolyn Sullins, Ph.D.
The Kercher Center for Social Research
Western Michigan University

2. 

Comprehensive Community Mental Health
Services for Children and Their Families
Program: “Systems of care.”
incorporates a broad, flexible array of effective services
and supports for a defined, multi-system population
that is organized into a coordinated network… is
culturally and linguistically competent, builds
meaningful partnerships with families and youth at
service delivery, management and policy levels, and
has supportive policy and management infrastructure.
(Pires, Lazear, & Conlan, 2008).

3. 

144 sites have been or are in the process of
being evaluated. Each 1-2 year cohort: +/- 30
sites.
Each SoC has distinct:




geographic location and scope (e.g., statewide,
county wide, city-wide, tribal)
Ages of the youth served
Mental health issues facing the targeted youth
Racial, ethnic, and cultural factors

4. 


Child centered and
family driven
Community based
Culturally
competent

5. 






Presenting issues of the youth
Youth’s level of functioning (strengths and
weaknesses)
Family strengths and barriers
Types of services family and youth are
receiving
Satisfaction with services
Cultural competence of services
Youth and family input into services

6. 



“One size fits all” battery
of questions
HSIRB mandates re
language on consent
forms
Ensuring an adequate
sample size
Ensuring retention in a
mobile population

7. 
History of racist abuse by researchers (E.g., Tuskegee
syphilis study)



Misinterpretation of data, or no access to results
Sensitive or stigmatized topics even more difficult
Families overwhelmed or embarrassed

8. Insurance
n=244
Private Ins
Public Ins
9%
90%
Caucasian:
Ages 7-17
Diagnosed
with a
Severe
Emotional
Disturbance
49.1%
(including White Latino/a: < 5%)
African-American/Multiracial:
50.9%

9. 


Informed consent
Voluntary
participation
Confidentiality
and its exceptions

10. Parents, various social service workers, eval staff
 Reviewed consent forms for clarity
 Gave opinions to HSIRB re child abuse
reporting
 Parent input re: communication among
clinicians, families, and interviewers
 Interpretation of data
 Reporting of results

11. 



We came to their group
Help re local language
Helped us make it more
comfortable for
participants
Info that later helped us
interpret data

12. Does confidentiality mean…
 No interviews in public
places, even if that’s what
participants request?
 Kicking Grandma out of the
room?
 Pretending you don’t see
participant in public?

13. 

Some found it emotionally draining
Too long and redundant



Questionnaires with overlapping questions
Categories of services – national vs. local terms
Keeping in touch with families every 6 months

14. 
“Evaluating System of Care
– not you”


Non-judgmental attitude for
better rapport, retention,
AND accuracy.


Yet up front about sensitive,
personal questions
Balance – we can’t act as
friends or counselors!
If SoC not working, or not
working with all groups of
people, we need to know.

15. 


Offer breaks, gum,
stress balls, etc. to
participants
Coloring books, DVDs
for young kids
Redundant questions
from multiple surveys:
propriety and accuracy
trump methodological
“letter of law.”

16. 




Family address tracking
form
Birthday and holiday cards
with coupons
Incentives for families to
contact us
Annual dinner as a “thank
you”
Results in bimonthly
newsletter, website, and
other venues

18. African Americans
00
Caucasians
19.1%
00
Participants
Partcipants
22.8%
77.2%
NonParticipants
Chi Square (1, N=224) = .466, p = .495
80.9%
NonParticipants

19. African American
Caucasian
00
00
Participants
27.9%
Participants
22.1%
72.10
NonParticipants
77.9%
Chi Square= (1, N= 145) = .665, p = .415
NonParticipants

20. African American
Caucasian
00
00
Participants
36.8%
Participants
30.6%
63.2%
NonParticipants
Chi Square= (1, N=119) = .511, p = .561
69.4%
NonParticipants

21. African American
Caucasian
00
00
Participants
40.4%
Participants
43.2%
59.6%
NonParticipants
Chi Square= (1, N=91) = .071, p = .834
56.8%
NonParticipants

22. African American
Caucasian
00
00
Participants
37.9%
47.8%
52.2%
Participants
NonParticipants
62.1%
Chi Square= (1, N=52) = 1.055 , p = .402
NonParticipants

23. 

Fulfilling national
evaluation
requirements within a
local context is a
constant balancing act.
With enough input
from local stakeholders
at each stage, it can be
achieved!

24. 



Please contact
carolyn.sullins@wmich.edu
Or Ladel_lewis@yahoo.com
For more info, please see also
http://www.wmich.edu/sociology
/kzoowraps.html