Presentation: "Measuring Sustainment of Multiple EBPs Fiscally Mandated in Children's Mental Health Services: Knowledge Exchange on Evidence-based Practice Sustainment (4KEEPS) Study" Speaker: Lauren Brookman-Frazee Southern California Regional Dissemination, Implementation and Improvement Science Webinar Series June 1, 2016 Sponsored by UCLA CTSI

1. Measuring Sustainment of Multiple EBPs
Fiscally Mandated in Children’s Mental
Health Services: Knowledge Exchange on
Evidence-Based Practice Sustainment
(4KEEPS) Study
Lauren Brookman-Frazee
Southern California Regional Dissemination,
Implementation and Improvement Science Webinar Series
June 1, 2016

2. Objectives
1. Provide an overview of system driven EBP
implementations and community context and the
4KEEPS Study.
2. Describe study methods and initial findings:
• Characterizing sustainment outcomes
• Characterizing potential inner context determinants of
sustainment outcomes

3. Public Mental Health Care System (Local, State)
Organizations contracted to provide MH Care
Therapists providing direct services to consumers
Exploration
Phase
Preparation
Phase
Implementation
Phase
Sustainment
Phase
Phases and Levels of EBP
Implementation
Adapted from Aarons, Hurlburt & Horwitz, 2011

4. Public Mental Health Care System (Local, State)
Organizations contracted to provide MH Care
Therapists providing direct services to consumers
Exploration
Phase
Preparation
Phase
Implementation
Phase
Sustainment
Phase
Sustained Delivery of Multiple EBPs in
System-Driven Implementation
Efforts
Adapted from Aarons, Hurlburt & Horwitz, 2011

5. Statewide Reform: California’s
Mental Health Services Act
Prevention and
Early Intervention
(PEI)

6. Los Angeles County
• Nation’s largest county
mental health
department
• 75 directly operated sites
• 288 contracted agencies
(120 with child MH
services)
• 8 Service Provision Areas
range from rural to inner
city communities
• Serves an ethnically
diverse, disadvantaged
population
Hispanic/Latino
Asian/Pacific Islander
African American

7. Applying Aarons et al (2011) Model to LACDMH PEI Timeline

8. Applying Aarons et al (2011) Model to LACDMH PEI Timeline

9. Characteristics of Practices
Practice Age Range
(years)
Target Problem(s)
Triple P 2-12 conduct
CPP 0-6 trauma; attachment
MAP 0-21 anxiety; trauma; depression; conduct
TF-CBT 3-18 trauma
CBITS 11-15 trauma
Seeking Safety 13-18 trauma; substance use

10. Applying Aarons et al (2011) Model to LACDMH PEI Timeline

11. Applying Aarons et al (2011) Model to LACDMH PEI Timeline

12. 4KEEPS Study Aims
• A neutral, observational study investigating the
sustainment of original 6 practices with
implementation support.
• Aim 1: Characterize sustainment outcomes
• EBP Concordant Care
• Volume/penetration of each practice over time
• Aim 2: Use mixed methods to characterize inner
context factors and early implementation conditions
that potentially predict EBP sustainment.
• Aim 3: Identify inner context and early implementation
conditions that determine sustainment outcomes
Funded by NIMH Grant # R01 MH100134
MPI Anna Lau and Lauren Brookman-Frazee

13. Multiple Sources of Data
LACDMH
Administrative
Claims Data
LACDMH Site
Visit
Documents
Online Surveys
of Program
Leaders and
Therapists
Semi-
Structured
Interviews
Session
Recordings
Sustainment outcomes and potential inner context determinants of outcomes

14. Today
2014 2015 2016 2017 2018
4KEEPS Study
launch
1/1/2014
Online survey 1
launch
3/1/2015
In-depth sample
launch
5/1/2015
Online survey 1
supplement
2/1/2016
In-depth sample
wrap-up
Spring 2017
Online survey 2
launch
Spring 2017
4KEEPS StudyTimeline
4KEEPS Events Timeline

15. Sustainment outcomes
• # & % agencies continuing to be reimbursed for the practice
• # & % Therapists continuing to claim to the practice
• # & %Unique clients served by the practice
• # $ % Units of services being provided within each practice
Practice
Volume/Penetration
• Degree to which a therapist’s practice resembles the essential
strategies one would expect within an evidence-based
protocol for a given problem focus.
EBP Concordant Care

16. Practice Volume/Penetration
94
agencies
8,514
providers
87,100 children
2,331,000
psychotherapy claims
3,014,353 total claims
2009 2015

17. Cumulative # of Claims Per
Practice
• MAP = 905,395
• TF-CBT = 662,184
• Seeking Safety = 515,208
• Triple P = 140,147
• CPP = 105,231
• CBITS = 2835

18. Volume Over Time
Initial increases in raw volume
for most practices in the first
quarters with peak and leveling
off after the initial few years
Highlight the rapid impact of a
fiscal policy change restricting
reimbursement to specific
practices and training in these
practices

19. Gross Penetration Over Time
During the initial ramp-up
period, some practices
ramped up more quickly than
others.
Reordering and stabilization
after 2 years
MAP, TF-CBT, SS > Triple P,
CPP, CBITS

20. Characteristics of Practices
Higher Penetration (MAP, TF-
CBT, CPP)
• Apply to a broad age range of
clients
• Can be delivered in multiple
settings
• Addresses common
presenting problem -trauma
• MAP also covers a range of
presenting problems
• Train-the-trainer capacity
(MAP, SS)
• Minimal training required for
billing (SS)
Lower Penetration (Triple P,
CPP, CBITS)
• Apply to a narrower age
range of clients (CPP, CBITS)
• Restrictions on
settings/format
• Group (CBITS, Triple P)
• School setting (CBITS)
• Caregiver directed (CPP,
Triple P)
• Additional requirements to
deliver such as MOAs
between programs and
schools (CBITS)

21. Sustainment outcomes
• # & % agencies continuing to be reimbursed for the practice
• # & % Therapists continuing to claim to the practice
• # & %Unique clients served by the practice
• # $ % Units of services being provided within each practice
Practice
Volume/Penetration
• Degree to which a therapist’s practice resembles the essential
strategies one would expect within an evidence-based
protocol for a given problem focus.
EBP Concordant Care

22. Measuring Sustainment: EBP
Concordant Care
Addresses concerns about the feasibility and
appropriateness of using traditional fidelity instruments to
asses ongoing delivery of multiple EBPs

23. EBP Concordant Care Assessment
(ECCA) Development Process
Reviewed existing practice inventories.
Adapted/selected/generated items (strategies).
Collected data from practice experts about how
essential/interfering strategies are to their practice.
Delphi rating system used to determine item selection for
therapist-report ECCA (alpha version).
Collected data from large sample of therapists using ECCA
(alpha version).
Examine the properties of the therapist-report ECCA (alpha
version).
Validate therapist-report ECCA with observational coding
system.
Finalize therapist-report ECCA based on #6 and 7.

24. Current ECCA
Purpose Assesses the extent to which a
therapist delivered individual
psychotherapeutic strategies
considered essential for a given EBP
Target
Online
Versions
Session (therapist report, observer
rated) and Bi-monthly (therapist
report)
Items 38 psychotherapeutic content and
techniques
EBP Targets #Content #Techniques
Conduct 18 9
Trauma 10 5
Anxiety 7 7
Depression 9 6

25. ECCA Data Collection To Date
• 710 therapists from 54 agencies
completed a Bi-Monthly ECCA
Full Sample
• 71 therapists submitted 459
Session ECCAs with corresponding
audio recordings of sessions as
well as Bi-Monthly versions for
each client
Validation
Sub-Sample

26. ECCA Preliminary Findings
High internal consistency for each EBP Target Composites
for Anxiety, Conduct, Depression and Trauma (alphas ranged
from .86 to .95; M=.90).
Discriminant validity of the scales supported by significant
differences in EBP Target composites by EBP delivered
 When therapists delivered an EBP that targeted Conduct or Trauma,
their ratings on the content items from the corresponding
composite were significantly higher than for other EBP targets
Construct Validity supported by Item Response Theory
Analyses indicating that items with high item difficulty
appeared to include strategies that reflect high integrity EBP
concordant care (Ignoring/Differential Reinforcement of
Other Behaviors > Praise)

27. ECCA Preliminary Findings
Continued
Concordance between Session and Bi-monthly
version supporting by significant and large
correlations between the average Session ECCA
with the Bi-Monthly version on the EBP Target
composites (range= .62 to .71; M=.68).
Concordance between therapist report and direct
observation in process
 Early data based on a small subset of therapist reports
with corresponding observer ratings indicate greater
concordance for strategies that are part of the practice
type being delivered.

28. Next Steps
1) Finish validation of therapist-report ECCA with
observational coding system
1) Projected sample of 1080 sessions with therapist report
and observer ratings
2) Examine inner context factors associated with
ECCA composite scores.
3) Refine ECCA instrumentation based on
concordance analysis and end user feedback on
utility.

29. Potential Inner Context
Determinants of Sustainment
• Changes in state regulatory requirements
• Changes in implementation strategies used
• Adoption, de-adoption of EBPs over time
System Level
• Implementation support
• Organizational climate
• Early implementation condition
Organizational
Level
• Therapist attitudes: perceptions of
effectiveness and fit
• Therapist clinical adaptations to practices
Therapist Level

30. Therapist and Practice Characteristics
as Facilitators of Multiple EBP
Implementation
Practices Therapists
• The PEI Context and fiscal
mandate for EBP reform
presents a unique context
• EBP vs. Practice as Usual
• EBP1 vs. EBP2 vs. EBP3…
• Allows for parsing the
variance in attitudes
associated with
• Therapists and therapist
characteristics
• Practices and practice
characteristics

31. Demographics M (SD) or %
Age 37.00 (9.28)
Gender (female) 88%
Hispanic 43%
Race
White 50%
Asian/ Pacific Islander 11%
African American 7%
American Indian/Alaska Native 1%
Multiracial 8%
Other 23%
Deliver Services in >1 Language
Spanish 47%
Other Language 10%
Survey: Therapist Characteristics
(n=790)
Training Background M (SD) or %
Years Practiced as Therapist 7.21 (6.16)
Years Worked at Current Agency 5.12 (4.56)
Current # Clients 14.65 (10.48)
MH Discipline
MFT 55%
Social Work 30%
Clinical Psychology 11%
Counseling 2%
School Psychology <1%
Other 2%

32. Measuring Therapist Attitudes
Practice-Specific Attitudes General Attitudes towards EBP
Perceived Characteristics of Intervention Scale, Cook et
al., 2014
• Relative Advantage
• Complexity
• Compatibility
• Potential for Reinvention
• Total
Evidence-Based Practice Attitudes Scale, Aarons,
2004
• Openness
• Divergence

33. Attitudes Differ by Practice
F b
Intercept (grand mean) 9667.23*** 3.29***
TF-CBT 183.10 *** .45***
CPP 29.73*** .28***
Triple P 11.77** .16**
MAP 5.28* .08*
SS 11.47** -.12**
CBITS 117.16*** -.84***
*p<.05; **p<.01; ***p<.001

34. Practice Characteristics
Practice
PCIS
Alphas
Narrow Age
Range
Consultation
Required
Prescribed
Session
Content/ Order
CBITS (N = 65) .96
  
CPP (N = 140) .94
 
MAP (N = 527) .92 
SS (N = 491) .93

TF-CBT (N = 582) .93
 
Triple P (N = 184) .94


35. Therapist Attitudes Differ by
Practice Characteristics
1
2
3
4
5
Prescribed Session Content/
Order
Consultation Required Narrow Age Range
PCISITEMMEAN
PRACTICE CHARACTERISTIC
Yes
No
* **
* = p < .001
Very Great
Extent
Not at All

36. • Use sequential QUANqual
design to understand:
• The types of adaptations that
therapists make in community
mental health settings.
• Which therapist
characteristics predict types of
adaptations (fidelity
consistent vs. inconsistent).
• The reasons therapists make
adaptations.
Therapist Adaptations
Fidelity
Consistent
Modify
presentation
Integrate
supplemental
content
Lengthen/exten
d Pacing
Fidelity Inconsistent
Remove/skip
components
Shorten/condense
pacing
Adjust order of
sessions/components

37. Characterizing Adaptations
1
2
3
4
5
MeanExtensiveness
Fidelity Consistent
CBITS CPP SS TFCBT TP
1
2
3
4
5
Fidelity Inconsistent

38. Types of Adaptations
• Therapists reported more
fidelity consistent adaptations
(EMM= 9.30; SE = 1.79) than
fidelity inconsistent (EMM=
5.88; SE = 1.79) adaptations
• F(1, 1,075.59) = 1,332.00, p <
.001
• Therapist-reported attitudes
towards a practice:
• Did not predict fidelity
consistent adaptations B = -
.02, t = -1.47, p = .14
• Did predict fidelity
inconsistent adaptations
• B = -.07, t = -5.67, p < .001 0
1
2
3
4
5
6
7
8
9
10
Fidelity Consistent Fidelity Inconsistent
MeanComposite
CBITS CPP SS TFCBT TP

39. Reasons for Adaptations
Culture
Crises
Adjust Order
Extend Pacing
Modify Presentation
Omit
Developmental
Level I think the length of treatment is part of that too,
especially with the Hispanic families. Like you can’t just
jump right in; you need to spend a little more time
building rapport.
Some of the examples that they have there I think, you
know, might not necessarily fit like this particular culture
so I’ll find something that they could relate to a little bit
better, a situation they can relate to a little bit better.
I did work with one client that was developmentally
delayed. So for some of the cognitive behavioral piece it
was just really difficult to understand…I did my best to
teach it, but I didn’t feel like I focused a lot on it because
dude wasn’t really understanding it.
Oh, we’ll still go back to some of the relaxation
components, or we’ll kind of focus on what we can do to
manage our anxiety about it and whatnot.
You know, with people that are developmentally
delayed or whether there needs to be more repetition,
I’ll repeat it more.
I think it’s just different working with a child that young.
So we do a lot of art. So instead of like a trauma
narrative, we’ve been doing a lot of art on the
chalkboard.
You do have those clients that maybe might become
homeless, or maybe parents are having a really hard time,
maybe with other kids, older kids. So you might have a lot
of crises that might surface throughout treatment, and that
might postpone certain parts of it

40. Summary of Preliminary Findings
The scope and size of the PEI transformation provides a critical
opportunity to examine multiple EBP implementation in a large
and diverse natural laboratory
Sustainment varies by practice and over time
Inner context factors are critical to examine within EBP
implementation efforts
• Therapist attitudes vary across practices.
• Therapists report more extensive fidelity consistent adaptations than fidelity
inconsistent adaptations

41. Funding Source: NIMH R01MH100134