Understanding the Signs of Safety Theory of Change and comparing outcomes for children pre and post Signs of Safety
1. Understanding the Signs of Safety Theory of Change
and comparing outcomes for children pre and post
Signs of Safety
Dr Mary Salveron, Professor Leah Bromfield and Professor Fiona Arney
British Association for the Prevention and Study of Child Abuse and Neglect
(BASPCAN)
12-15th April 2015, University of Edinburgh
2. Overview
• Signs of Safety Post-Doctoral Research
Program
• Evaluation approaches
• Signs of Safety Theory of Change
• Signs of Safety Administrative Data Study
• Limitations and Discussion
3. Signs of Safety Post-Doctoral
Research Program
• 3 year research collaboration between ACCP
and WA Department for Child Protection and
Family Support
• To examine the impact and implementation of
the Signs of Safety child protection practice
model/approach on children, parents and
practitioners
4. Signs of Safety Post-Doctoral
Research Program
2013
• Theory of Change
• Implementation
• Children’s Study
2014
• Practitioner Study
• Administrative Data
Study
• Parent Study
2015
• Aboriginal Staff and
Practitioner Study
• Triangulation Paper
5. Signs of Safety
• Home-grown, locally developed approach to child protection
practice
• Origins in Solution-focused brief therapy
• Used in 12 countries and spreading across the world
• Designed to provide child protection practitioners with
straight forward methods and tools to tackle child abuse in
partnership with parents and professionals/service providers
• Limited evidence base
6. Administrative Data Study
• Use Departmental data from ASSIST (known as
‘administrative data’)
• Examine the impact of Signs of Safety on outcomes for
children
• Compare these to outcomes for children pre Signs of
Safety
• Essentially we want to know if Signs of Safety is making
a difference at the highest level (substantiations, entry
into care) (Department KPIs)
7. Hypotheses
Compared to children pre Signs of Safety, there will be:
1. A decrease in the number of children coming into care of
Department
2a. A decrease in proportion of children with substantiations
following an unsubstantiated notification
2b. A decrease in proportion of children with substantiations
following a substantiated notification
3. An increase in proportion of Safety and Wellbeing
Assessments (SWA) outcomes proceeding to family support
or child centred family support
8. Limitations
• Retrospective study
• Use of administrative data that was not designed for
evaluative purposes
• Outcomes are multiply determined and influenced by:
– Changing policy and practice context
– Evolving Signs of Safety child protection practice framework
– Family and other service factors
– Demands on the system and resourcing
9. Methodology
• Retrospective (follow forward) historical
control using Departmental administrative
data from 3 consecutive years pre-Signs of
Safety implementation (2005-2007) and 3
consecutive years post-Signs of Safety
implementation (2011-2013)
• First notifications
10. Cohort 1 – Pre SoS
(2005-2007)
Cohort 2 – Post SoS (2011-
2013)
Number of Children
(Pre-birth to 18 years)
3,011 7,562
Gender 1425 boys, 1565 girls 3383 boys, 3813 girls
Aboriginality 854 Aboriginal, 1738 non-
Aboriginal
419 not recorded
1602 Aboriginal, 1928 non-
Aboriginal
4032 not recorded
Notifications 4,340 12,003
Investigations/Safety and
Wellbeing Assessments
2,391 (55% of notifications) 8,050 (67% of notifications)
Substantiations 962 (40% of investigations) 2,160 (27% of investigations)
Sample and Context
11. Findings
Hypothesis 1: There will be decrease in the number of children coming into
care of Department
There was an increase in the number and rate of of children coming into care
of Department at a population level
Cohort 1 (pre-SoS) Cohort 2 (post-SoS)
All children 2.06 per 1000 children in
2005 (347 children)
3.89 per 1000 children in
2011 (647 children)
Aboriginal children 0.3 per 1000 children in
2005 (147 children)
0.49 per 1000 children in
2011 (268 children)
12. Findings
Hypothesis 1: There will be decrease in the number of children coming into
care of Department
There was a decrease in the proportion of children who were notified who
subsequently entered care
Cohort 1 (Percentage of children
who were first notified and came
into care)
Cohort 2 (Percentage of children
who were first notified and came
into care)
All children 3011 children with a notification
367 who were first notified and
came into care (12.1%)
7562 children with a notification
647 who were first notified and
came into care (8.5%)
Aboriginal children 3011 children with a notification
147 Aboriginal children who were
first notified and came into care
(4.8%)
7562 children with a notification
268 Aboriginal children who were
first notified and came into care
(3.5%)
13. Findings
Hypothesis 2a: There will be a decrease in proportion of children with
substantiations following an unsubstantiated notification
There was an increase in the proportion of children with substantiations
following an unsubstantiated notification between Cohort 1 and Cohort 2
Number of children with
substantiations following an
unsubstantiated notification
Number of children with
substantiations following an
unsubstantiated notification
All children 1,271 children with unsubstantiated
notification
99 children (7.8%) had a
substantiated notification
4,578 children with unsubstantiated
notification
523 children (11.4%) had a
substantiated notification
Aboriginal children 370 children with unsubstantiated
notification
44 children (11.9%) had a
substantiated notification
976 children with unsubstantiated
notification
181 children (18.5%) had a
substantiated notification
14. Findings
Proportion of all children with substantiations following an unubstantiated
notification by primary harm type
Primary Harm Type Cohort 1 Cohort 2
Emotional Abuse 5 out of 105
children (4.8%)
66 out of 1053
children (6.3%)
Neglect 20 out of 296
children (6.8%)
114 out of 1067
children (10.7%)
Physical Abuse 18 out of 492
children (3.7%)
19 out of 861
children (2.2%)
Sexual Abuse 9 out of 441
children (2%)
35 out of 1324
children (2.6%)
15. Findings
Proportion of Aboriginal children with substantiations following a
unsubstantiated notification by primary harm type
Primary Harm Type Cohort 1 Cohort 2
Emotional Abuse 5 out of 44 children
(11.4%)
9 out of 142
children (13.4%)
Neglect 7 out of 116
children (6%)
46 out of 193
children (23.8%)
Physical Abuse 5 out of 119
children (4.2%)
2 out of 85 children
(2.4%)
Sexual Abuse 3 out of 109
children (2.8%)
4 out of 126
children (3.2%)
16. Findings
Hypothesis 2b: There will be a decrease in proportion of children with
substantiations following a substantiated notification
There was an increase in the proportion of children with substantiations
following an substantiated notification between Cohort 1 and Cohort 2
Number of children with
substantiations following an
unsubstantiated notification
Number of children with
substantiations following an
unsubstantiated notification
All children 854 children with substantiated
notification
100 children (11.7%) had a
substantiated notification
1,779 children with substantiated
notification
319 children (17.9%) had a
substantiated notification
Aboriginal children 304 children with substantiated
notification
52 children (17.1%) had a
substantiated notification
501 children with substantiated
notification
117 children (23.4%) had a
substantiated notification
17. Findings
Proportion of all children with substantiations following a substantiated
notification by primary harm type
Primary Harm Type Cohort 1 Cohort 2
Emotional Abuse 13 out of 112
children (11.6%)
59 out of 529
children (11.2%)
Neglect 30 out of 276
children (10.9%)
82 out of 575
children (14.3%)
Physical Abuse 15 out of 273
children (5.5%)
17 out of 315
children (5.4%)
Sexual Abuse 4 out of 223
children (1.8%)
18 out of 379
children (4.7%)
18. Findings
Proportion of Aboriginal children with substantiations following an
substantiated notification by primary harm type
Primary Harm Type Cohort 1 Cohort 2
Emotional Abuse 5 out of 49 children
(10.2%)
19 out of 178
children (10.7%)
Neglect 17 out of 115
children (14.8%)
39 out of 217
children (18%)
Physical Abuse 9 out of 89 children
(10.1%)
6 out of 69 children
(8.7%)
Sexual Abuse 3 out of 64 children
(4.7%)
6 out of 46 children
(13%)
19. Findings
Hypothesis 3: There will be an increase in proportion of Safety and
Wellbeing Assessments (SWA) outcomes proceeding to family support
or child centred family support
Hypothesis cannot be answered as no comparable fields for Cohort 1 and
Cohort 2.
20. Limitations and understanding the
context
• Dramatic increase in children coming into
contact with the Department
• Policy and practice changes – Legislation
• Entry in to out-of-home care – Early
intervention services, placement prevention
21. OTHER IMPORTANT
PEOPLE (Carers, Relatives,
Friends, Non-Government
Practitioners/Staff) ARE
ENGAGED
Figure 1: Signs of Safety Results Logic
Children and young people are SAFE (long term sustained safety)
Children and young people are SAFE from imminent/immediate
harm
3
Future-focused | Working relationships based on partnership | SIGNS OF SAFETY PRINCIPLES | Critical Inquiry | Recognising what is working well
Signs of Safety Tools for
Children and Young People
Signs of Safety
Mapping
(incl Three Columns)
Creating Safety
Networks
Workforce Wellbeing/Resilience
CHILDREN AND YOUNG PEOPLE
ARE ENGAGED
2
PARENTS ARE ENGAGED
1
SHARED
UNDERSTANDING
SAFETY
PLAN
CASE PLAN SHARED
AGREEMENT
Children and Community Services Act 2004 (WA), Western Australia Child Protection Policy
Context
Children participateNoticing what works Families are socially connected
SIGNS OF SAFETY DISCIPLINESSigns of Safety IMPLEMENTATION
(e.g. training, coaching, leadership, appreciative inquiry)
Signs of Safety
practiced with fidelity
Children’s
perspective acts as
circuit breaker
4
22. Key Messages
• Importance of articulated theory of change – What is Signs of
Safety intended to do?
• Limitations of administrative data as evidence in a changing
context
• Problems related to the use of blunt indicators and
directionality of hypotheses (surveillance effects, enhanced
decision making)
• Signs of Safety fidelity, implementation and dose
• Best outcomes for children may not be measured by
administrative data
23. Dr Mary Salveron
Post-Doctoral Fellow (Signs of Safety)
Australian Centre for Child Protection
University of South Australia
+61 8 83022980 / 0400 863 784
mary.salveron@unisa.edu.au
www.unisa.edu.au/childprotection
Hinweis der Redaktion
FIONA
Many of you may know that Signs of Safety is a child protection practice framework/model implemented across Western Australia. Now in its seventh year of implementation, Signs of Safety was home-grown and locally developed back in the late 90s by Andrew Turnell and Steve Edwards but wasn’t taken up by the Department until 2008 with its origins based on solution-focused brief therapy and narrative therapy. To date SoS is utilised in over 12 different countries and over 100 jurisdictios across the world.
SoS is designed to provide practitioners with straightforward methods to tackle child abuse in partnership with parents and professionals and service providers.
While there is limited evidence of how it works and what the outcomes are for children, families and practitioners, there appears to be some promising elements to the approach.
I will go through in more detail the implementation of Signs of Safety within the Department in the next presentation and the strategies they used utilising Roger’s diffusion of innovation theory and Fixsen’s implementation stages and drivers model.
Evaluation best practice is pre-test/post-test with a comparison group
Major Limitation: Historical Control Design is Retrospective
Means we are limited to the information already collected across the whole time period (pre and post ASSIST)
Linked to KPIs of the Department
FIONA
FIONA
Aboriginal children
147/485829 x 1000 = 0.3
268/544493 x 1000 = 0.49
Aboriginal children
147/3011 x 100 =
268/7562 x 100 =
FINDINGS FOR ABORIGINAL CHILDREN
The patterns for Aboriginal children were similar to the entire Cohorts, however the rates for neglect were much higher.
For all Aboriginal children with an unsubstantiated notification (370 children in Cohort 1 and 976 children in Cohort 2), 11.9% of children in Cohort 1 (44 children) and 18.5% of children in Cohort 2 (181 children) had a subsequent notification.
FINDINGS FOR ABORIGINAL CHILDREN
The patterns for Aboriginal children were similar to the entire Cohorts, however the rates for neglect were much higher.
For all Aboriginal children with an unsubstantiated notification (370 children in Cohort 1 and 976 children in Cohort 2), 11.9% of children in Cohort 1 (44 children) and 18.5% of children in Cohort 2 (181 children) had a subsequent notification.
Mandatory Reporting
Changes in data reporting, recording
As represented visually on this slide and read from bottom up), the Master Signs of Safety Results Logic situates Signs of Safety in the Western Australian legislative and practice context. It shows the disciplines and principles (the how) along with the Signs of Safety Mapping (the what), as the core elements of the approach, which are supported by other Signs of Safety tools and techniques. Where these approaches are implemented with fidelity, the theory suggests that they feed into the inter-related and mutually reinforcing casework objectives of developing safety plans and case plans and reaching shared understanding and shared agreement. These casework objectives are influenced by the extent to which parents, children and other important people are ‘engaged’.
Additional logic models (denoted by blue stars numbered one to three) have also been created to further expound the notion of engagement and the way in which this interfaces with the casework objectives for each of these groups. Along the top of this diagram are the outcomes: Children and young people are safe from imminent harm and Children and young people are SAFE. The arrow leading from children and young people are safe from imminent harm back to the inputs depict the cyclical notion of assessment and intervention that may be required in order to shift from short-term to long-term safety.
Finally, in recognition of the broader aspects of Signs of Safety and the anticipated benefits for the workforce , the bottom left hand corner includes the diagram of the key implementation expectations and the dual and mutually reinforcing outcomes of fidelity to the approach and workforce wellbeing/resilience.
The development of the results logic and theory of change that underpinned Signs of Safety provided us with the evaluation components essential for developing the program of research. While we knew there needed to be research that encompassed outcomes for children, parents, practitioners and other important stakeholders, the outcomes (not perceptions) were more clearly defined with the theory of change. The logic gave us an understanding of the relationship between the various outcomes and components such as the relationship between fidelity and workforce wellbeing, fidelity and safety outcomes for children, children and parent engagement, parent engagement and safety outcomes.
In addition, the logic also provided more efficiency so as not to burden participants (so in undertaking the research we could ask participants the right and relevant questions)
Road map for evaluation
Deeper understanding of Signs of Safety practice approach
Shared experience
Went on journey with us, to help us as evaluators
Theory of change significant in articulating assumptions of how a program is meant to work
Helpful for evaluators to understand and provide guidance to what we need to measure
Adds rigour to evaluation
Many spin offs – not just research but collaboration and an analysis of gaps and what needs to change or be better understood
Administrative data system – need to build data system to answer evaluation questions and clinically relevant
Directionality – engagement of families – do they make changes (behaviour change) and report to Department
Achieving positive results in child welfare generally requires a team effort and multiple interventions. Thus, it may never be possible for child welfare and other community service providers to pinpoint any one particular strategy that impact on children entering care. This is why it is essential to take a multiple strategy approach to careful implement practice methods, successfully engage families and have equitable access to services.