Key steps in creating a client outcomes measurement framework for a welfare / human service organisation. Particular focus on homelessness assistance services.

1. Homelessness assistance sector, Australia, November 2010
Mark Planigale
Research & Consultancy
results by design

2. 2
 Performance measurement
 Compliance reporting
 Program logic

3. 3
Performance measurement
 Bigger picture of organisational performance
measurement
 Compliance reporting currently focuses on effort
 Many organisations monitor inputs and client satisfaction
 But are we making a difference?
 National frameworks: National Affordable Housing
Agreement, National Partnership Agreement on
Homelessness
 ‘Manufacturing-based’ performance models need to be
enhanced

4. 4
Program logic Mission Vision
Environment • Problem or need
• Barriers
• Enablers
Inputs Activities Outputs Outcomes Impacts
People and What gets What clients Changes in Changes in
resources done receive clients’ lives society
• Clients • Functions • Client • Short term • Social
• Human • Processes participation • Long term outcomes
resources • Actions benefits • Interim
• Other • Client direct
resources benefits
Objectives Aims

5. 5
Useful information
is produced through
well-planned systems.

6. 6
 Purpose and scope
 Defined outcomes
 Defined measures and tools
 Data collection and storage processes
 Analysis and reporting processes
 Use of outcomes information
 Strategies for sustaining the system

7. 7
Clarify purpose and scope
Why measure outcomes? Questions to consider
 Compliance  Which programs?
 Individual assessment and  Which populations?
planning  What do we mean by
 Oversight outcomes?
 Quality improvement
 Advocacy

8. 8
Define key outcome areas
Outcome: Questions to consider
“... a change or absence of  If the program works
change that results (at least in really well, how are
part) from actions of staff of clients’ lives better?
the organisation”  If things go wrong
 Desired vs. undesired for our clients, where
do we see this in
 Short term, long term, interim
their lives?
 Domains vs. locus of change
 What is our core
business?

9. 9
Example outcomes
1. Person obtains appropriate, affordable long-term
housing
2. Person takes medication more regularly
3. Person has a better understanding of tenancy rights and
responsibilities
4. Person is placed on the OOH Early Housing waiting list
5. Person gains part-time employment
6. Person becomes homeless

10. 10
Select measures and tools
Measure: Questions to consider
“...an observable, measurable  If our key outcomes
characteristic of a person or occurred, how would
their situation, which is linked we know?
with a state or condition of
interest to us”  Which measures
have the greatest
 Base measures, derived communicative
measures and KPIs power?
 Varied vantage points
 What data do we
 Use modules for flexibility and already have?
consistency

11. 11
Example measures
Status change/maintenance scale:
1. % of clients who were homeless at the end of the
period
Level of functioning scale:
2. % of clients who rate their parenting skills better at exit
than they did at entry
Goal attainment scaling:
3. % of clients who achieved better than expected
outcomes in majority of goals reviewed this period

12. 12
Example tool: Outcomes Star
Triangle Consulting / London Housing Foundation. http://www.homelessoutcomes.org.uk/The_Outcomes_Star.aspx

13. 13
Collect data
Data collection and entry: Questions to consider:
 Has a huge impact on data  Who will collect the
quality data? What training
will they require?
 Should be integrated as far as
 All clients or a sub-
possible with service delivery
sample?
 Requires client consent
 How and when will
 May need to respond flexibly data be collected?
to circumstances of client  How will data be
stored?

14. 14
Analyse and report
Outcomes reporting enablers: Questions to consider:
 A knowledge of stakeholder  Who will use the
requirements reports?
 Data analysis skills  What levels of
 Standardised: aggregation are
 calculation procedures useful?
 report templates  How can complexity
 database queries
be acknowledged?
 Access to data!

15. 15
Example: control graph
Proportion of service episodes with housing situation improved at completion
Quarterly proportion Cumulative proportion
41.3%
40% 38.4%
37.4%
Percentage of
completed service 34.5% 35.2%
34.3% 34.4%
33.3% 34.9%
episodes 31.8%
(current quarter
n = 165, 28.5%
30%
cumulative 28.6%
n = 1097)
20%
Jun Sep Dec Mar Jun Sep Dec Mar Jun
2009 2009 2009 2010 2010 2010 2010 2011 2011

16. 16
Use outcomes information
Contexts for use: Questions to consider:
 Individual casework:  How can outcomes
assessment, planning and information be
review shared with clients?
 Service review and reflection  How can staff play a
 Strategic planning: “turning the part in “giving
curve” meaning” to
 Advocacy
outcomes
information?

17. 17
Caution!
 Outcomes monitoring data by itself cannot “prove”
service effectiveness
 To build your case, triangulate with other evidence

18. 18
Sustain the system
What helps?
 Using the information – complete the loop
 Strong commitment to and endorsement of outcomes
measurement by senior management
 Performance expectations – staff to collect outcomes
data
 Ongoing resourcing
 Regular review of framework

19. 19
 Risks
 Implementation process
 The bottom line

20. 20
Risks
 Poor value for money
 Outcome measurement can be resource-intensive and
time-consuming
 Information produced may not be high quality
 Distortion of service delivery
 Staff opposition
 Data collection burden
 Feeling scrutinised
 Negative results

21. 21
Process tips
 Involve stakeholders early and ongoingly
 Board, Executive, management, service delivery staff,
clients
 Combine top-down and bottom-up elements
 Pilot locally, implement sequentially
 Typically 6 – 24 months to end of pilots
 Varies with size of organisation, complexity of services, and
level of resourcing

22. 22
The bottom line
 Client wellbeing comes first
 Client rights are respected
 Information produced is useful
 Relevant
 Reliable
 Valid
 System is affordable
 Data collection burden on staff is minimised

23. 23
Useful resources
Results-Based Accountability
 www.raguide.org
 Friedman (2009) Trying Hard Isn’t Good Enough
UK Approaches (incl. Outcomes Star)
 www.homelessoutcomes.org.uk
 Burns & Cupitt (2003) Managing Outcomes: A Guide for Homelessness Organisations
US: National Alliance to End Homelessness
 Spellman & Abbenante (2008) What Gets Measured, Gets Done: A Toolkit on
Performance Measurement for Ending Homelessness
Reference – technical and implementation issues
 Planigale (2010) Literature Review: Measurement of Client Outcomes in Homelessness
Services. http://www.homeground.org.au/assets/literature-review-measurement-of-
client-outcomes-in-homelessness-services.pdf

24. Mark Planigale
Research & Consultancy
results by design
Phone: 0429 136 596
Mail: PO Box 754, Macleod VIC 3085
Email: results@planigale.com
Web: www.planigale.com