Measuring the effectiveness
of e-health initiatives in
hospitals
Prof Johanna Westbrook
Health Informatics Research & Evaluation Unit
The University of Sydney

Health Informatics Research &
Evaluation Unit
• 17 research staff most funded by grants
• Aims:
– Develop and test rigorous and innovative evaluation
tools & approaches.
– Produce research evidence about impact of ICT on
health care delivery, professionals’ work and patient
outcomes.
– Disseminate evidence to inform policy, system
design, integration and effective use of ICT in health
care.

Research Questions
• Do pathology order entry systems deliver more
efficient care?
• Do electronic medication management systems make
health care safer?
• Do clinical systems make clinical work more efficient
and release clinicians to spend more time with
patients?
• What is the role of mobile technologies in supporting
clinical work in hospitals?
• Approaches, results to date, methodological
challenges

Is care delivery more efficient?
• Few studies
all specialised units
all reported
improved
turnaround times.
Computerised test ordering
Turnaround time = Time from receipt of specimen in
laboratory to report of result

AIMS
1. Do turnaround times decrease in
the first 12 months following
system introduction and are
improvements sustained?
3. What is the impact on pathology
staff?

Methods
650 teaching hospital
Measurement of TAT pre & post CPOE -Cerner
Millennium PowerChart
Periods
Jul – Aug 2003
Jul – Aug 2004 (post 1)
Jul – Aug 2005 (post 2)
Westbrook JI, et al. (2006) Computerised pathology test order-entry reduces
laboratory turnaround times and influences tests ordered by hospital clinicians: A
controlled before and after study. Journal of Clinical Pathology, 59, 533-536.

Test turnaround time significantly
declined
Year 1 by 18.6% , Year 2 by 12.6%
Period No. tests Mean in minutes
(95% C.I.)
2003 97851 35.35
(35.11,35.59)
2004 113752 28.77
(28.59,28.95)
All
tests
2005 131022 25.14
(24.99,25.29)
• Average number of tests per patient did not change:
92.5 assays/pt vs 103.2 (P=0.23)

Changes in TAT post CPOE in
four hospitals
2005
Before
2006
After
2007
After
Kruskal-
Wallis
Hospital A - Median TAT
77 68 66
P<0.001
% tests using CPOE 75% 80%
Hospital B - Median TAT
145 129 108
P<0.001
% tests using CPOE 0-44% 57%
Hospital C- Median TAT 138 135 113
P<0.001
% tests using CPOE 29-38% 53%
Hospital D- Median TAT 141 139 128
P<0.001
% tests using CPOE 56-71% 74%

Effectiveness – Does a reduction
in TAT really matter?
Is there a relationship between TATs and lengths of
stays in an emergency department prior to CPOE?
Regression analyses - TAT was a significant factor
contributing to patients’ length of stay in ED
(p<0.0001). Westbrook JI, et al (2009) Does computerised provider order entry
reduce test turnaround times?: a before and after study at four
hospitals. Stud Technol Inform; 150: 527-531.

Qualitative studies to assess the impact
pathology work
Focus groups & interviews with management,
pathology, clinical and IT department staff
Observational video study of pathology staff over
several months

“…I don’t have figures to
prove this, but in my
estimation it has made the
turnaround time longer.”
(Senior scientist, 2004)

Implementing Systems
Changes in roles & responsibilities
Elimination of some tasks but creation of new
tasks
Failure of one group to use the system as
expected impacts upon the work of others
These elements of system impact are as important
as quantitative indicators!

Benefits realisation framework
Efficiency Effectiveness Quality
Test costs
Redundant test rates
Turn around times
Work practices
Patient safety
Compliance with
guidelines
Patient management
Length of stay
Test volumes
Communication
Georgiou A, et al (2007) The impact of computerised physician order entry systems on
pathology services: a systematic review. Intern J Med Informatics 76 (7), 514-529.
Georgiou et al. (2008) Electronic test management systems and hospital pathology services
– a framework for investigating their impact. Encyclopaedia of Healthcare Information
Systems

Will electronic medication
management systems
make our health services
safer?

Do e-prescribing systems reduce
prescribing errors in hospital inpatients?
13 papers (US 6, UK 4, Europe 2, Israel 1)
– 9 showed significant decrease
– 2 decrease in some categories
– 2 an increase in errors
• Limitations in study designs, eg only specific drugs
• Only 5 studies examined severity of errors – 2 defined their
scales
• Very limited evidence of effectiveness to reduce serious
errors
Reckmann, Westbrook et al (2009) Does computerized order entry reduce prescribing errors
for hospital inpatients? A systematic review. Journal of American Medical Informatics
Association. 16 (5) 613-623.

Controlled Before & After study
2 Hospitals 2 Systems 6 wards

Methods
Prospective medication chart
review pre & post.
– Inter-rater reliability, kappa =
0.82-0.84
Classification of:
– error types
– severity – 5 point scale
– Clinical
– Documentation
– System-related
• 2006 – pre 2008/9 - post
Prescribing error types
Wrong medication
Wrong dose / volume
Wrong rate /frequency
Wrong route
Wrong timing
Wrong strength
Wrong formulation
Wrong patient
Not prescribed
Drug-drug interaction
Duplicated drug therapy
Patient Allergic
Drug not indicated
Inadequate monitoring
Unclear order
Incomplete order
Unsigned order
Legal/Procedural
System related

Do electronic medication
administration records reduce errors?
Few studies – all
flawed methods
– Perceptions of staff
– Examination of
voluntary incident
reports

Observational Medication
Administration Error Study
• Observe nurses as they
prepare & administer
medications
• Record interruptions &
multi-tasking
• Compare observed data
with patients’ charts to
identify errors

Study Methods
6 wards at 2 hospitals
Information sessions to recruit nurses
– - 82% response rate (n=98 nurses pre)
• Researchers arrived on the wards at peak
medication times (7:00-19:30)
• Approx 8 administrations/observation Hr
• Inter-rater reliability – Kappa score 0.94-0.96
• Serious error protocol used 10 times

How does system use impact
upon patterns of work?
Will these systems save time?
Do drs & nurses spend more time with
patients?

Aim: To develop a reliable method for
observing and recording time spent by
clinicians in different work tasks
Work Observation
Method By Activity
Timing (WOMBAT)
Westbrook JI, Ampt A (2009) Design, application and testing of the Work Observation Method by
Activity Timing (WOMBAT) to measure clinicians’ patterns of work and communication.
International Journal of Medical Informatics. 78S, S25-S33.

PDA data collection tool
What task?
With whom?
With what?
Interruptions
Multi-tasking

Controlled before and after study
nurses and doctors
4 wards at baseline
1 or 2 intervention wards
2 control wards post
Completion date Dec 2009

Proportions of observed time in tasks
across four wards (Before)
Task Nurses
N=52
Ward
Drs
N=19
ED Drs
N=40
Hours of observation 250 hrs 151 hrs 210 hrs
Direct Care 24% 15% 29%
Professional
Communication
22% 33% 24%
Medication tasks 17% 7% 5%
Indirect Care 13% 18% 26%
Social Activities 13% 17% 6%
In transit 9% 6% 3%
Documentation 8% 14% 16%
Supervision/education 3% 7% 2%
Administration 3% 2% 2%
Answering Pager 1% 1% <1%

Time with patients & interruptions
(Baseline data)
Nurses = 34.5%, interrupted
1/49mins, 12% multi-tasking
Ward Drs = 15.0%, interrupted
1/21mins, 20% multi-tasking
• On average nurses spend 8.4
mins/shift talking with a Dr.
Westbrook JI, et al (2008) Medical Journal of Australia. 188(9):
506-509.

Distribution of doctors’ tasks over the day
2006
0%
5%
10%
15%
20%
25%
Time
Proportion
of
task
Direct care
Indirect care
Medication tasks
All documentation
Prof Communication
In transit

Distribution of doctors’ tasks over the day
including social tasks 2006
0%
5%
10%
15%
20%
25%
Time
Proportion
of
task
Direct care
Indirect care
Medication tasks
All documentation
Prof Communication
In transit
Social

Data Analysis
• Changes in
– distribution of time across tasks
– average time for each task
– frequency of each task
– times of the day when tasks completed
– with whom tasks are completed
• A lot more to come …….

Challenges of integrating the
use of technology into
everyday work practices

Poor mobility
workarounds may result in less
safe practices

Paper is a highly mobile
technology!

Capturing what happens on a
ward
Structured observations
Video observations
Talking to staff
80 hours observation, 2 wards
Aim: To measure which devices nurses and
doctors select
Andersen P, Lindgaard A, Prgomet M, Creswick N, Westbrook JI (2009) Is selection of
hardware device related to clinical task?: A multi-method study of mobile and fixed
computer use by doctors and nurses on hospital wards. J Medical Internet Research. 11(3)
Available devices on each ward:
Two forms of COWs (n=5 & 6)
Two forms of tablets – (Motion computing C5
& LE1700) (n=2/ward)
Fixed PCs (n=7)

Computers on wheels
82% of nurses’ tasks 3% of nurses’ work tasks

Doctors’ on ward rounds
57% of tasks completed using a generic COW
36% of tasks completed using a tablet
Only 3% of tasks completed at the patient’s bedside

Conclusions
Recognise the limitations of existing
evidence-base
Use explicit indicators & measure them
Engagement of academics/clinicians/
vendors
Feedback impact data to staff
Create a market for evidence of impact
Share & compare between systems, organisations

Acknowledgements
HIREU Team
• Andrew Georgiou
• Joanne Callen
• Amanda Woods
• Margaret Reckmann
• Connie Lo
• Yvonne Koh
• Fiona Ray
• Nerida Creswick
• Marilyn Rob
• Mirela Prgomet
• Antonia Hordern
• Fiona McWhinney
• Pia Andersen
• Anne-Mette Lingaard
Funding Bodies
• Australian Research Council
• NH & MRC
• NSW Health
• HCF Research Foundation
• SSWAHS
Hospital staff at our study sites
Publications available at:
www.fhs.usyd.edu.au/hireu/
J.Westbrook@usyd.edu.au