6. Heterogeneous concept that must be broken down
There is no specialized concept or metric for the
cognitive aspect of work ability and there are also no
methods at present to measure / assess this.
11. CASNUR: Developing a software to assess cognitive
work ability among aging nurses
by Maria Dominique Rustia & Rosemary Seva
Human Factors and Ergonomics Center, De La Salle University, Manila, Phil.
rustiamd@gmail.com
May 28, 2012
IFA 2012 Global Conference on Ageing
12. Background Method Concept & Design Testing Conclusion
PROBLEM STATEMENT
1
• Existing assessment methods for work ability among nurses are
subject to response bias and fail to consider the cognitive
dimension of tasks.
• There is a need for a performance-based assessment method
for cognitive work ability (CWA) to be established, especially in
the context of ICU nursing.
13. Background Method Concept & Design Testing Conclusion
OBJECTIVES, SCOPE & LIMITATIONS
1
OBJECTIVES
• to provide a reliable and valid measure of CWA among hospital nurses
by developing a software application that can assess CWA among them.
1. To test the validity of the CWA assessment software
2. To perform an initial evaluation of the usability of the software
3. To determine the reliability & effectiveness of software in capturing data for CWA
metric derivation
4. To identify areas for improvement in the software
SCOPE & LIMITATIONS
Hospital nurses, ages 40-65, Intensive Care Unit (ICU) setting
Philippine setting
Will only test variables of CWA model of Rustia & Seva (2011)
Will not propose any interventions
Conceptualization and design of software
Gender, affective & physical needs not considered
14. Background Method Concept & Design Testing Conclusion
COGNITIVE WORK ABILITY
1
Work Ability Index (WAI) “how good workers are at present and in the
near future, and how they will be able to do their work with respect to work
demands, health and mental resources” (Ilmarinen & Tuomi, 2003)
speed accuracy
Cognitive Work Ability (CWA) the ability of a person to fulfill
mentally demanding tasks efficiently and effectively
15. Background Method Concept & Design Testing Conclusion
EXISTING ASSESSMENT METHODS
1
• Reliability and validity have
Work Ability WHO Quality of been verified but coverage of
Index (Ilmarinen Life (WHO, components of CWA is poor
& Tuomi, 1992) 1991) • Questionnaire-based, risk of
biased and inaccurate answers
Expanded • No performance-based
TISS-28 evaluation methods to date
Nursing Stress
(Miranda et al.,
Scale (French et
1996)
al., 2007)
Revised Nursing
Work Index SIMULATION OF THE
(Aiken & NURSING ENVIRONMENT
Patrician, 2000) (low cost software)
21. Background Method Concept & Design Testing Conclusion
SOFTWARE DESIGN – Requirements Planning
3
SIMULATION INTERVIEW &
KNOWLEDGE AUDIT
MONITOR and ALARM
hard to say immediately
where the problem is
EMERGENCIES lead to
improvising
presence of mind and
alertness, prioritization, good
memory, speed
checking, finding
discrepancies in medicine
arrangement and labeling
of medicine, better systems
23. Background Method Concept & Design Testing Conclusion
SOFTWARE DESIGN – Requirements Planning
3
Constantly
changing
E-cart to
concentrate on
individual skill
Computerized to
remove bias
related to
handwriting &
technique
Other Characteristics:
Simulation must also have the same layout as actual ICU generation
• 1 SIMULATION Abnormal event in the Philippines
• Emergency ICU case feeling of “time pressure” and urgency
DATA-GATHERING Errors, Actions, Responses
2
27. Background Method Concept & Design Testing Conclusion
EXPERIMENTATION- Factor Selection
3
Measurable by CASNUR
Measurable by CASNUR
and eye-tracker software
Measurable by CASNUR
and eye-tracker software
Measurable by CASNUR
28. Background Method Concept & Design Testing Conclusion
PILOT TEST – Subjects & Assumptions
4
SUBJECTS & ENVIRONMENT
• Filipino nurses, ages 45+
• Fluent in English and Filipino
• Capable of using computers
• No physical or mental diseases / impairments
• Must have had experience in ICU / emergencies
• Day shift to avoid influence of night shifts
• Pre-scheduled appointments to avoid rushing
ASSUMPTIONS
The participants selected are representative of the
targeted users of CASNUR.
The medications are assumed to be correct even if
they medically are not.
The frequency of errors is assumed to be correct.
The subjects were physically, mentally and
emotionally fit before testing.
30. Background Method Concept & Design Testing Conclusion
PILOT TEST - Methodology
4
RELIABILITY VALIDITY USABILITY
EFFICIENCY
Testing of successful
How close the simulation
generation of variables EFFECTIVENESS
is to real-life situations
and events
SATISFACTION
31. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
RELIABILITY
SIMULATION DATA-
Vital Signs Monitor changes,
GATHERING
alarms and indicates abnormality Error identification,
counting and recording
• Emergency cart with full
functionality of selecting and Response / action
adding medicine to table identification, counting and
recording
• Medicine Table with maximum
capacity of 3 medications Visual timer on interface
• Administration of oxygen, Export feature to EXCEL
dextrose and medicine (input of
dosage)
32. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
RELIABILITY
10 variables measured
33. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
VALIDATION OF CWA MODEL FROM RESULTS
Working Memory
Attentive Resources
Capacity
TESTS
CONDUCTED:
Perception Responsiveness
Collinearity
Heteroskedasticity
Normality
Task structure
34. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
VALIDITY
Shapiro-Wilk W-test for normality Mostly normal in distribution
Correlation analysis Number of wrong medicine types = positive
correlation with unnecessary actions
Response time = +correlation with suffering
Simulation evaluation survey duration
Correct responses = -correlation w/ wrong
type errors
Rating of how realistic simulation is 4 / 5 (5 = very realistic)
35. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
USABILITY
EFFECTIVENESS
High Learnability (12.03%
unnecessary actions)
• 40.66% of abnormal events
solved by the users
EFFICIENCY
“Normal” task completion
times
36. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
USABILITY
SATISFACTION
50% satisfaction rating
5.25 / 6 in ease of use
Second trial slightly easier than first
37. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
DEBRIEFING RESULTS
38. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
AREAS FOR IMPROVEMENT
NORMALITY OF DATA More runs / replications
PREPARATION
1. Only qualitative testing for presence
of mental impairments Use of COGLAB Memory Test
2. Inconsistencies in pacing and tone of Design of a briefing video for CASNUR
briefing voice.
SOFTWARE
1. Unstoppable alarms make nurses more irritable or tense.
2. Nurses tend to forget to select the medicine first before clicking on GIVE MEDICINE.
3. Confusion with color on vital signs monitor.
4. Nurses forget to click DONE / CANCEL to return to main interface from e-cart.
5. Scattering of physician’s orders during experiment
6. Too many abnormal events occurring within the span of 5 minutes.
39. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
AREAS FOR IMPROVEMENT
SOFTWARE
ON/OFF toggling of alarm
Reprogramming of functions
Use of brighter color (YELLOW)
Placing of “GO BACK TO PATIENT”
button in e-cart and changing CANCEL
to “CANCEL addition”
40. Background Method Concept & Design Testing Conclusion
PILOT TEST - Results and Analysis
4
AREAS FOR IMPROVEMENT
EXPERIMENT Touchscreens
1. Difficulty moving mouse around. More subjects and runs
2. More than one run per setting is Do not use beyond 3 tries per
needed. subject.
3. Subjects tend to get significantly Compute for performance
better after 4th / 5th try. value through control limit
4. Need a basis / ideal performance computation & interviews with
value for the task times. nursing educators.
5. Usability testing needs more Inclusion of NASA TLX test.
standardization.
41. Background Method Concept & Design Testing Conclusion
CONCLUSION & RECOMMENDATIONS
5
• Cognitive Work Ability (CWA) is a specialized • Larger experiment with more
concept for the ability of a person to fulfill samples and actual derivation
mentally demanding tasks efficiently and and evaluation of CWA index.
effectively.
• Simulation as performance-based method for • Incorporation of eye-tracking and
assessing CWA among ICU nurses. screen-recording features in
• A simulation software has been successfully system to reduce CASNUR’s
designed with acceptable usability ratings, usage costs.
reliability in data collection & simulation, and
• Testing by wider demographic
validity (face and construct).
(young and old).
• Recommended use of software by the
academe, health professionals, nurses, • Longitudinal testing of software
hospitals. and CWA index.
42. “With society becoming older and more active, will
we change the way we work…? Does it make any
sense to stop being productive at a particular age?”
– Ken Dychtwald (1990)
43. TRY OUT CASNUR
Look for me after the presentation / session /
anytime during the IFA 2012 Conference.
