SlideShare verwendet Cookies, um die Funktionalität und Leistungsfähigkeit der Webseite zu verbessern und Ihnen relevante Werbung bereitzustellen. Wenn Sie diese Webseite weiter besuchen, erklären Sie sich mit der Verwendung von Cookies auf dieser Seite einverstanden. Lesen Sie bitte unsere Nutzervereinbarung und die Datenschutzrichtlinie.
SlideShare verwendet Cookies, um die Funktionalität und Leistungsfähigkeit der Webseite zu verbessern und Ihnen relevante Werbung bereitzustellen. Wenn Sie diese Webseite weiter besuchen, erklären Sie sich mit der Verwendung von Cookies auf dieser Seite einverstanden. Lesen Sie bitte unsere unsere Datenschutzrichtlinie und die Nutzervereinbarung.
Virtual Operating Room for Team Training in Surgery
Virtual operating room for team training in
Jonathan S. Abelson, M.D.a,
*, Elliott Silverman, P.A.a
, Alexandra Naidesb
, Ricardo Costaa
Gregory Dakin, M.D.a
Department of Surgery, New York Presbyterian HospitaldWeill Cornell Medical College, New York,
NY 10068, USA; b
Department of Physiology and Biophysics, Weill Cornell Medical College, New York,
BACKGROUND: We proposed to develop a novel virtual reality (VR) team training system. The
objective of this study was to determine the feasibility of creating a VR operating room to simulate
a surgical crisis scenario and evaluate the simulator for construct and face validity.
METHODS: We modified ICE STORM (Integrated Clinical Environment; Systems, Training, Oper-
ations, Research, Methods), a VR-based system capable of modeling a variety of health care personnel
and environments. ICE STORM was used to simulate a standardized surgical crisis scenario, whereby
participants needed to correct 4 elements responsible for loss of laparoscopic visualization. The
construct and face validity of the environment were measured.
RESULTS: Thirty-three participants completed the VR simulation. Attendings completed the simu-
lation in less time than trainees (271 vs 201 seconds, P 5 .032). Participants felt the training environ-
ment was realistic and had a favorable impression of the simulation. All participants felt the workload
of the simulation was low.
CONCLUSIONS: Creation of a VR-based operating room for team training in surgery is feasible and
can afford a realistic team training environment.
Ó 2015 Elsevier Inc. All rights reserved.
The beneﬁts of simulation in surgery are well docu-
mented, allowing trainees to achieve proﬁciency in shorter
times, acquire news skills, and retain skills, all in environ-
ments that are inexpensive, reproducible, and safe.1–12
the abundance of evidence to support simulation, surgical
residency programs have rapidly adapted inanimate training
into their curricula, with most programs stafﬁng full-time
The American College of Sur-
geons has recognized the value of simulation in surgery
and has developed an extensive accreditation program for
Most of the focus in surgical simulation has been on task
training of surgical skills, ranging from knot tying to chest-
tube insertion to ﬂexible endoscopy. Perhaps the most salient
example is laparoscopic surgical skill training, whereby a
systematic course of box trainer–based laparoscopic tasks
has proved so effective, and the course has been mandated for
certiﬁcation by the American Board of Surgery.16
technical skill is only one aspect of being an effective
This study was funded in part by Lockheed Martin.
* Corresponding author. Tel.: 11-914-980-4530; fax: 11-212-746-
E-mail address: firstname.lastname@example.org
Manuscript received November 26, 2014; revised manuscript January
0002-9610/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved.
The American Journal of Surgery (2015) -, -–-
surgeon. Furthermore, the surgeon is only one part of the
team necessary to deliver effective surgical care.
Successfully performing an operation requires the co-
ordinated teamwork of the surgeon, anesthesiologist,
nurses, hospital staff, and clinical information systems.
Recognizing the importance of teamwork in the operating
room (OR) has led educators to move from the relatively
simplistic training of surgical tasks to the more complicated
world of whole team training in the surgical environ-
Team training in surgery involves creation of a
simulated OR, in which combinations of real equipment
coupled with mannequins and computerized integration
allow a human team to recreate real-life operative sce-
narios. The American College of Surgeons has created a se-
ries of standardized OR situations that can be used in
Several groups have reported success with multidisci-
plinary OR simulation.20–22
Others although have had difﬁ-
culty executing such simulation secondary to cost and
Furthermore, with limited duty
hours, it can be difﬁcult to assemble the necessary team
members to undergo the training. These factors have
limited the widespread use of team training simulations in
surgery compared with the relatively simple task training
that has been widely adapted.
We hypothesized that VR software can offer realistic team
training environments that overcome some of the current
limitations. Therefore, we proposed to create a multiuser
interactive environment in which both team-based skill
coupled with surgical decision making can be simulated,
critiqued, and evaluated. Lockheed Martin Corporation
(Oswego, NY), well known for its military-based simulation
and training programs, has developed a VR-based environ-
ment for use in medical training called ICE STORM (Inte-
grated Clinical Environment; Systems, Training, Operations,
Research, Methods). The objective of this pilot study was to
determine the feasibility of modifying the ICE STORM VR
OR to simulate a standardized surgical crisis scenario and
evaluate the simulator for construct and face validity.24,25
The existing ICE STORM platform contains all the
equipment and personnel necessary to simulate a variety of
scenarios in a virtual OR. A core team of researchers from
both Lockheed Martin and Weill Cornell Medical College
(New York, NY) was established to modify ICE STORM to
simulate an intraoperative crisis scenario. In addition to
modeling the necessary elements of this crisis scenario,
additional software was created to allow a human proctor to
serve as an interface between study participants and the
virtual world. The interface software used a standard iPad
(Apple Inc., Cupertino, CA) to allow a human proctor to
modify the simulation environment as necessary during the
course of the training exercise, depending on the partici-
The laparoscopic troubleshooting module is a team
training scenario published by the American College of
Surgeons and has been previously validated.16
module is beyond the scope of this pilot project and entails
a comprehensive team of surgeon, assistant surgeon, nurses,
and anesthesiologist working through several crisis situa-
tions while performing a standard laparoscopic operation.
In this project, the focus was narrowed to a small portion
of the module, called ‘‘loss of laparoscopic visualization.’’
This scenario was then programmed into the modiﬁed
ICE STORM platform.
In the ‘‘loss of laparoscopic visualization’’ scenario, the
team was performing a laparoscopic cholecystectomy and
the laparoscopic monitor suddenly went dim. It was the
job of the operating surgeon to troubleshoot the problem
and return the monitor to working form. There were several
possible problems that the participant, functioning as the
operating surgeon, had to identify and check to restore
function to the monitor. Each participant was required to
perform 4 mandatory treatments: (1) check camera box
and cord; (2) check light-source box and cord; (3) check
(clean/inspect) or replace laparoscope (use spare); and (4)
exchange and use spare camera. Participants were evalu-
ated by time to completion of the simulation. Participants
were given a ‘‘pass’’ if they completed all mandatory treat-
ments in less than 270 seconds. This number was based on
preliminary analysis of test subjects.
Participants, acting as the surgeon, interacted with the
VR world using the Gyration Air Mouse (SMK-Link
Electronics, Camarillo, CA; Supplementary Fig. 1) to
manipulate the surgeon avatar and interacted with the hu-
man proctor administrating the study by simply speaking
aloud. The proctor then used the proctor-tool software on
a standard iPad (Apple Inc.) to make modiﬁcations to the
VR world (eg, participant instructs nurse to replace the
laparoscope, the proctor enters the appropriate command,
and then the nurse avatar replaces the laparoscope). Suc-
cessful completion of the module was achieved when the
participant identiﬁed all 4 of the critical elements that could
lead to loss of visualization. No matter what order a partic-
ipant identiﬁed the element, the simulation would not end
until all 4 components were identiﬁed and correctly acted
on. This concept, known as the ‘‘full cycle test,’’ ensured
that all participants would demonstrate a complete under-
standing of the most critical elements of the trouble-
shooting scenario and eliminated the possibility that the
participant could end the simulation simply by picking
the ‘‘correct’’ cause of failure on the ﬁrst try.
Metric data from the simulation exercise were used to
evaluate the construct validity of the virtual system. Three
methods were used to determine the face validity: Likert
scale questionnaires (Table 1), the Bedford Workload Scale
(Supplementary Fig. 2), and the modiﬁed NASA-Task Load
Index (NASA-TLX) scale (Supplementary Fig. 3). Subjects
included attending surgeons (experts) and residents and
medical students combined (trainees). Statistical analysis
was conducted using SPSS 12.0 statistical software (IBM,
2 The American Journal of Surgery, Vol -, No -, - 2015
New York, NY). Speciﬁc modalities used were Fisher exact
test, ANOVA, and t tests. Metric data were analyzed using
the Fisher exact test comparing pass–fail rates and time to
completion because the sample size was low.
Seven-point Likert scale questionnaires were used to
assess a variety of opinions, including participants’ impres-
sion of the virtual environment with regard to realism,
beliefs on whether the system would improve performance
in the OR, and inclination to use the system in the future26
(Table 1). Median scores were used to determine a central
tendency. Independent-sample Mann–Whitney test was
used to detect any difference in the median scores between
trainees and attendings. We also performed a 1-sample
Whitney–Wilcoxon signed rank test comparing median
scores against the midpoint of the evaluation scale (ie, 4).
The aim of this analysis was to reveal whether the trainees’
evaluations were above the midpoint of the scale (ie, posi-
tive) or below it (ie, negative).19
The Bedford Workload scale is an unidimensional rating
scale designed to identify operator’s spare mental capacity
while completing a task.27
The single dimension is assessed
using a hierarchical decision tree. Supplementary Fig. 2 ex-
plains the scoring system from workload 1 to 10.
Independent-sample Mann–Whitney test was used to detect
any difference between workload scores of attendings vs
trainees. One-sample Wilcoxon signed rank test was calcu-
lated to understand any difference between the scores of all
participants against the midpoint of the scale, which was 5.
The modiﬁed NASA-TLX is a subjective, multidimen-
sional, validated assessment tool that rates perceived
workload on 6 different subscales: Mental Demand,
Physical Demand, Temporal Demand, Performance, Effort,
Results are combined but not weighted.
Supplementary Fig. 3 reveals each category investigated
and the scaling system. An independent-sample Mann–
Whitney test was used to detect any difference between
workload scores of attendings vs trainees. The 1-sample
Wilcoxon signed rank test was calculated to understand
any difference between the scores of all participants against
the midpoint of the scale, which was 10.
A total of 33 participants, including 26 trainees and 7
attendings, completed the virtual simulation. Attendings
completed the simulation is less time than trainees (201 vs
271 seconds, P 5 .032; Fig. 1). A higher percentage of at-
tendings passed the simulation compared with trainees
although this was not statistically different (86% vs 58%,
P 5 .223).
Eleven questions were included in the Likert scale
questionnaire. Those questions that pertain directly to
opinions on the VR simulation are reviewed here (questions
1, 2, and 7 to 11). Those questions that addressed
background information on participants’ surgical experi-
ence or previous virtual world experience are not discussed
in this section (questions 3 to 6; Table 1).
Overall, participants agreed that they liked the simulator
(median 5 5, P 5 .000) and disagreed with the statement
that they disliked the simulator (median 5 2, P 5 .000).
Study participants felt that the training environment was
realistic (median 5 5, P 5 .014) and did not feel that
communication in the environment was difﬁcult (median
5 2, P 5 .000).
However, participants did not feel that the simulation
would improve their performance (median 5 4, P 5 .534).
Furthermore, they did not agree that they would like to do
VR training before going to the OR (median 5 4, P 5
Using the Bedford Workload scale, 82% of all partici-
pants felt that the workload was either low or that they had
enough spare capacity for desirable additional tasks
(Fig. 2). Using the modiﬁed NASA-TLX scale, all
Table 1 Likert scale results
Likert scale question number Median P value*
1. The training environment was realistic† 5 .014
2. This course will improve my performance in the OR† 4 .534
3. How would you rate your prior virtual world?‡ 1 .000
4. How would you rate your surgical experience level?x 4 .073
5. How often do you use surgical simulation training modules?‡ 3 .001
6. How would you rate your experience with non-surgical virtual world?x 3 .001
7. The VR training module differed greatly
from other non–computer-based communication training modules†
8. I liked the VR training environment† 5 .000
9. I did not like the VR training environment† 2 .000
10. I found it difﬁcult to communicate using the VR training environment† 2 .000
11. I would like to do VR training prior to going to the OR† 4 .607
*One-sample Whitney–Wilcoxon signed rank test value to determine if the median score was different from the midpoint of the scale, which was 4.
Scoring system: 1 5 strongly disagree, 4 5 moderately agree, 7 5 strongly agree.
Scoring system: 1 5 never, 4 5 moderate, 7 5 very often.
Scoring system: 1 5 novice, 4 5 moderate, 7 5 expert.
J.S. Abelson et al. Virtual OR for team training in surgery 3
participants were found to have minimal mental, physical,
and temporal demand. Likewise, none of the participants
reported requiring a high amount of effort to complete
the simulation (Fig. 3).
Finally, there was no statistically signiﬁcant difference
in responses between attendings and trainees for all
responses in the Likert scale, Bedford Workload scale,
and Modiﬁed NASA-TLX scale.
The OR is a dynamic, high-risk environment where
successful delivery of care depends on the co-ordinated
action of surgeons, anesthesiologists, nurses, hospital staff,
and clinical information. With the increasing complexity of
surgical instrumentation and patient disease, efﬁcient OR
processes and teamwork will become more important. The
value of teamwork among medical professionals is well
documented. Failures in co-ordination and communication
of information among hospital clinicians have been asso-
ciated with worse outcomes,29–31
longer lengths of stay, and
higher nurse turnover in intensive care units32
postoperative pain with lower functioning levels for
across specialties. Surgical teams at the
Department of Veterans Affairs hospitals with low mortal-
ity rates communicate more effectively and more often than
surgical teams associated with high mortality rates.36
study of anesthetic-related errors, 80% of occurrences
were considered preventable with human error accounting
for 75% of them.37
Studies in cardiac surgery show the
clear impact of human factors in ‘‘near misses’’ in the
high technology OR.38
Primary teamwork competencies, including knowledge,
skill, and attitudes, positively correlate to effective teamwork
and patient safety.39,40
However, these competencies of effec-
tive teamwork are often lacking in the modern OR. Surveys
evaluating surgeons working with anesthesiologists have sug-
gested that substandard collaboration occurs 50% of the
Other studies have discovered discrepancy in OR
team members’ views on appropriate team structure and
importance of effective communication.42,43
The aviation in-
dustry noted years ago that 70% of errors were because of pre-
ventable human factors, such as failed interpersonal
communication, decision making, and leadership.44
grams that use simulator-based training to understand the lim-
itations of human performance and to develop a culture of
Anesthesiologists have addressed human factors in
errors by developing simulators to train staff during anesthetic
crises, allowing participants to integrate technical and team
training skills with feedback on their performance.46–49
the importance of teamwork and communication in the OR,
several groups are investigating the role offull-scale OR simu-
lation to both study and improve these skills.17,21,22
mate goals of such simulation include not only training of
participants but identiﬁcation of factors important to overall
team performance and improved efﬁciency of OR systems.
In this study, we created an interactive environment to
simulate both team-based skills and surgical decision
making. This virtual environment might ultimately enable
simulation of a variety of OR circumstances, from patient
transfer, to hand-off, to intraoperative crisis scenarios. A
multiuser virtual world could ultimately be linked to
operable surgical procedure simulation platforms to allow
full-scale simulation of both surgical procedures and team
dynamics. Once a complex OR system is modeled, we
Figure 1 Average simulation completion times, attendings vs
trainees. *P 5 .032.
Figure 2 Bedford workload scale: all participants. Figure 3 Modiﬁed NASA-TLX: all participants.
4 The American Journal of Surgery, Vol -, No -, - 2015
could not only train personnel but would be able to
investigate clinical practices and identify inefﬁciencies
with the beneﬁts of speed, safety, measurability, reproduc-
ibility, and reduced cost afforded by advanced simulation.
This study set out to test such a virtual environment by
modifying the ICE STORM technology created by
Lockheed Martin. The objective of this pilot study was to
determine the feasibility of creating a VR OR to simulate a
standardized surgical crisis scenario and evaluate the
simulator for construct and face validity. Our results
conﬁrm that the VR simulator was capable of simulating
the American College of Surgeons ‘‘Loss of laparoscopic
visualization’’ scenario. Metric data revealed that attend-
ings completed the simulation in less time, thus conﬁrming
construct validity. Attendings and trainees liked the simu-
lator, felt that it was realistic, easy to communicate with,
and similar to other non–computer-based,
communication-based training modules, thus conﬁrming
face validity. However, they did not believe it would
improve their performance in the OR, possibly because of
the short simulation session. If the simulation incorporated
the entire laparoscopic troubleshooting scenario, it is
possible that participants would feel that their experience
was more useful in their preparation for the OR. Further-
more, participants did not want to complete the VR training
before going to the OR. Again this may be because of the
limited scope of the pilot simulation. It also may reﬂect
the current lack of experience and acceptance using VR
simulation. This will likely require a culture change once
VR simulation gains more traction.
Using the Bedford Workload Scale, we found that the
simulator did not create excessive workload for partici-
pants, allowing spare capacity for additional tasks. Finally,
using the Modiﬁed NASA-TLX scale, we showed that the
simulator was neither signiﬁcantly mentally or physically
demanding nor exceedingly frustrating or stressful to use.
There are several limitations of this study. First, our
sample size is low, and further research is needed with more
participants. Second, although the simulator is equipped to
accommodate multiple participants simultaneously, this
study only investigated one participant at a time. The
beneﬁt of having multiple participants present at once is
clear as surgeons, anesthesiologists, and nurses would be
able to interact with each other in real time to troubleshoot
common OR problems. This would force the team to
establish a shared goal and shared mental model of the
situation, maintain situational awareness as the situation
evolves, and communicate with each other professionally
so that all members of the team may contribute thoughts in
the best interest of the patient. By simplifying the
simulation in its current form, we might be overestimating
the feasibility and ease of use. Future models will address
these limitations by incorporating multiple participants in
each scenario. Only then can VR simulation be compared
directly with current live simulations to determine if VR is
as effective in team training.
Eliminating the human proctor tool is also necessary to
achieve a program that would meet the demands of today’s
training needs by reducing the workload required of those
institutions performing the training. The scenario would
still need to be programmed to maximize learning by
evolving such that the participants would be forced to
attempt all possible solutions to successfully complete the
simulation. Furthermore, this would contribute to create a
fully simulated environment allowing participants to use
the system in remote locations, possibly achieving a
completely immersive ‘‘holodeck,’’ like that seen in ‘‘Star
As this is a pilot study, we opted to simulate a relatively
more complex and stressful scenarios to fully demonstrate the
utility of an integrated VR system as a training instrument.
Finally, a key component to team training simulation is
automatic debrieﬁng of participants after the simulation.
This will culminate in fully functional team training
environment with widespread appeal.
We conclude that it is feasible to simulate the OR
environment using VR and replicate a standardized surgical
crisis scenario. This VR simulation serves as a proof-of-
concept for construct and face validity; however, more
study is necessary to increase the capability of the
simulation and apply it to the entire OR team.
Supplementary data related with this article can be
found at http://dx.doi.org/10.1016/j.amjsurg.2015.01.024
1. Al Jamal YN, Ali SM, Ruparel RK, et al. The rationale for combining
an online audiovisual curriculum with simulation to better educate
general surgery trainees. Surgery 2014;156:723–8.
2. Araujo SE, Delaney CP, Seid VE, et al. Short-duration virtual reality
simulation training positively impacts performance during laparo-
scopic colectomy in animal model: results of a single-blinded random-
ized trial: VR warm-up for laparoscopic colectomy. Surg Endosc 2014;
3. Brooks KD, Acton RD, Hemesath K, et al. Surgical skills acquisition:
performance of students trained in a rural longitudinal integrated clerk-
ship and those from a traditional block clerkship on a standardized ex-
amination using simulated patients. J Surg Educ 2014;71:246–53.
4. Fernandez GL, Page DW, Coe NP, et al. Boot cAMP: educational out-
comes after 4 successive years of preparatory simulation-based
training at onset of internship. J Surg Educ 2012;69:242–8.
5. Gurusamy KS, Aggarwal R, Palanivelu L, et al. Virtual reality training
for surgical trainees in laparoscopic surgery. Cochrane Database Syst
6. Khatib M, Hald N, Brenton H, et al. Validation of open inguinal hernia
repair simulation model: a randomized controlled educational trial.
Am J Surg 2014;208:295–301.
7. Nagendran M, Gurusamy KS, Aggarwal R, et al. Virtual reality
training for surgical trainees in laparoscopic surgery. Cochrane Data-
base Syst Rev; 2013:CD006575.
J.S. Abelson et al. Virtual OR for team training in surgery 5
8. Schell SR, Flynn TC. Web-based minimally invasive surgery training:
competency assessment in PGY 1-2 surgical residents. Curr Surg
9. Shetty S, Zevin B, Grantcharov TP, et al. Perceptions, training experi-
ences, and preferences of surgical residents toward laparoscopic simu-
lation training: a resident survey. J Surg Educ 2014;71:727–33.
10. Varas J, Mejı´a R, Riquelme A, et al. Signiﬁcant transfer of surgical
skills obtained with an advanced laparoscopic training program to a
laparoscopic jejunojejunostomy in a live porcine model: feasibility
of learning advanced laparoscopy in a general surgery residency.
Surg Endosc 2012;26:3486–94.
11. Villamaria CY, Eliason JL, Napolitano LM, et al. Endovascular Skills
for Trauma and Resuscitative Surgery (ESTARS) course: curriculum
development, content validation, and program assessment. J Trauma
Acute Care Surg 2014;76:929–35.
12. Zendejas B, Cook DA, Bingener J, et al. Simulation-based mastery
learning improves patient outcomes in laparoscopic inguinal hernia
repair: a randomized controlled trial. Ann Surg 2011;254:502–9.
13. Henry B, Clark P, Sudan R. Cost and logistics of implementing a
tissue-based American College of Surgeons/Association of Program
Directors in Surgery surgical skills curriculum for general surgery res-
idents of all clinical years. Am J Surg 2014;207:201–8.
14. Krajewski A, Filippa D, Staff I, et al. Implementation of an intern
boot camp curriculum to address clinical competencies under the
new Accreditation Council for Graduate Medical Education supervi-
sion requirements and duty hour restrictions. JAMA 2013;148:
15. Robinson WP, Baril DT, Taha O, et al. Simulation-based training to
teach open abdominal aortic aneurysm repair to surgical residents re-
quires dedicated faculty instruction. J Vasc Surg 2013;58:247–53.
16. Dunnington GL, MacRae H, DaRosa DA, et al., ACS/APDS Surgery
Resident Skills Curriculum. Available at: https://www.facs.org/
education/program/apds-resident. Accessed October 21, 2014.
17. Aggarwal R, Undre S, Moore K, et al. The simulated operating theatre:
comprehensive training for surgical teams. Qual Saf Health Care 2004;
18. Moorthy K, Munz Y, Forrest D, et al. Surgical crisis management skills
training and assessment: a simulation [corrected]-based approach to
enhancing operating room performance. Ann Surg 2006;244:139–47.
19. Undre S, Koutantji M, Sevdalis N, et al. Multidisciplinary crisis sim-
ulations: the way forward for training surgical teams. World J Surg
20. Arriaga AF, Gawande AA, Raemer DB, et al. Pilot testing of a model
for insurer-driven, large-scale multicenter simulation training for oper-
ating room teams. Ann Surg 2014;259:403–10.
21. Paige J, Kozmenko V, Morgan B, et al. From the ﬂight deck to the
operating room: an initial pilot study of the feasibility and potential
impact of true interdisciplinary team training using high-ﬁdelity simu-
lation. J Surg Educ 2007;64:369–77.
22. Steinemann S, Berg B, Skinner A, et al. In situ, multidisciplinary,
simulation-based teamwork training improves early trauma care.
J Surg Educ 2011;68:472–7.
23. Phitayakorn R, Minehart R, Pian-Smith MC, et al. Practicality of intra-
operative teamwork assessments. J Surg Res 2014;190:22–8.
24. Arain NA, Hogg DC, Gala RB, et al. Construct and face validity of the
American College of Surgeons/Association of Program Directors in
Surgery laparoscopic troubleshooting team training exercise. Am J
25. Maithel S, Sierra R, Korndorffer J, et al. Construct and face validity of
MIST-VR, Endotower, and CELTS: are we ready for skills assessment
using simulators? Surg Endosc 2006;20:104–12.
26. Robbins NB, Heiberger RM. Plotting Likert and Other Rating Scales.
Joint Statistical Meetings. Available at: https://www.amstat.org/
Accessed October 10, 2014.
27. Roscoe AH, Ellis GA. A Subjective Rating Scale Assessing Pilot
Workload in Flight: A Decade of Practical Use. Royal Aerospace
Establishment Technical Report 90019. Available at: http://www.
dtic.mil/cgi-bin/GetTRDoc?AD5ADA227864; 1990. Accessed
September 20, 2014.
28. Carswell CM, Lio CH, Grant R, et al. Hands-free administration of
subjective workload scales: acceptability in a surgical training environ-
ment. Appl Ergon 2010;42:138–45.
29. Agarwal HS, Saville BR, Slayton JM, et al. Standardized postoperative
handover process improves outcomes in the intensive care unit: a
model for operational sustainability and improved team performance.
Crit Care Med 2012;40:2109–15.
30. Davenport DL, Henderson WG, Mosca CL, et al. Risk-adjusted
morbidity in teaching hospitals correlates with reported levels of
communication and collaboration on surgical teams but not with scale
measures of teamwork climate, safety climate, or working conditions.
J Am Coll Surg 2007;205:778–84.
31. Knaus WA, Draper EA, Wagner DP, et al. An evaluation of outcome
from intensive care in major medical centers. Ann Int Med 1986;
32. Shortell SM, Zimmerman JE, Rousseau DM, et al. The performance of
intensive care units: does good management make a difference? Med
33. Gittell JH, Fairﬁeld KM, Bierbaum B, et al. Impact of relational coor-
dination on quality of care, postoperative pain and functioning, and
length of stay. Med Care 2000;38:807–19.
34. Pandian V, Miller CR, Mirski MA, et al. Multidisciplinary team
approach in the management of tracheostomy patients. Otolaryngol
Head Neck Surg 2012;147:684–91.
35. Suarez JI, Zaidat OO, Suri MF, et al. Length of stay and mortality in
neurocritically ill patients: impact of a specialized neurocritical care
team. Crit Care Med 2004;32:2311–7.
36. Young GJ, Charns MP, Daley J, et al. Best practices for managing sur-
gical services: the role of coordination. Health Care Manage Rev
37. Chopra V, Bovill JG, Spierdijk J, et al. Reported signiﬁcant observa-
tions during anaesthesia: a prospective analysis over an 18-month
period. Br J Anaesth 1992;68:13–7.
38. Carthey J, de Leval MR, Reason JT. The human factor in cardiac sur-
gery: errors and near misses in a high technology medical domain.
Ann Thorac Surg 2001;72:300–5.
39. Paige JT, Aaron DL, Yang T, et al. Improved operating room teamwork
via SAFETY prep: a rural community hospital’s experience. World J
40. Einav Y, Gopher D, Kara I, et al. Preoperative brieﬁng in the operating room:
shared cognition, teamwork, and patient safety. Chest 2010;137:443–9.
41. Sexton J, Helmreich R, Thomas E. Error, stress and teamwork in
medicine and aviation: cross sectional surveys. BMJ 2000;320:
42. Flin R, Yule S, McKenzie L, et al. Attitudes to teamwork and safety in
the operating theatre. Surgeon 2006;4:145–51.
43. Undre S, Sevdalis N, Healey AN, et al. Teamwork in the operating
theatre: cohesion or confusion? J Eval Clin Pract 2006;12:182–9.
44. Helmreich RL, Merritt AC, Wilhelm JA. The evolution of crew
resource management training in commercial aviation. Int J Aviat Psy-
45. Helmreich RL, Wilhelm J, Gregorich S, et al. Preliminary results from
evaluation of cockpit resource management training: performance rat-
ings of ﬂight crews. Aviat Space Environ Med 1990;61:576–9.
46. Gaba DM, Howard SK, Flanagan B, et al. Assessment of clinical per-
formance during simulated crises using both technical and behavioral
ratings. Anesthesiology 1998;89:8–18.
47. Holzman RS, Cooper JB, Gaba DM, et al. Anesthesia crisis resource
management: real-life simulation training in operating room crises.
J Clin Anesth 1995;7:675–87.
48. Matveevskii AS, Gravenstein N. Role of simulators, educational pro-
grams, and nontechnical skills in anesthesia resident selection, educa-
tion, and competency assessment. J Crit Care 2008;23:167–72.
49. Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis
management with repeated exposure to simulation-based education.
6 The American Journal of Surgery, Vol -, No -, - 2015