1. 7th ATM R&D Seminar – Barcelona (Spain) – July 2-5, 2007
Kevin M. Corker, PhD
Professor, San Jose State
University
&
Jose L Garcia-Chico
ATC Research Analyst
Titan Industries
An analysis of Operational Errors:
An Analysis of Operational Errors and the
interaction with with TCAS
Interaction TCAS
July 2-5 , 2007
2. Agenda
Problem Statement - Motivation
1
TCAS system and operators’ behaviour
2
Methods
3
Results
4
Conclusions
5
Slide 2 July 2-5 , 2007
3. Problem Statement
• Operational Error (OE) rate has been increasing through 2003 and
reaching a plateau in the US airspace. The absolute number of OEs
is still increasing.
OE rate per 100,000 facility activities
0.8
0.78 0.78 0.77
0.74
0.69
1211
0.66
1216 1506
0.6 0.6
0.56
0.53
0.52 0.52 0.51
FAA (2006, April). Administrator’s Fact
0.4
Book. Washington, DC: Department of
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Transportation.
• Specific Interest: recent accidents/incidents involving TCAS
• For Example
• Yaizu (2001), TCAS involved in both a/c. No fatalities
• Uberlingen (2002), TCAS involved in both a/c. 71 fatalities
Slide 3 July 2-5 , 2007
4. Motivation & Ongoing Study
• Classify operational errors and contextual factors in ATC in search of
trends and consistency in the classification.
• Assumption: classification of errors provides understanding of work
performance and organizational/operational context.
• Focus on presence of TCAS RA in evolution of the operational error
• TCAS is an effective safety system, with caveats…
It might disrupt the controller’s SA (Brooker, 2004; Wickens et al, 1998)
Amplified by the fact that changes FL (vertical resolution) are only
provided as number in data block)
It might create inconsistent pilot and controller responses (Rome et al.,
2006, Wickens et al, 1998)
Intention: Understand procedural and informational context of OEs co-
occurring with TCAS RAs.
Slide 4 July 2-5 , 2007
5. TCAS – expected behavior
• For TCAS to work as designed, immediate and accurate crew
response to TCAS advisories (action within 5 sec.) is essential.
• Regulation of TCAS: operational procedures and practices (FAA
AC 120-55B)
Pilots:
• Should follow TCAS RA, unless doing so would jeopardize the safe
of operation. (Required response within 5 sec of RA display)
• During an RA, do not maneuver contrary to the RA based solely
upon ATC instructions.
• S/he has to report any deviation from ATC clearance, as soon as
practicable after responding to the RA, and resume previous
clearance after “clear of conflict”
Controllers:
• Will not knowingly issue instructions that are contrary to RA
guidance when they are aware that a TCAS maneuver is in
progress.
Slide 5 July 2-5 , 2007
6. TCAS events timeline (“desired”)
Pilot notifies
Return to clearance
Pilot follows RA &
deviates from clearance Clear of conflict
Pilot notifies ATC of
deviation
RA
ATC aware deviation
Timeline ATC SA Impaired
Window to receive ATC
Window to receive ATC
clearance in opposition to RA
clearance in opposition to RA
with controller aware of RA
without controller aware of RA
Controller provides traffic info,
If workload permits
Controller is not responsible for separation
Adapted from: Brooker, P. (2004). Thinking about downlink
of resolution advisories from airborne collision avoidance
systems. Human Factors and Aerospace Safety 4 (1), 49-
65.
Slide 6 July 2-5 , 2007
7. Other TCAS research: Operator behavior during TCAS
• TCAS in simulation settings (Rome et al., 2006)
• Variability and deficiencies in pilot communications
• TCAS RA maneuvers increased stress
• Controller cleared vertical deviations during RA maneuvers (4
out 32).
• Research on RA downlink (Brooker, 2004;
Eurocontrol, 2003b, 2004)
• Controllers found it beneficial:
– Improve SA and
– avoid contradictory ATC clearances.
• Problematic issues:
– overload of information,
– pilot compliance,
– change in responsibilities,
– procedures,
– liability.
Slide 7 July 2-5 , 2007
8. Methods
• Exploratory Study: mapping relationships in the
data.
Analysis of errors based on preliminary and final Air Traffic
Controller Reports
• Excluded: Surface and Oceanic Errors
• Two studies/datasets:
Taxonomic Study: classification of OE initial incident
reports (Jan-Jun 04 period: 480 OE reports)
• Classification of OEs based on FAA classification schema.
• Relevance of coordination, training, proximity, time on position.
Focused Study: OEs with presence of TCAS RAs. Final
reports (Jan-Jun 2004 & 2005: 62 reports)
• Use of same classification.
• Characterization of the TCAS RA events. Human response.
Slide 8 July 2-5 , 2007
9. Slide 9
Fa
il
C
on
0
10
20
30
40
50
60
70
80
90
C v er
on gi
D ng
es trol
28
c
ce
84
nd oor
d
t
O
32
ve rho
44
rlo ug
9
Ve oke h
d
66
ct
T
or
H in rf
22
ea ad
48
r/R eq
25
Fa Alti ead u
t
il
43
ba
Al ude ck
tC In
24
TOTAL
lim ad
41
ARTCC
eq
b/
D
7
C es u
49
Fa lim cen
b
il th d
O
22
32
v e r ou
In
Terminal Radar
rt gh
st
r u ak i
10
c ng
36
-
no
te -in Trf
8
m te
p
23
nd
er
M ro ed
13
r-
is
17
da ap i s s u
p e
ta
bl l Pr 16
oc oc
k- ed
m
is
10 8 8
en 2
Ai ter
r
Tr
an s pa
sp ce
13 12
FP ose
OE Classification
a/
S-
34
Sp mis c
Reports
Operational Errors
318 (66.04%)
162 (33.96%)
480
7
en
ee
t
d
Results Study 1: Taxonomic Study
i n er
1
ad
9
W equ
ro
1
n
a/ g a
/
c
54
ov c
er
l
C LO ap
le A
ar
m
ed
bl i s
M wm
is in
re
ad
ARTCC
ot
he info
TRACON
rs
1 13 12 11 03
/w
560 (69.1%)
250 (30.9%)
Classification
Operational Error
810
ha
t
56
July 2-5 , 2007
10. OEs co-occurring with TCAS
Jan-Jun 04 Jan-Jun 05
Terminal Radar 8 (30.8%) 34 (43.6%)
ARTCC 18 (69.2%) 44 (56.4%)
TOTAL 26 78
Proportion of OE types (based on the total OE number)
20% 18.3%
Full set of reports TCAS RA reports
13.8%
1 .9%
1
10.9%
9.4%
9.4%
10% 9.3% 8.9%
8.6% 8.4% 8.4%
8.0%
7.9%
6.9% 6.7%
5.7%
5.0%
4.0% 3.8% 3.7%
3.5% 3.2%
2.5%
1.0%
0% Altitude Inadequate
Instruction not
Fail Converging
Descend through
Climbing/Descending
Climb through
Fail Overtaking Traffic
Temporal error-issue
Overlooked Traffic
Vector Inadequate
Hearback / Readback
Control coordination
Fail Identification
Intended
Altitude
Slide 10 July 2-5 , 2007
11. ATC Commands IN TCAS Situations
ATC Com m ands In RA OE
45
40
35
30
25
% of 104
Series1
20
15
10
5
0
Before RA After RA None Undetermined
Slide 11 July 2-5 , 2007
12. ATC Commands Dependence on Information Integrity
Inform ation Integrity
40
35
30
25
% of 59 Reports
Complete Info
20 Incomplete Info
15
10
5
0
Horizontal:Before Vertical: Before Horizontal:After Vertical: After None
RA RA RA RA
Slide 12 July 2-5 , 2007
13. ATC Vertical Commands after RA and Flight Deck Report
Vertical Com m ands After RA
60
50
40
% Correct out of 20
30 Series1
20
10
0
Vertical Correct Vertical In Opposition
Slide 13 July 2-5 , 2007
14. Deviations from “expected” behavior
Clearances issued by controller upon triggered TCAS RA
25
NO REPORT
COMPLETE INCOMPLETE
% of incidents
20 4.8 opposite to RA
0 opposite to RA
15 6.5
Traffic
12.9
Heading
10 17.7
4.8
Altitude
12.9
14.5
5 9.7
8.1
8.1 3.2
1.6
0
Before After None Before After None Before After None
Incomplete = missing any pilot’s message, missing callsign, TCAS direction or excessive delay
Before and after refers to the action of controller in relation to the TCAS RA event.
Traffic, heading, or altitude mean ATCO gave traffic info, or change heading, or altitude
Slide 14 July 2-5 , 2007
15. Highlights on the chain of events during TCAS RA
encounters in OE reports
• Controllers issued clearances after TCAS RA in the vertical plane
in 13 situations (21 %).
• Controllers received incomplete information in 26 situations
(43.5%) and no information in 3 (5%). Opportunities for wrong
decisions.
• Controllers issued vertical clearances after TCAS RA and
incomplete pilot’s reports in 12 situations (19.4 %).
• opposite altitude clearance in 3 reports (4.8%)
– Pilot reports were all late after TCAS RA and controller clearance
• Data suggests that it is more likely to receive an opposite
clearance if the controller receive incomplete pilot information.
Slide 15 July 2-5 , 2007
16. Proposed Actions
• Increase training recreating TCAS RA situations
Under stress situation, abnormal events trigger more familiar responses (i.e.,
issue vertical clearance)
• Revisit downlinking RAs
Future research needed
Not obvious solution, with important implications
• Draw too much controller attention
• TCAS RA is not the most relevant information, but the pilot deviation from
clearance
• Controller’s responsibility and liability implications
Slide 16 July 2-5 , 2007
17. Conclusions
• Value of systematic characterization of errors
OE classification would allow prioritization of actions.
Failure to notice converging aircraft, control coordination,
hearback/readback, and overlook traffic are the most frequent
• Error reports concurrent with TCAS RA:
OEs with similar patterns to full dataset
Not consistent pilot-controller behavior (deficient information/actions)
Incomplete/late information increases chances of vertical clearances
incompatibles with RA direction
Slide 17 July 2-5 , 2007
18. Acknowledgement
• Special thanks for comments on this paper and
insightful ideas during the study to Dr. Kim Cardosi
(Volpe Laboratories) & Ms. La Gretta Bowser
(FAA)
• Thanks to Mr. Bill Davis (OSTP) for his
sponsorship and comments
Slide 18 July 2-5 , 2007
21. Proximity Rating
• Proportion of higher-proximity events in terminal areas.
• Errors with low frequencies have higher proximity (reduced
cross check)
ARTCC TRACON
Proximity Rating A 7 (2.1 %) 51 (33.3%)
Proximity Rating B 31 (9.5 %) 65 (42.5 %)
Proximity Rating C 285 (86.9%) 21 (13.7 %)
No rated 5 (1.5 %) 16 (10.5%)
(Chi-square X2 (2,N=460)=226, p<0.001)
Slide 21 July 2-5 , 2007
22. Error Severity and Frequency by Time on Shift
• No statistical significance in the distribution of frequencies
(60 min.)
• Not been able to claim that errors are more likely after break
relief or transition into position.
• No evidence that errors were more severe in the first 30
minutes after taking over control (X2 (10,N=373)=7.27, p=0.700)
40 38
37
37
35 35
35 33 32 32
30
30 28
27
OE Frequency
24
25
20 18
15 13
11
10
10 8
7
5
4
4
4
5
1
0
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 110 120 >
120
Minutes on Position
(Chi-square X2 (11,N=388)=6.575, p=0.832)
Slide 22 July 2-5 , 2007
23. TCAS RA and the proximity of aircraft
• Higher proximity when TCAS RA is triggered (only in centers we
could proof statistically)
• Smaller than expected.
Consequence of time logic implemented by TCAS, and/or
Proof of global efficiency of TCAS (“safe the day”)
Proximity rating of OE
80%
70%
2.1%
60% 0.0%
50% Unk
C
40% 1.0% 40.4%
B
7.7% 59.4%
3.1% A
30%
6.5%
18.3%
20%
13.5%
14.4%
10% Centers (Chi-square=32.037, p<0.001)
13.5% 10.6% 4.4% TRACON (Chi-square=0.254, p=0.88)
4.8% 1.5%
0%
TRACON (RA) TRACON ARTCC (RA) ARTCC
Slide 23 July 2-5 , 2007
