The September 8, 2011 Pacific Southwest disturbance resulted in a cascading outage that left over 2.7 million customers in Arizona, Southern California, and Mexico without power for up to 12 hours. It was initiated by the tripping of a single 500kV transmission line, but was exacerbated by high pre-disturbance system loading and the subsequent overloading and tripping of additional lines and transformers as power flows redistributed across the underlying transmission systems. This led to a ripple effect of equipment trips, automatic load shedding, and the operation of special protection schemes, ultimately resulting in the separation and collapse of the main affected balancing areas.

1. Pacific Southwest Disturbance
September 8 2011
8,
Event Analysis Overview Presented to IEEE PSRC
May 17, 2012 – Main Committee Meeting

2. September 8, 2011 Event
• 11 minute cascading outage in Pacific Southwest
• 2.7 million customers out in AZ, S.CA, MX, some up to 12
2.7 million customers out in AZ, S.CA, MX, some up to 12
hours
• Initiated when single 500 kV line tripped, but not sole cause
• Power redistributed, increasing flows through underlying
systems, causing voltage drops and equipment overloads
• Ripple effect led to tripping of lines and generators,
automatic load shedding, and operation of a Remedial
Action Scheme and an intertie separation scheme
Action Scheme and an intertie separation scheme
• Restoration process generally effective
2 RELIABILITY | ACCOUNTABILITY

3. Customer Impacts
Entity Load Lost (MW) Customers
SDGE 4,293 1.4 million
CFE 2,150
2 150 1.1
1 1 million
IID 929 146 thousand
APS 389 70 thousand
WALC 74 Note
Note: 64 MWs of WALC s load loss affected APS s customers
Note: 64 MWs of WALC’s load loss affected APS’s customers
3 RELIABILITY | ACCOUNTABILITY

4. Joint FERC/NERC Inquiry
• Announced September 9, 2012
• Teams formed comprising over 30 senior technical staff of
Teams formed comprising over 30 senior technical staff of
FERC and NERC, plus several NERC contractors and industry
subject matter experts
• Multiple meetings and exchanges with affected entities to
gather facts
• T
Team products combined into final report
d bi d i fi l
• Outreach sessions to gain feedback on draft findings and
recommendations
4 RELIABILITY | ACCOUNTABILITY

5. Inquiry Teams
• Data Requests/Management
• Sequence of Events
Sequence of Events
• Modeling/Simulation
• Cause Analysis/Human Performance
y /
• Operations Tools/ SCADA/EMS
• Frequency Analysis
q y y
• System Planning/Design
• Equipment Performance/System Protection
• Restoration
5 RELIABILITY | ACCOUNTABILITY

6. Sequence of Events
6 RELIABILITY | ACCOUNTABILITY

7. Simplified System Diagram
Serrano Devers Palo Verde
S-Corridor
SONGS
Path
44 Imperial WAPA
Irrigation District Lower Hassayampa
H
161 kV 92 kV Colorado
H-NG
SDG&E
Corridor
S-Line
Miguel Imperial Valley North Gila
CFE
7 RELIABILITY | ACCOUNTABILITY

8. Phase 1 – Pre-Disturbance
• Hot, shoulder season day; some
generation and transmission
outages
• High loading on some key
facilities: H-NG at 78% of normal
rating; CV transformers at 83%
• 44 minutes before loss of H-NG,
IID’s RTCA results showed loss
of CV 1 transformer would load
CV-1
CV-2 transformer above its relay
trip point
• 15 27 39 APS t h i i
15:27:39: technician
skipped a critical step in isolating
the series capacitor bank at
North Gila substation; H-NG trips
; p
8 RELIABILITY | ACCOUNTABILITY

9. Phase 2 – Trip of H-NG 500 kV
• H-NG 500 kV trips at
15:27:39
15:27:39 15:28:16
15 27 39 – 15 28 16 • APS tells WECC RC line
expected to be restored
quickly
• H-NG flow redistributes: 77%
Gf %
to SCE-SDGE (Path 44);
remainder to IID, and WALC
• CV transformers immediately
overloaded above relay
settings
• Path 44 at 5,900 amps; 8,000
amp limit on SONGS
separation scheme
p
9 RELIABILITY | ACCOUNTABILITY

10. Initiating Event – Tripping of
Hassayampa – North Gila 500 kV Line
Series Capacitor
Bypass Switch
Arcs Over
10 RELIABILITY | ACCOUNTABILITY

11. Hass. – N. Gila 500 kV Line Trip
South of SONGS Current
CCM Unit 1
generator trip
Hassyampa –
N. Gila 500 kV
line trip
11 RELIABILITY | ACCOUNTABILITY

12. Phase 3 – Trip of CV Transformers
• 15:28:16 – CV-2 and CV-1
230/92kV transformers trip on
overload relays
15:28:16 15:32:10
15 28 16 – 15 32 10
• Severe low voltage in WALC
161 kV system
• Loading on Path 44
increases to 6,700 amps
12 RELIABILITY | ACCOUNTABILITY

13. CV Transformers Trip
South of SONGS Current
Coachella Valley
230/92 kV
transformers trip
13 RELIABILITY | ACCOUNTABILITY

14. Phase 4 – Ramon Xfmr Trip
• 15:32:10 Ramon 230/92kV
transformer trips on overload
relay
15:32:10 15:35:40
15 32 10 – 15 35 40
• 15:32:13 Blythe-Niland
161kV line trips
• 15:32:15 Niland – CV 161kV
line trips
• IID undervoltage load
shedding; loss of generation
and 92 kV transmission lines
• Severe low voltage in WALC
161 kV system
t
• Loading on Path 44
increases to 7,800 amps;
, p ;
settles at 7,200 amps
14 RELIABILITY | ACCOUNTABILITY

15. Ramon Transformer Trip
South of SONGS Current
Voltage collapse in
l ll i
pocket, followed
by load tripping
Multiple line, generator
and load trips
Ramon 230/92 kV
transformer trip
15 RELIABILITY | ACCOUNTABILITY

16. Voltage in Northern IID 92 kV System
Over‐Voltage Trip
of 92 kV System
Capacitors
Trip of Over 400
Ramon 230/92 kV MW in Northern
Transformer Trip
Transformer Trip IID 92 kV Load
IID 92 kV Load
Pocket
16 RELIABILITY | ACCOUNTABILITY

17. Blythe 161 kV Voltage
Trip of Hassayampa
p y p El Centro – Pilot
El Centro Pilot
– North Gila 500 kV Knob 161 kV Line
Line Trip
Trip of Coachella
Valley 230/92 kV
Transformers Yucca 161/69 kV
Transformers 1 and
2 Trip
2 Trip
Trip of Over 400
Trip of Over 400
Ramon 230/92 kV MW in Northern
Transformer Trip IID 92 kV Load
Pocket
17 RELIABILITY | ACCOUNTABILITY

18. Phase 5 – Yuma Separates
• Yuma AZ Separates from IID
and WALC when Gila and
Yucca transformers trip
15:35:40 15:37:55
15 35 40 – 15 37 55
• Yuma load pocket isolated on
single tie to SDG&E
• Loading on Path 44
increases to 7,400 amps after
Gila transformer trip; to 7,800
amps after Yucca
ft Y
transformers and generator
trip
18 RELIABILITY | ACCOUNTABILITY

19. Yuma Separation
South of SONGS Current
YCA
generating
g g
units trip
Yucca 161/69 kV Pilot Knob
transformers trip 161/92 kV
transformers
trip
Gila 161/69 kV
transformers trip
19 RELIABILITY | ACCOUNTABILITY

20. Phase 6 – High-Speed Cascade
• El Centro – Pilot Knob 161kV
line trips; all IID 92 kV system
radial from SDG&E via S-Line
15:37:55
15 37 55
• WALC 161 kV system voltage
returns to normal
• Path 44 exceeds 8,000 amp
setting and timer starts
20 RELIABILITY | ACCOUNTABILITY

21. El Centro – Pilot Knob 161 kV Trips
Early
Phase 4
activity
Gila
transformers
trip
Pilot Knob
transformers
trip
Zone 3
pickup
Yucca
transformers
trip
21 RELIABILITY | ACCOUNTABILITY

22. Phase 6 – High-Speed Cascade
• S-Line RAS trips IV
generation and S-Line,
isolating IID from SDG&E
15:37:58 15:38:02.4
15 37 58 – 15 38 02 4
• Path 44 loading increases to
as high as 9,500 amps and
reduces to 8,700 amps after
isolating IID
22 RELIABILITY | ACCOUNTABILITY

23. Phase 6 – High-Speed Cascade
South of SONGS Current
Imperial Valley El
Imperial Valley – El
Centro 230 kV
CLR “S” line trip
generating
units trip SONGS
separation
i
El Centro – Pilot
Knob 161 kV line
trip
23 RELIABILITY | ACCOUNTABILITY

24. Phase 6 – High-Speed Cascade
• SONGS separation scheme
operates; forms SDG&E,
CFE, Yuma combined island
15:38:21.2
15 38 21 2
• Combined island frequency
begins to collapse
24 RELIABILITY | ACCOUNTABILITY

25. SONGS
separation
SONGS Unit 3
electrically
disconnects
SONGS Unit 2
electrically
disconnects
SONGS
SONGS
generators
ring down
25 RELIABILITY | ACCOUNTABILITY

26. SONGS
SONGS
separation SONGS Unit 2 GSU
trip and transfer of SONGS aux load
aux load on startup
transformers
SONGS
generators
ring down
ring down
SONGS Unit 3 GSU
trip and transfer of
aux load
26 RELIABILITY | ACCOUNTABILITY

27. Phase 7 – S CA Separates
• Yuma island separates from
SDG&E when IV-NG 500 kV
trips on underfrequency
15:38:23.12
15 38 23 12
• APS UFLS operates, but
insufficient to stabilize load
pocket
• Yuma load pocket collapses
27 RELIABILITY | ACCOUNTABILITY

28. UFLS Operations in the Island
28 RELIABILITY | ACCOUNTABILITY

29. Voltage
Voltage
peaks as tie
lines open
SONGS
separation
SDGE UFLS begins SDGE ties to CFE and Yuma
shedding load open on underfrequency
29 RELIABILITY | ACCOUNTABILITY

30. Phase 7 – CFE Separates
• Underfrequency UFLS
SDG&E and CFE
continues
15:38:23.13
15 38 23 13 • CFE separates from SDG&E
30 RELIABILITY | ACCOUNTABILITY

31. Actual vs. Simulated Frequency
31 RELIABILITY | ACCOUNTABILITY

32. Phase 7 – Complete Blackout
• IID, Yuma, SDG&E, and CFE
blacked out
By 15:38:38
B 15 38 38
32 RELIABILITY | ACCOUNTABILITY

33. Use of PMUs in EA
33 RELIABILITY | ACCOUNTABILITY

34. Sammis-Star
Hard-Chamb
Hard Chamb
Star-S.Cant
CantC-Tidd
Hanna-Jun
2003 Blackout Simulations
RELIABILITY | ACCOUNTABILITY
345 kV
345 kV
345 kV
345 kV
345 kV
Dale-W.Can
138 kV
W.Ak 138 kV
Bkr Failure
Cham-W.Ak
138 kV
E.Lima-N.Fin
138 kV
CantC Xfmr
W.Ak-PV Q21
W
138
1 kV
Outages
Babb-W.Ak
138 kV
E.Lima-N.Lib
E
138 kV
Clov-Torrey
138 kV
Star-S.Cant
345 kV
Hanna-Jun
345 kV
Hard-Chamb
345 kV
34
140
120
100
80
60
40
20
0
Amps)
% of Norma Ratings (A
al

35. PMU Data from SONGS
35 RELIABILITY | ACCOUNTABILITY

36. 7 Phases of Event
Trip or Gila 161/69 kV
Trip or Gila 161/69 kV
Trip of Coachella Transformers &
Valley 230/92 kV Yuma Cogen
Transformers
Trip of Pilot Knob
161/92 kV & Yucca
161/69 kV Trip of Pilot
Transformers Knob – El
Trip of Hassayampa
Centro 161 kV
– North Gila 500 kV
North Gila 500 kV
Line
Line
Trip of Ramon South of SONGS
230/92 kV
/ Separation
Transformer
1 2 3 4 5 6 7
Disturbance Phases
36 RELIABILITY | ACCOUNTABILITY

37. EA Modeling
37 RELIABILITY | ACCOUNTABILITY

38. Simulation vs. Actual Flows
South of SONGS Flows
4000
3500
3000
2500
MW
2000
1500
1000
500
0
15:27:30.240 15:28:56.640 15:30:23.040 15:31:49.440 15:33:15.840 15:34:42.240 15:36:08.640 15:37:35.040
Local Time (PDT)
L l Ti (PDT)
Simulated Actual (PMU)
38 RELIABILITY | ACCOUNTABILITY

39. Inquiry Findings
39 RELIABILITY | ACCOUNTABILITY

40. Key Findings
Weaknesses in two broad areas:
• Operations planning
Operations planning
• Real‐time situational awareness
Contributing factors:
Contributing factors:
• Not studying impact of sub 100 kV facilities parallel to EHV
• Failure to recognize IROLs
Failure to recognize IROLs
• Not studying/coordinating effects of protection systems and
RASs during contingency scenarios
RASs during contingency scenarios
• Not providing effective operator tools and instructions for
reclosing lines with large phase angle differences
40 RELIABILITY | ACCOUNTABILITY

41. Operations Planning
Failure to consider in seasonal, next‐day, and real‐time studies:
• Status of external generation and transmission facilities
Status of external generation and transmission facilities
• Impact of external contingencies
• Impacts on external systems
p y
• Impact on BPS of reliability of sub‐100 kV facilities parallel to
EHV system
41 RELIABILITY | ACCOUNTABILITY

42. Operations Planning
Recommended TOP/BA Improvements in WECC:
• Obtain information on neighboring BAs and TOPs, including
g g , g
transmission outages, generation outages and schedules, load
forecasts, and scheduled interchanges
• Identify and plan for external contingencies that could impact
Id tif d l f t l ti i th t ld i t
their systems and internal contingencies that could impact
neighbors’ systems
• Consider facilities operated at less than 100 kV parallel to EHV
system that could impact BPS reliability
• Coordinated review of planning studies to ensure operation of
C di d i f l i di i f
affected Rated Paths will not result in reliability problems
42 RELIABILITY | ACCOUNTABILITY

43. Situational Awareness
Recommended Improvements in WECC:
• Expand entities’ external visibility in models through more
p y g
complete data sharing
• Improve use of real‐time tools to ensure constant monitoring of
potential internal or external contingencies
t ti l i t l t l ti i
• Improve communications among entities to help maintain
situational awareness
situational awareness
• TOPs should review their real‐time monitoring tools (State
Estimator and RTCA) to ensure critical facilities are represented
43 RELIABILITY | ACCOUNTABILITY

44. System Protection Issues
Recommendations:
• IID and other TOs review transformer overload protection
p
relay settings
• TOPs plan to take proper pre-contingency mitigation
measures considering emergency ratings and overload
id i ti d l d
protection systems
• All protection systems and separation schemes (Safety
Nets, RAS, and SPS) studied and coordinated periodically to
understand their impact on BPS reliability to ensure no
unintended or undesirable effects
44 RELIABILITY | ACCOUNTABILITY

45. Questions?
45 RELIABILITY | ACCOUNTABILITY