Neiberding

Space System Development:
Lessons Learned
The Roots of Mission Assurance

Excerpts

NASA PM CHALLENGE
February 9, 10, 2010

Joe Nieberding
Used with permission

Introduction

• Excerpts from a two day presentation developed to assist
today’s space system developers
– Explore overarching fundamental lessons derived from
• 35 specific mishap case histories from multiple programs
• “Root” causes not unique to times/programs
• Will cover some material from the two day presentation:
– A few of the detailed case histories
– Sample countermeasure “principles”
• References given for all resource information
– Lessons learned charts (yellow background) were either developed
independently by AEA or extracted from resource information

2
© 2006 All Rights Reserved. Aerospace Engineering Associates LLC

35 Case Histories Covered In The Two Day
Presentation
Case Event Case Event Case Event
Loss of Centaur attitude control
Atlas Centaur Centaur weather shield Titan Centaur Degraded Titan Stage 2 engine
Titan IVB-32 – spacecraft delivered to useless
F1 structural failure 6 performance
orbit

Atlas Centaur Atlas booster engine shutdown Atlas Centaur Atlas thrust section in-flight On pad rain damage to
GPS IIR-3
5 on pad – vehicle destroyed 43 explosion spacecraft

Loss of electrical power (L+105 Mars Climate Spacecraft impacted Martian
Apollo 1 Command Module fire Seasat
days) Orbiter surface

Apollo 13 Diverging stage 2 POGO – Atlas Centaur Loss of Centaur attitude control Mars Polar Uncontrolled descent to Martian
POGO premature engine shutdown 62 due to LOX tank leak Lander surface

Apollo 13 Command Module LOX tank Atlas Centaur Loss of vehicle attitude control
X-33 Program canceled
Explosion explosion 67 following ascent lightning strike

Atlas Centaur Failure of Nose Fairing to High Gain Antenna failed to
Galileo X-43A Loss of Pegasus attitude control
21 jettison deploy

Atlas Centaur Loss of control at Centaur Structural failure of spacecraft
Mars Observer Loss of contact with spacecraft CONTOUR
24 ignition due to SRM plume heating

Orbiter damaged upon TOS
Program cancelled after four Loss of control under turbulent
N-1 STS-51/TOS separation system Super Zip Helios
flight failures flight conditions
firing

Atlas Centaur Inertial Reference Systems Spacecraft severely damaged in
Increased frequency of launch
Launch Ariane 501 shutdown during first stage NOAA N Prime procedure-challenged ground
aborts due to upper air winds
Availability burn – vehicle destroyed handling

Orbital Workshop Spacecraft parachutes failed to
Loss of attitude control – battery
Skylab Micrometeoroid Shield structural Lewis Genesis deploy – some data obtained
depletion; mission failure
failure after crash into the earth

Spacecraft hit target; premature
Titan Centaur Loss of attitude control –
Centaur engines failed to start SOHO DART depletion of attitude control
1 communication lost for 3 months
propellants

Atlas Centaur Loss of Atlas control – booster Loss of primary instrument INDICATES THIS CASE INCLUDED IN THIS
WIRE
33 separation disconnect anomaly cryogens – mission failure SUMMARY

3
© 2006 All Rights Reserved. Aerospace Engineering
Associates LLC

Historical Perspective: Failures Have Always Been With Us
• 1100 – 1250: Italian Master Masons learned how to build The Milano Cathedral*
Gothic structures via practical skill
• 1250 +: Master masons moved on to other parts of Europe
– This skill in practice decayed as rules of practice were transferred to
apprentices
• 1399: A wall collapsed in the Milano Cathedral
– Cardinal-Archbishop in Milano called in the masons
• What would be done differently if they re-built the wall?
• The answer was “nothing” for they only knew how to build it by rote
– Pope convened a board of Master Masons from throughout Europe
• This “Cathedral Accident Investigation Board” identified the problem
• Lesson: Artisans engaged in “operational” activities, i.e.
those implementing designs according to rules of thumb,
lose the “tactile feel” for and an understanding of the basis
of the rules
• Recovering from this loss often requires the assembly of
scarce expertise from across a field (the NASA Engineering Ars sine scientia nihil est
http://verostko.com/mignot.html
and Safety Center is a fine example of this)
*Milano Cathedral historical commentary and photograph courtesy of Dr. Joseph R. Fragola,
Vice President, Valador Corporation.
4
Associates LLC

Total Number of Spacecraft Launched 1957 - 2006
Sponsor* Number %
Russian 3476 54%
American 1735 27%
European 279 4%
Japanese 107 1.5%
Chinese 103 1.5%
Indian 43 0.7%
Canadian 27 0.4%
Israeli 10 0.1%
Other Government 102 1.3%
Commercial 529 8%
Amateur/Student 80 1.0%
Total 6376
* Sponsor means Spacecraft owner – not always the same as the entity launching it.
5
Associates LLC

Worldwide Space Mission Success Rate by Decade 1957 - 2006

1600
1400
1200
1000
Number of
800
Space Missions
600
400
200
0
1957- 1967- 1977- 1987- 1997-
1966 1976 1986 1996 2006
Success 601 1401 1462 1270 1043
Fail 232 165 112 89 116
Success Rate 72% 89% 92% 93% 90%
Decade

6
Associates LLC

Worldwide Space Mission Failures by Mission Phase since 1957
35
End of Mission Pad: Pre-launch event
Launch: Launch to earth
Mission orbit failure
30 Orbital: Spacecraft/Upper
Orbital stage to final orbit failure
Mission: Mission
Launch
objectives unmet
25
Number of Missions

Pad
End of Mission: Payload
not recovered as intended

20

15

10

5

0

Year of Launch
7
Associates LLC

Perspective

• Space system reliability has improved dramatically
over a five decade history
• Of 6376 total space missions attempted, 694 failures
occurred = 89% average rate of success
– First decade = 72% success
– Last decade = 90% success

• The material to follow places a magnifying glass on
the relatively small fraction of space mission mishaps

8

A Quick Aside About Design Error “Screens”
Design Error
GIVEN: “Screens”
Our design “machine” (humans)
WILL produce errors at some finite rate-not zero

Test

Design
Error

Design Review

Unexpected
Behavior

“Engineers today, like Galileo three and a half centuries ago, are not superhuman. They
make mistakes in their assumptions, in their calculations, in their conclusions. That they
make mistakes is forgivable; that they catch them is imperative.” (1)

(1)“To Engineer is Human”; Henry Petroski, Vintage Books, 1992
9

Atlas Centaur AC-5

Liftoff Apogee Pad Impact

• Category: Hardware Design/System Engineering
• Problem: Atlas engine shutdown twelve feet off the
launcher – vehicle and pad destroyed (3/2/1965)
• Impact: R&D Flight; launch pad heavily damaged

10

Atlas Centaur AC-5 (cont’d) - Video

11

AC-5 Booster Low Pressure Fuel Duct

12
Associates LLC

Atlas Centaur AC-5 (cont’d) - Video

13

Atlas Centaur AC-5 (concluded)

• Why: Booster engine fuel pre-valve not fully open
– Flow through partially open pre-valve will completely close it
– Position switch design unreliable indicator of valve position

LESSONS:
• Design to ensure fail-safe feedback to confirm proper
configuration before launch commit
• Do not use valve designs subject to flow-induced closure

Source: Subject Matter Experts: (Karl Kachigan, John Silverstein)

14

Galileo

• Category: Hardware
Design/System Engineering
• Problem: Partial Deployment of
high gain antenna (launched
10/18/1989; deployment attempt
4/11/1991)
• Impact: Significant reduction in
downlink data rate
– Subsequent workarounds ameliorated
losses significantly
Source: http://www.nasa.gov/offices/oce/llis/0492.htmlNASA Public Lessons
Learned entry 0492; The Galileo High Gain Antenna Deployment Anomaly,
Michael R. Johnson, JPL; NASA TM-1999-209077, Aerospace Mechanisms
and Tribology Technology: Case Study

15
Associates LLC

Galileo High Gain: Antenna General Arrangement

16
Associates LLC

Galileo Failure Mode: Lubrication Breakdown

Receptacle Fitting
17
Associates LLC

Galileo Failure to Fully Deploy Cause
• Excessive friction in the antenna mid-point restraint pin/ “V” -
socket interface
• Rib preloading caused high pin – socket contact stress
– Relative pin – socket motion broke pin ceramic coating
– Further relative motion in air removed coating and dry lube leaving rough
surfaces
– Further relative motion in vacuum removed oxidized and contaminated Ti
leading to galling of pin and socket
– Asymmetric rib deployment eventually stalled ballscrew motor
– Contributing factor is number and configuration of cross country shipping
events
• Shuttle Centaur cancellation caused one additional cycle
• Antenna deployment testing failed to detect phenomenon
– Vacuum test occurred before vibration damage to pin-socket interface
– Ambient tests went OK due to lower coefficient of friction of Titanium pin-
socket interface in air
18
Associates LLC

Galileo (concluded)

LESSONS:
• Good design vetting is first line of defense in catching subtle
design weaknesses (the V-notch was added to the TDRSS
design by JPL)
• Needs the right kind of reviewers (multidiscipline experts)
• Designs that have implicit time/vibration limits need to be flagged and
factored into operational planning
• When unexpected and unplanned events occur
• Place special focus to understand if any systems impacts result
• Significant storage, transport and mission time was introduced into the spacecraft
life after upper stage reassignment
• Successful environmental testing can be dependent on some
extremely subtle factors
• E.g. in this case only vibration testing in vacuum would have a chance of
catching the problem

19
Associates LLC

Atlas Centaur AC-62

• Category: Production/Operations - Systems Engineering
• Problem: Centaur stage loss of control during coast
phase (6/9/1984)
• Impact: Intelsat V placed in useless orbit
• Why: LOX tank leak
– Minor LOX tank leak escaped build, test and inspection procedures
– Small amount of SOX collected in Centaur to Interstage Adapter area
– SOX augmented normal shock caused by shaped charge firing
– Amplified shock exceeded LOX tank allowables causing four inch crack in
tank
– Escaping LOX propulsive forces not controllable in coast phase
Source: Atlas/Centaur AC-62 Failure Investigation Final Report (NASA GRC Archives) 20
Associates LLC

Atlas Centaur AC-62 General Arrangement

21
Associates LLC

Atlas Centaur AC-62 Crack Development Sequence

22
Associates LLC

Atlas Centaur AC-62 Station 412 Structural Detail

23
Associates LLC

The Balloon Structure of Atlas (until Atlas V) and Centaur

Video

• A Convair innovation - design concept for the original Atlas ICBM
– Subsequently extended to Centaur (to the present)
• Propellant tanks are thin walled pressure vessels
– Strength and stability derived from internal pressure
• Advantage is very light structure (and corresponding attractive mass
fraction)
• Disadvantage is handling complexity – the need to insure tanks are
always pressurized or “stretched”’
– The consequence of not doing so is dramatic
24
Associates LLC

Atlas Centaur AC-62 Leak Location Photograph

25
Associates LLC

Atlas Centaur AC-62 (cont’d)
• Additional Factors:
– First flight with LOX tank pressure elevated ~25% (to
compensate for deletion of boost pumps)
– Tank had been properly qualified at higher pressure
– However, on this tank, factory acceptance leak tests were not
run at the elevated pressure for no explainable reason
– Original LOX tank stress analysis omitted shock effect of
shaped charge firing
• When included however, margins still positive
– Leak check and X-ray procedure protocols dated
• State-of-the-art advances not taken advantage of
– Tank fabrication staff experience level significantly less than for
preceding vehicles

26
Associates LLC

Atlas Centaur AC-62 (concluded)

LESSONS:
• Keep the test program synched up with the design
• Watch out for test/inspection technology insertion
opportunities
• Analyze all relevant environments

27

Atlas Centaur A/C-33

Booster Section

Sustainer Section

Atlas – Stage and a Half Configuration

• Category: Production/Operations – Program Management
• Problem: Vehicle loss of control on ascent (2/20/1975)
• Impact: Loss of Intelsat IV mission
• Why: Atlas booster staging disconnect failed to separate
– Disassembly of swivel in disconnect lanyard (highly likely)
Source: Atlas/Centaur A/C-33 Failure Investigation and Flight Report, Lewis Research
Center, December, 1975 (NASA GRC Archives)
28
Associates LLC

Atlas Centaur A/C-33 Staging Disconnect

Atlas Booster – Sustainer Staging Disconnect
B600P/J 12 29

Atlas Centaur A/C-33 Staging Disconnect Lanyard

30

Atlas Centaur A/C-33 Staging Disconnect Lanyard Swivel

31

AC-33 Booster Launch Configuration and Separated 48 Inches
(Ground Test)

32
Associates LLC

Atlas Centaur A/C-33 - Observations
• The reliability and quality control systems were indicating
swivel failures for nearly eight years, from as early as 1967!
− Several instances of the swivel’s separating into two pieces at the
mating face
• It is incomprehensible that effective action was not taken to
correct the serious problems with this system and its
components
− The lack of follow-up and urgency suggests that the personnel involved
did not understand the disastrous flight consequences that could and
did occur when the system malfunctions
− This was truly an accident waiting to happen!

33

Atlas Centaur A/C-33 (concluded)

LESSONS:
• Adopt an over-arching principle: redundancy is required in
flight critical mechanisms
• Treat anything that performs an in-flight actuation like a
system
• Assign a cognizant lead engineer
• Ensure extra quality attention where there are unavoidable
single points of failure (e.g. acceptance testing)
• Ensure a reliable way of flagging and correcting flight critical
part quality problems
• Absent at GD
• Resulted in this completely preventable loss
• Conduct full scale tests to understand dynamic aspects (e.g.
loads) of separation systems
34
Associates LLC

The Engineering Challenge of Electrical Disconnects - Video

35
Associates LLC

Titan Centaur TC-6 Voyager 1

• Category: Near Miss
• Problem: TC-6 Stage II oxidizer tank
diffuser lodged in a position to
restrict flow to the engine (9/5/77)
• Why: Probably a breakdown in
factory discipline (inspection paper
processed in advance - subsequent
fastener installation not completed)
• Impact: Stage II shut down 544
ft/sec. slower, and 3,000 ft lower,
than predicted, placing a huge
burden on the Centaur upper stage
to compensate

36
Associates LLC

Titan IIIE Stage II Autogeneous Pressurization System

37

Titan Centaur TC-6, Voyager 1 (cont’d)

Fell to near bottom of tank

Restricted N2O4 flow

38

Titan Centaur TC-6 Voyager 1 (cont’d)
• Impact (cont’d): Favorable Earth-
Jupiter geometry on 9/5/77 permitted
Centaur to have enough propellant to
make up the shortfall
• Irony: Had the same failure occurred
on TC-7 (launched 8/20/1977) instead
of TC-6, Centaur would very likely
have run out of propellants before
achieving the target
• Consequence: Voyager 1 would have
failed, and Voyager 2 could not have
completed the “Grand Tour” of
Jupiter, Saturn, Uranus, and Neptune

Sources: NASA Lewis TC-6 Voyager Flight Report (undated) (NASA GRC Archives); Titan III E-6
Stage II Investigation (undated) (NASA GRC Archives); Subject Matter Experts: (Richard
Greenspun, Martin Marietta (ret.);Tom Chandik, General Dynamics);
39

Titan Centaur TC-6 Voyager 1 (concluded)

Lesson:
• If the hypothesis is correct, factory floor discipline broke down
• Human systems fail
• There’s a limit on what one can do to prevent it
• Doesn’t mean one shouldn’t try!
• Protect against these failings by:
• Training
• Audits
• Increasing awareness of consequences of carelessness
• Impact to company
• Impact to Nation
• Impact to individual
• Avoid “blind” installations as a matter of good engineering
practice

40
Associates LLC

The Largest Disaster in the History Of Rocketry
• Baikonur Cosmodrome Russia,
10/24/1960
• Preps for first test flight of R-16 ICBM
• Program rushed to launch on
anniversary of Bolshevik revolution
(as a present for Premier Kruschev)
Mitrofan Nedelin R-16 ICBM
• Lead by head of the Soviet Ballistic
Missile Forces Marshal Mitrofan
Nedelin
• 250 people on and around pad
– Viewing stand for visiting dignitaries Destroyed Pad and Memorial at Baikonur (Tyuratam)

• Unsafe design and undisciplined
procedures caused 2nd stage ignition
• More than 120 people were killed Video

including Nedelin
41
Associates LLC

Observations

• Only one of the 35 mishaps analyzed (Atlas Centaur
24) had failure of a proper part as the cause!
– Programs doing good job of acceptance testing
• Therefore, conventional risk assessment based on
piece part failure rates is, at best, incomplete
• The other 34 were caused by human error,
management weaknesses, systems engineering
shortcomings, etc., which are not easily modeled

42

Another Observation:
Testing Deficiencies Had a Pivotal Role in 20 of 35 Cases
Analysis/Modeling/System Testing to:
Case Acquire Data to Qual/Accept Verify Operational
Enable Design Hardware or S/W Functionality
Atlas Centaur F-1 ?
Apollo 13 POGO ? ?
Apollo 13 Explosion X
Atlas Centaur 24 O O
N-1 O X
Titan Centaur 1 O
Seasat O
Atlas Centaur 62 X
STS-51/TOS X
Ariane 501 O
Lewis X
SOHO X
WIRE X
Titan IVB-32 O
Mars Climate Orbiter O
Mars Polar Lander X
X-43A X
Helios O
Genesis O
DART X
O-Omitted; X-Mis-performed; ?-Category Unknown

43

Observations (concluded)

• Programs that adopt a minimalist approach to
testing are betting on the ability of the engineering
community to foresee all aspects of system
performance under all conditions
– This is a very risky bet!

History demonstrates that tests frequently, if not
usually, produce unexpected (and unwanted) results

44

What Can Be Done to Avoid Mistakes of the Past?

• Every project must implement a very specific set of
principles which must be followed and rigorously
enforced
• For example:
• Everything which can be tested on the ground will be
• All analyses must be independently verified and based on test
data where possible
• Heritage Systems: Any system flown successfully on an earlier
mission will be deemed unsuitable for flight until retested

Enforced principles such as those above would
have prevented many mission failures !

45

Conclusions

46

Conclusions
• Most mishaps can be broadly attributed to human error, not
rocket science
– Lack of complete understanding of how complex systems interact with each other
– Inadequate attention to every detail
– Flawed analyses or tests
– Improper use of “heritage” systems
– Flawed processes
– Flawed understanding of how software fails
– Reaction to budget or schedule pressure
– Imperfect management
• Often, a complex, subtle, sequence of events is needed
– If just one event in the chain were prevented, the failure would not have happened
• Must ensure quality in all the above areas
• Essential for mission success
• Over decades, the same root causes of failures appear repeatedly
• There are few new ones!
47
Associates LLC

Conclusions (cont’d)
Why Don’t We Learn From Past Mistakes?
• The lessons are learned mostly by the people who were there
when the failure occurred
– With time, the keepers of the knowledge disappear
– What’s left is a diminishing, second-hand memory that also fades quickly
– Paper, or even electronic, lessons learned data bases (as good as they’re
getting) are still insufficient by themselves to keep the memory alive
• Lack the live element to
– Reveal the nuances
– Fill in the details the official “record” omits
– Convey the passion
• Nothing beats talking to those who “were there”
• Basically, there is no universally successful approach to learning
the lessons from the past
The business of transferring lessons
learned is best done as a “contact sport” 48
Associates LLC

P. O. Box 40448
Bay Village OH 44140
www.aea-llc.com
Joe Nieberding, President Larry Ross, CEO
Email: joenieber@sbcglobal.net Email: ljross1@att.net
Cell: 440-503-4758 Cell: 440-227-7240
MISSION
AEA’s mission is to leverage the vital lessons
learned by NASA’s spacefaring pioneers to
strengthen the skills of today’s aerospace
explorers.


Appendix A: Glossary of Terms
Term Definition Term Definition Term Definition

ACS Attitude Control System FOD Foreign Object Debris JPL Jet Propulsion Laboratory

Advanced Communications
ACTS GAO Government Accountability Office JSC Johnson Space Center
Technology Satellite

AEA Aerospace Engineering Associates GD General Dynamics KSC Kennedy Space Center

Automatic Determination and
ADDJUST Dissemination of Just Updated GPS Global Positioning System LCCE Life Cycle Cost Estimate
Steering Terms

APL Applied Physics Laboratory GN&C Guidance Navigation and Control LM Lockheed Martin

APU Auxiliary Power Unit I&T Integration and Test LOX Liquid Oxygen

ATK Alliant Techsystems IC Integrated Circuit LMA Lockheed Martin Astronautics

AV AeroVironment, Inc. IIP Instantaneous Impact Point LSP Launch Service Provider

Independent Program Assessment Microgravity Droplet Combustion
BFC Better Faster Cheaper IPAO MDCA
Office Apparatus

CAIB Columbia Accident Investigation Board IRU Inertial Reference Unit MES Main Engine Start (Centaur)

CONOPS Concept of Operations ISA Initial Sun Acquisition (SOHO) MMH Monomethylhydrazine

Defense Meteorological Satellite
DMSP ISSP International Space Station Program MO Mars Observer
Program

Independent Verification and
ESR Emergency Sun Reacquisition (SOHO) IV&V MOU Memorandum of Understanding
Validation

50
Associates LLC

Appendix A: Glossary of Terms (concluded)
Term Definition Term Definition Term Definition

MS Meteoroid Shield (Skylab) S&MA Safety and Mission Assurance SRR System Requirements Review

MSFC Marshall Space Flight Center S/C Spacecraft SSME Space Shuttle Main Engine

Science Applications International
NAC NASA Advisory Council SAIC SSTO Single Stage to Orbit
Company

National Aeronautics and Space
NASA SDR System Design Review STS Space Transportation System
Administration

National Oceanic & Atmospheric
NOAA SE Systems Engineering TOS Transfer Orbit Stage
Administration

NRA NASA Research Announcement SEB Source Evaluation Board UAV Uncrewed Aerial Vehicle

Systems Engineering Management
NTO Nitrogen Tetroxide (N2H4) SEMP USAF United States Air Force
Plan

OSP Orbital Space Plane SLI Space Launch Initiative VSE Vision for Space Exploration

P&W Pratt and Whitney SOA State of the Art

PDT Product Development Team SOX Solid Oxygen

Longitudinal oscillation (as in POGO Space Plasma High Voltage
POGO SPHINX
stick – not an acronym) Interaction Experiment

RLV Reusable Launch Vehicle SRB Solid Rocket Booster

RSRM Redesigned Solid Rocket Motor SRM Solid Rocket Motor

51
Associates LLC

Neiberding

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (20)

Mehr von NASAPMC

Mehr von NASAPMC (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Neiberding