1. NASA Activities in Risk
Assessment
NASA Project Management Conference
March 30-31, 2004
Michael G. Stamatelatos, Ph.D.,Director
Safety and Assurance Requirements Division
Office of Safety and Mission Assurance
NASA Headquarters
1

2. NASA is a Pioneer and a Leader in Space;
Therefore Its Business Is Inherently Risky
International Space Station
Safe assembly and operation
Space Transportation
Space Shuttle
Orbital Space Plane
2

3. Our Goal
Improve risk awareness in the Agency
Conduct PRA training for line and project managers and for
personnel
Develop a corps of in-house PRA experts
Transition PRA from a curiosity object to baseline
method for integrated system safety, reliability and
risk assessment
Adopt organization-wide risk informed culture
PRA to become a way of life for safety and technical
performance improvement and for cost reduction
Implement risk-informed management process
3

4. Probabilistic Risk Assessment (PRA)
Answers Three Basic Questions
Risk is a set of triplets that answer the questions:
1) What can go wrong? (accident scenarios)
2) How likely is it? (probabilities)
3) What are the consequences? (adverse effects)
Kaplan & Garrick, Risk Analysis, 1981
Event
Event
Sequence
Sequence
Frequency
Modeling
Evaluation
2. How frequently does it happen?
(Scenario frequency quantification)
Event
Initiating
Sequence Risk
Event PRA Insights
Logic Integration
Selection
Development
1. What can go wrong? Risk statement Decision Support
(Definition of scenarios) 3. What are the consequences?
(Scenario consequence quantification)
Consequence
Modeling PRA is generally used for
low-probability and high-
consequence events
4

5. Relationship Between Risk Management
and Probabilistic Risk Assessment (PRA)
Continuous Risk Management Method Technique Application
Qualitative
Qualitative
Risk FMEA.
FMEA.
Risk
Assessment MLD,
MLD,
Assessment
ESD,
ESD, Technical
ETA, Technical
ETA, Risk
Risk
FTA,
FTA,
RBD
RBD and/or
Probabilistic
Probabilistic and/or
Risk
Risk
Assessment Actuarial/ Program
Program
Assessment Actuarial/
Statistical Risk
Risk
Statistical
Analyses
Analyses
Legend:
Decision
Decision FMEA - Failure Modes & Effects Analysis
Analysis
Analysis
Management MLD - Master Logic Diagram
Management
System ESD - Event Sequence Diagram
System
ETA - Event Tree Analysis
FTA - Fault Tree Analysis
RBD - Reliability Block Diagram
5

6. NASA Risk Management and Assessment
Requirements
• NPG 7120.5A, NASA Program and Project Management Processes
and Requirements
The program or project manager shall apply risk management
principles as a decision-making tool which enables programmatic
and technical success.
Program and project decisions shall be made on the basis of an
orderly risk management effort.
Risk management includes identification, assessment, mitigation,
and disposition of risk throughout the PAPAC (Provide Aerospace
Products And Capabilities) process.
• NPG 8000.4, Risk Management Procedures and Guidelines
Provides additional information for applying risk management as
required by NPG 7120.5A.
• NPG 8705.x (draft) PRA Application Procedures and Guidelines
Provides guidelines on how to apply PRA to NASA’s diversified
programs and projects
6

7. How Does PRA Help Safety?
Provides a basis for risk reduction through:
1. Accident/Mishap Prevention (best)
2. Accident/Mishap Consequence Mitigation
Adverse Event
in a Sequence
Prevention Mitigation
7

8. The Concept of an Accident Scenario
PIVOTAL EVENTS
Pivotal Pivotal
IE End State
Event 1 Event n
Detrimental
The perturbation Mitigative consequence of
Aggrevative interest
(Quantity of intereset
to decision-maker)
Risk Scenario is a string of events that (if they occur) will lead to
an undesired end state.
8

9. Exact vs. Uncertain Probabilities
Uncertainty distribution
of probability values
Exactly
Probability
known
Probability
probability
value
A narrower range means a more
precise knowledge of the distribution,
or less uncertainty
9

10. Quantification of Uncertainty
Probability density function,
e.g., probability of LOCV
Uncertainty Distribution:
P(x) is the probability (or
xth percentile ρ(x)
confidence) that the
result value is x
P(x)
median is the 50th
percentile is area
under 5% x 50% 95%
curve
between Median
0 and x
Uncertainty Range:
Uncertainty range 5th percentile 95th percentile
(spread) from
the 5th to the 95th
percentile Uncertainty
(confidence)
range
10

11. Event- and Fault-Tree Scenario
Modeling
No damage to No damage to
Hydrazine leaks Leak detected Leak isolated flight critical scientific End state
Leak not detected avionics equipment
IE LD LI A S
OK
Loss of science
Loss of
Spacecraft
Presure Presure
Controller fails transducer 1 transducer 2 OK
fails fails
Loss of science
CN P1 P2
common cause Loss of
failure of P. Spacecraft
transducers
OK
PP
distribution Loss of science
Loss of
Spacecraft
Fault tree Event tree
11

12. PRA Methodology Synopsis
Inputs to Decision Making Process Master Logic Diagram (Hierarchical Logic) Event Sequence Diagram (Logic)
End State: ES1 IE A B End State: OK
End State: ES2
End State: ES2
End State: ES2
C D E
LOGIC MODELING
End State: ES1
End State: ES2
Event Tree (Inductive Logic) Fault Tree (Logic) One to Many Mapping of an ET-defined Scenario
End Not A NEW STRUCTURE
IE A B C D E
State
1: OK Logic Gate
Basic Event Internal initiating events One of these events
External initiating events
2: ES1
Hardware components AND
3: ES2 Human error
4: ES2 Software error one or more
Common cause of these
5: ES2 elementary
Environmental conditions events
6: ES2 Other
Link to another fault tree
Probabilistic Treatment of Basic Events Model Integration and Quantification of Risk Scenarios Risk Results and Insights
30 50
60
25 40 End State: ES2 Integration and quantification of
50
PROBABILISTIC
20
40 100 logic structures (ETs and FTs)
30
15
30
20 and propagation of epistemic
Displaying the results in tabular and graphical forms
10
20
5 10
80
End State: ES1 uncertainties to obtain Ranking of risk scenarios
10
0.01 0.02 0.03 0.04 0.02 0.04 0.06 0.08 0.02 0.04 0.06 0.08
60
Ranking of individual events (e.g., hardware failure,
minimal cutsets (risk
40
scenarios in terms of
human errors, etc.)
Examples (from left to right): 20 basic events) Insights into how various systems interact
Probability that the hardware x fails when needed likelihood of risk Tabulation of all the assumptions
Probability that the crew fail to perform a task scenarios
Probability that there would be a windy condition at the time of landing
0.01 0.02 0.03 0.04 0.05
uncertainty in the
Identification of key parameters that greatly
likelihood estimates influence the results
The uncertainty in occurrence of an event is Presenting results of sensitivity studies
characterized by a probability distribution
12

13. What Decision Types Can PRA Support?
Safety improvement in design, operation, maintenance and
upgrade (throughout life cycle);
Mission success enhancement;
Performance improvement; and
Cost reduction for design, operation and maintenance
For all these areas of application, PRA can help:
Identify leading risk contributors and their relative values
Indicate priorities for resource allocation
Optimize results for given resource availability
13

14. Areas of PRA Application at NASA
In Design and Conceptual Design (e.g., Crew
Exploration Vehicle, Mars missions, Project
Prometheus)
For Upgrades (Space Shuttle)
For Development/construction/assembly (e.g.,
International Space Station)
When there are requirements for Safety Compliance
(e.g., nuclear missions like Mars ’03; Project
Prometheus, Mars Sample Return)
14

15. NASA Procedural Requirement NPR 8705
(Draft)
CONSEQUENCE NASA PROGRAM/PROJECT
CRITERIA / SPECIFICS PRA SCOPE
CATEGORY (Classes and/or Examples)
Planetary Protection Program
Mars Sample Return Missions F
Requirement
Public Safety White House Approval Nuclear Payloads
F
(PD/NSC-25) (e.g., Cassini, Ulysses, Mars 2003)
Human Safety and Space Missions with Flight
Health Launch Vehicles F
Termination Systems
International Space Station F
Human Space Flight Space Shuttle F
Orbital Space Plane/Space Launch Initiative F
High Strategic Importance Mars Program F
Launch Window
High Schedule Criticality F
(e.g., planetary missions)
Mission Success Earth Science Missions
(for non-human L/S
(e.g., EOS, QUICKSCAT)
rated missions) Space Science Missions
All Other Missions L/S
(e.g., SIM, HESSI)
Technology Demonstration/Validation (e.g.,
L/S
EO-1, Deep Space 1)
F = Full scope; L/S = Limited or Simplified
PRA Scope Legend: F = Full scope; L = Limited scope; S = Simplified PRA
15

16. NASA Special PRA Methodology Needs
Broad range of programs: Conceptual non-human rated science
projects; Multi-stage design and construction of the International
Space Station; Upgrades of the Space Shuttle
Risk initiators that vary drastically with type of program
Unique design and operating environments (e.g., microgravity
effects on equipment and humans)
Multi-phase approach in some scenario developments
Unique external events (e.g., micro-meteoroids and orbital debris)
Unique types of adverse consequences (e.g., fatigue and illness in
space) and associated databases
Different quantitative methods for human reliability (e.g., astronauts
vs. other operating personnel)
Quantitative methods for software reliability
16

17. Space Shuttle
Probabilistic
Risk
Assessment
17

18. STS Nominal Mission Profile
ASCENT completes
ORBIT completes
APU Shutdown
TAL
TIG-5
t ~ 150kft)
SSME start ENTRY completes
APU start
18

19. Current Shuttle PRA Results for LOCV
(provisional)
1/25
100
median = 1 in 123
P ro b a b ility D e n s ity
F u n c tio n
mean = 1 in 76
5th percentile = 1 in 617
95th percentile = 1 in 23
0
1/25
1/500
0.00 0.01
1/100 0.02
1/50 0.03
1/33 0.04
1/25 0.05
1/20
truncated
Number of Missions
19

20. Summary of Shuttle PRA Historical Results
Probability of Loss of Crew and Vehicle
1993 PRA
1988 PRA
update the
Median values (log scale)
2003 PRA 1998 PRA First PRA
Galileo study
Integrated Unpublished 1995 PRA conducted on
results to reflect
PRA with all analysis using First major the Space
the current
0.1 elements. QRAS. No study of (April 1993) test
Shuttle for
18 Orbiter Integration of 1996 PRA ascent only,
1997 PRA Shuttle PRA. and operational
systems. elements. Bayesian up for Galileo
First use of Analysis by experience
Incl. MMOD Limited to 3 date of the Mission
QRAS tool. SAIC with base of the
Orbiter 1995 model by
Update of input from Shuttle.
systems and adding 17
1995 PRA with prime
the propulsion new look at
successful
contractors
1/78
elements flights
APU
0.01
1/90
1/123
1/245 1/147 1/145
1/161
1/254
1/245
0.001
SPRAT PRA Shuttle PRA Shuttle PRA Shuttle PRA Shuttle PRA Phase 1 - Galileo 1988
2003 1998 1997 1996 1995 1993 (Ascent (Ascent
Ascent, only) only)
Ascent and entry only
entry, and
orbital debris 20

21. Annual Voluntary Risks in Some Sports -
Comparable in Magnitude to Shuttle Risk
Professional stunting 1/100
Dedicated mountain climbing 1/167
Air show/air racing and acrobatics 1/200
Amateur flying in home-built aircraft 1/333
Experienced whitewater boating 1/370
Sport parachuting 1/500
Source: R. Wilson and E. Crouch,
Risk-Benefit Analysis,
Harvard University Press, 2001
21

22. International Space Station (ISS) PRA
• 1999 -- The NASA Advisory Council
recommended, the NASA Administrator
concurred, and the ISS Program began
a PRA.
− The modeling will be QRAS-
compatible.
− First portion of PRA (through Flight
7A) - delivered in Dec. 2000; Second
portion (through Flight 12A)
delivered in July 2001.
22

23. 23

24. Important ISS PRA Findings
MMOD: lead contributor to
loss of station (LOS) risk
Illness in space: lead
contributor to loss of crew
(LOC) risk
24

25. Approach to PRA for NASA Top-Level
Designs (e.g., Crew Exploration Vehicle)
Strawman
Mission
Level 1 Architecture Design Activities
Requirements
Cost and
Design
Risk an Safety Schedule
Concepts
Goals and Risk
Requirements Assessments
Definition of Safety,
Operational Reliability,
Mission
Phases PRA Schedule,
Requirements
and Cost
Trade-offs
First-Order
Risk Goal
Performance Allocation for
Requirements Each Phase
Design
Engineering
System
Reliability
Allocation
Past Related
PRA Efforts; Mature
e.g., Shuttle Design
Top Level
PRA
25

26. Advanced PRA Methods or Tools
QRAS (Quantitative Risk Assessment
System) – a state of the art integrated
PRA computer program
Galileo/ASSAP – Dynamic fault tree
program
Software reliability methodology for use
in PRA
External event methodology for micro-
meteoroid and orbital debris (MMOD)
risk into the overall risk assessment
26

27. QRAS 1.7 Is Being Commercialized
Space Shuttle
time in seconds
SSME SRB ORBITER ET
R(i,x, t) = R
0(i,x, t) t1 t2 t3 t4
-6 0 128 510
where: SSME
R is reliability of manifold weld HPOTP _______________
i is physical state LPFTP _______________
LPFTP HPFTP HEX MCC HEX _______________
x is vector of physical variables MCC _______________
t is time SRB
TVC _______________
Bolt
Manif Seal Engineering Modes NOZZLE _______________
Weld Fail to determine initiating event probability
Fail Fail
Mission Timeline
Element/Subsystem Hierarchy
Event Sequence Diagram
Initiating
Event Event Tree (quantification)
Manif (furure unhancement: dynamic)
Weld Is crack Is repair Yes Successful Manifold
Failure detectable? 100% effect.? op. Weld Success Prob.
S = 0.9 S = ... S = ...
No Failure LOV Prob.
F = ...
Success Prob.
Is crack small
enoung to survive Yes Successful
F = ... S = ... LOV Prob.
1 mission? op. F = ...
Mission Fail. Prob.
No
F = 0.1 S = 0.96 LOV Prob.
F = 0.04
Loss of flow HPFTP cavitates
to LPFTP No LOX rich op. Successful
op.
Fault Tree Risk Aggregation
Probability Distribution (use to quantify pivotal events)
for initiating event
What if? Tool Box
Probability of Results
1. Remove old subsystem and Bayesian Manifold Weld
replace with design. Updating Failure
2. Change failure probabilities for Mathematica
initiating events.
3. Eliminate failure modes. Others
4. Vary parameters of engineering
models. (future)
Pr
5% tile 95% tile
median
27

28. In Summary, We Plan to
Continue to improve risk awareness
Conduct PRA training for line and project managers
and for personnel
Continue to develop a corps of in-house PRA
experts
Transition PRA to baseline method for safety
assessment
Integrate risk assessment with system safety
and reliability assessment
Adopt organization-wide risk informed culture
PRA to become a way of life for safety and technical
performance improvement and for cost reduction
Implement risk-informed management process
28