TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
Β
K ingoldsby
1. Panning for Gold in Historical
Operations Records
Kevin Ingoldsby
Booz Allen Hamilton
Cape Canaveral, Florida
Ingoldsby_kevin@bah.com
2. The Data Miners Challenge
β’ Parametric cost estimation models are
only as valid as input data which founds
them
β’ Unfortunately for cost estimators and
modelers, very few operational programs
take the time and expense to record
operations data in formats that easily
facilitate future modeling and analysis
β’ But even in the mountains of seemingly
unrelated operations records, gold
knowledge dust and occasional nuggets
can be found
3. Challenge Details
β’ Operations Phase modeling of space systems relies
heavily on historical benchmark data to both anchor
parametric analysis methods and as validation data
to evaluate modeling tool outputs
β’ Unfortunately for most modelers there are often
limitations in the historical records:
β Available recorded data is often riddled with gaps
β Data is inconsistently recorded over the program life
(inadequate data breadth)
β Data is recorded with different rules within lower-level
program elements (inconsistent data depth)
β’ Sometimes, the only data deemed valid are singular milestones such as
hardware delivery, rollout to launch pad, final launch date, mission event
duration, etc.
4. Sources of Data Challenge
β’ The reasons for lack of easily useable data are many:
β Operations budgets are typically very tight with
technical problems during development consuming
margins and eating into operations phase allocations
β For missions with very narrow planetary launch
windows, the pressure to get the mission off the ground
on-time limits the attention spent of recording more
than the barest needed information for milestone
decision makers.
β Operations business support systems database schemas are
driven by the operations management need, typically the
implementation of work planning and closed-loop
accounting for operations requirements.
5. Hope for the Prospector
β’ Useable and valuable operations performance information
may be lurking in records that were created for different
purposes
β The presenter has applied techniques to extract operational metric
data from NASA Space Shuttle and other launch vehicle operations
records
β Products of these data mining efforts have informed development of
several cost and operations modeling tools across the agency
β’ Goals of this presentation:
β Share examples of data extraction efforts
β Show how the data was applied
β Identify some prospective mother-loads
that have yet to be prospected
6. Spaceflight Operations Modeling
β’ Most cost modeling tool development focus has been on
design, development, test and production phases of the
life-cycle
β These phases typically are the largest investment for a
transportation system
β Budgeting process tends to be an annual exercise, this near-
term scrutiny tends to obscure the assessment of recurring
costs
β Some DDTE&C models provide predictions of operations
infrastructure development cost, but not much fidelity of
recurring operations burdens
β’ Predicting the recurring costs and performance of the
operational phase of spaceflight systems motivated the
studies that will be discussed in following slides
8. Case Study: Vision Spaceport Project
(VSP)
β’ Joint Sponsored Research
Agreement involving
KSC, ARC, Boeing, Lockheed
Martin, UCF, CCT
β Follow-on effort from the Highly-
Reusable Space Transportation
Program (HRST)
β Project was conducted from
1998-2002
β’ Project Goal:
β Develop a modeling tool for
prediction of space launch
operations costs and performance
β Focus of the modeling effort was
the quest for Orders of
Magnitude improvement over
then-current systems
(STS, Titan, Atlas II, Delta
II, Pegasus)
9. VSP: Benchmarking
β’ Study team developed a functional model of spaceport
operations to organize the analysis and modeling efforts
β Model functions helped to organize the collection of benchmark
program/vehicle data
β Functions helped to communicate the varying infrastructure and
operational needs of different launch system concepts
β’ Each βModuleβ of the VSP functional model was
documented in the benchmarking effort
β Constituent sub functions described
β Current state examples identified
β Concepts identified for orders of magnitude improvement
10. VSP: Data Collection & Analysis
β’ With a 5-6 order of magnitude scale, a wide range of operational data was
investigated
β 1994 Access to Space study provided much information for STS operations
β’ Bottoms-up assessment was broken down by vehicle element (Orbiter, ET, SRB, Facility)
VSP functional module and cost category
β Historical launch vehicle data helped to expand the set
β’ Range of vehicles from contemporary ELVs to early launch vehicles of the 50βs
β’ Sources included photographs, schedules, narratives, budget data, technical reports
β In most cases, data would be found only for a subset of modules and cost
categories
β’ For example: Information on the launch pad crew headcount, turnaround times for X-15
vehicle flight attempts, cost for construction of the Saturn V launch complex, etc.
β Sources were captured and documented in spreadsheets
β’ Recorded by Vehicle configuration, Function and Cost Category
β’ Each vehicle that provided metric data points was scored for operability using the model
assessment algorithms
β’ Resulting scores used to plot data points and calibrate model output performance curves
11. VSP: Nuggets
β’ STS / Access to Space Study
β Good breakdown of labor and material costs
attributable to vehicle elements and most module
functions
β’ X-15 program flight logs
β Extensive information on turnaround and depot
operations cycle time from 150 missions flown by 3-
vehicle fleet
β’ WSMR research flight logs 1946-58
β Provided assembly and cargo integration cycle time
and crew size data points for suborbital launch
vehicles
12. Case Study: Reliability Modeling (RMS)
β’ LaRC Vehicle Analysis Branch was extending
USAF squadron logistics model to predict
operational performance of Reusable Launch
Vehicles
β Model based on historical aircraft maintenance
operations records of USAF and USN
β Sought assistance at KSC in developing similar
metric data from Space Shuttle operations history
β Initial study assessed missions from 1992-98
β Follow-on effort added missions from 1999-2002
13. RMS: Data Needs
β’ Model required RM&S metrics by subsystem
β Cycle Time metrics (MTBMA, MTTR, etc.)
β Event frequency/probability metrics (Parts Removal &
Replacement frequency, scrap rates, etc)
β’ Metric data for launch vehicle identified for
subsystem categories similar to aircraft
β Propulsion
β Avionics
β Hydraulics
β Structures
14. RMS: Data Mining Approach
β’ Obtained mission records from several SPDMS
databases:
β PRACA β Provided unplanned maintenance action
data
β AGOSS β Provided planned work data
β SFDC β Provided some direct labor data
β’ Surveyed SME community to identify typical
vehicle powered operations by subsystem
β Needed for failure rate calculations
15. RMS: Data Analysis
β’ Challenge: No single STS data system recorded
all the parameters needed to generate desired
metrics
β Operational records in dissimilar systems had
some common identifiers (WAD#)
β By use of a relational database (MS Access) the
interdependencies between the available data
sets were used to synthesize the metric data
16. RMS: Nuggets & Fools Gold
β’ Initial study produced useable data for model
β’ Revisit of study to incorporate additional 3 years
flights found discontinuity in numbers
β Number of Problem Reports dropped by order of
magnitude at STS-xxx
β Cause researched β Operations contract award fee
metrics changed
β’ Fee based on number of PRs β multiple items per PR were
now being recorded to depress the metric
β’ Required update to database schema to βcount the itemsβ
17. Case Study: STS Design Root Cause
Analysis (RCA)
β’ Questions to be answered:
β βWhy does it take so long to
process a vehicle for launch?β
β βWhy does it cost so much to
operate the STS systems?β
18. RCA: Source Data
β’ Study team built upon prior VSP and RMS work
β Used VSP Functional Model of Spaceport Operations
β Incorporated mission reliability analysis data from RMS
study
β Study focused on a year of STS operations
(1997 - 8 missions flown)
β’ Dug deeper into the STS operations processes at KSC
β Obtained template and as-run scheduling system data
(ARTEMIS) for the 8 STS missions conducted
β Engaged KSC engineering community in identification of
system and subsystem(s) driving each individual
operational task
19. RCA: Data Mining & Analysis
β’ Complexity of source data required programing support to produce
useable database records
β’ Interactive relational database forms used in working sessions with
SMEs to capture system / subsystem task knowledge
β Live sessions focused on single mission flow (STS-81)
β Information from that flow batch-processed against other 7 mission
data sets
β’ Unmatched items from batch processing were reassessed with SMEβs via e-
mail file exchange
β’ Flow-unique conditions were identified and documented (OMDP, Roll-
Back, etc)
β’ Limitations:
β Majority of operational performance data only provided task durations
β Study focused on direct vehicle operations, indirect operations
assessment limited to only a few subsystems
20. RCA: Nuggets
β’ Study provided a detailed window
into relationships between
subsystem design trade decisions
and the resulting recurring
cost/cycle time performance metrics
at mature state of system
β Study report published as a NASA
Technical Manual (TPβ2005β211519)
β http://ntrs.nasa.gov/archive/nasa/casi.
ntrs.nasa.gov/20050172128_2005171
687.pdf
β’ Highlights:
β Roughly 40% of recorded operations
were Unplanned Work
β Propulsion and Thermal Protection
systems drove majority of processing
tasks
21. Virgin Ground?
β’ With the conclusion of the STS
program, knowledge from it remains untapped
β Experts are departing for other work
β Records are being archived β some veins of insight
could be lost if not prospected soon
β’ ISS assembly & operations
β In-flight crew operations and maintenance metrics
β Ground support & logistics metrics
β’ Planetary missions
β Mars missions longevity could provide wealth of
ground support operations planning & control metrics
22. Summary
β’ Useful operational performance metric information is often
hidden or buried in seemingly unrelated data sources
β’ Keys to unlocking the hidden knowledge nuggets include:
β Establishing a framework model for data classification
β Identifying categories of performance data sought
β Subject Matter Experts to help sift the fools gold and tailings
from the true valid data
β Relational database tools to filter and aggregate the data as it is
accumulated
β Openness to deductive reasoning β finding the
implications, patterns and especially gaps in the raw data
β Curiosity, optimism and the patience to swirl the pan
repeatedly to capture the fine dust along with the obvious
nuggets