NOAA does an excellent job of generating an disseminating data to meet the primary mission of Preservation of Life and Property. There is an unrealized opportunity to exploit the data for research and profit. Much of the data is hidden deep in archives with community specific portals for access. Modern technologies allow new methods to expose more data to wider audiences in order to stimulate innovation and discovery. NOAA is currently experimenting with cloud technologies through the big data partnership by making high value data sets such as GOES East available on the cloud through cloud provider partners. Specifically: 1. To understand and predict changes in climate, weather, oceans and coasts; 2. To share that knowledge and information with others; and 3. To conserve and manage coastal and marine ecosystems and resources. There is an unrealized opportunity to exploit NOAA?s vast data holdings for research and profit. Much of the data is hidden deep in archives with community specific portals for access. Modern technologies allow new methods to expose more data to wider audiences in order to stimulate innovation and discovery. NOAA is currently experimenting with cloud technologies through the big data partnership by making high value data sets such as GOES East available on the cloud through the partners.

1. Mission Science Network
FOROFFICIALUSEONLY
Transforming Traditional Approaches
Providing access to NOAA distributed data
SAMSI-JPL Workshop
Remote Sensing, Uncertainty Quantification, and the Theory of Data Systems
February 12–14, 2018
California Institute of Technology
Presented by:
Jay Morris
Solutions Architect
Mission Science Network

2. Mission Science Network
FOROFFICIALUSEONLY
Outline
● OSGS, Satellite data and acquisition
● NCEI, Scientific product access and distribution
● NIST Big Data Interoperability Framework
● NOAA Big Data Partnership
● CICS-NC GraphDB proof of concept
2

3. Mission Science Network
FOROFFICIALUSEONLY
OSGS - Satellites
NOAA Environmental Satellite and Information Service
Office of Satellite Ground Systems
Steve Petersen

4. Mission Science Network
FOROFFICIALUSEONLY
NESDIS Ground Enterprise
4

5. Mission Science Network
FOROFFICIALUSEONLY
GOES Ground Segment
Geostationary Operational Environmental Satellites (GOES) Overview
The GOES satellite system has a
geosynchronous orbit about 22,300
miles above the earth. The first
satellite of the GOES system was
launched in December 1966. The
GOES system now consists of three
operational satellites. GOES
provides atmospheric triggers for
severe weather conditions such as
tornadoes, flash floods, hailstorms,
and hurricanes. The satellite imagery
is also used to estimate rainfall
during thunderstorms and hurricanes
for flash flood warnings, as well as to
estimate snowfall accumulations and
overall extent of snow cover.
5

6. Mission Science Network
FOROFFICIALUSEONLY
POES Ground Segment
Polar-orbiting Operational Environmental Satellites (POES)
The POES satellite system
provides visible, infrared, and
microwave data for a variety of
applications such as cloud and
precipitation monitoring,
determination of surface
properties, and humidity profiles.
POES makes polar orbits14 times
per day, approximately 520 miles
above the surface of the Earth,
allowing daily global coverage. The
first satellite of the system was
launched in April 1960 and
consists today of five operational
satellites.
6

7. Mission Science Network
FOROFFICIALUSEONLY
ANTENNAS
Antenna Infrastructure Sustainment Project (AISP)
Ground systems antennas, Low Earth Orbiting Tracker (LEO-T)
or Geostationary Tracker (GT), are the quintessential element of
the NOAA NESDIS infrastructure. All command data to and
sensing data from every satellite mission pass through this critical
part of the ground system. The NOAA NESDIS antenna
infrastructure includes 12 LEO-T, 14 GT, and 12 Fixed positioned.
The NOAA NESDIS antennas range in size from 1.2M to 26M in
diameter to determine global sea-level rise, ocean currents, wind
speed, ocean circulation, and other ocean-related altimetry
products. Jason-3 which was launched in July 2015, will
succeeded Jason-2.
All NOAA NESDIS antennas are part of the antenna infrastructure
refresh (AIR) process that not only assures basic mission
sustainment but also extends the systems' life expectancy.
Several of NOAA's antennas have been operating for nearly 50
years and all are operating beyond the normal and customary 20-
year life to provide an optimum value to the US taxpayer.
To extend the serviceable life of earth terminal equipment, major
maintenance or periodic overhaul is required. Major maintenance
entails maintenance that is considered too extensive or of much
greater scope than can be accomplished by the operations
personnel.
Today, six new GTs are planned for GOES-R and three Fixed to
support the GOES Data Collection System (DCS).
7

8. Mission Science Network
FOROFFICIALUSEONLY
ESPDS
NESDIS Environmental Satellite Processing and Distribution (ESDS)
To support satellite missions, the
Environmental Satellite Processing Center
(ESPC) in OSPO provides ingest of telemetry
data from geostationary and polar orbiting
systems and satellite product generation and
distribution services. The ESPDS is ESPC’s
next generation system for the delivery of
these services using an enterprise approach
across the various architectural segments.
The objective is to evolve the ESPDS into an
integrated enterprise ground system capable
of meeting technical and performance
requirements of future and current satellite
ground processing systems.
8

9. Mission Science Network
FOROFFICIALUSEONLY
Comprehensive Large Array-data Stewardship System (CLASS)
The CLASS system supports all NOAA missions and goals, and
supports NOAA’s crosscutting priority to provide an integrated
data environment and data management system for NOAA.
CLASS is NOAA's premiere on-line facility for the distribution of
data products and derived data from NOAA’s satellite systems,
including NOAA’s polar-orbiting and geostationary environmental
satellite systems and their follow-on programs.
• Environmental Data from Polar-orbiting Satellites
• Environmental Data from Geostationary Satellites
• Defense Meteorological Satellite Program (DMSP)
• Suomi National Polar-orbiting Partnership (NPP)
• Sea Surface Temperature data (SST)
• RADARSAT
• Altimetry / Sea Surface Height Data (JASON)
• Global Navigation Satellite Systems (GNSS)
• Other - Miscellaneous products in CLASS
9

10. Mission Science Network
FOROFFICIALUSEONLY
NCEI – Mission Science Network
National Centers For Environmental Information
Robert Bunge
Ken Casey
Scott Hausman

11. Mission Science Network
FOROFFICIALUSEONLY
Problem Statement
NCDC.NOAA.GOV
23 Quick Links to different
applications
Climate.gov
337 Content types to
various applications
CLASS
77 Products , 9 Categories
Data products are managed independently with unique software, systems, and workflows
which requires users to be familiar with muliple access systems, datatypes, and ingest models
Nomads
CDR
LSR
CSPC
CPC SST
NEXRAD
CLASS
Paleo
Ingest
Access
Ingest
Access
Ingest
Access
Ingest
Access
Ingest
Access
Ingest
Access
11

12. Mission Science Network
FOROFFICIALUSEONLY
Big Data Interoperability Framework
12

13. Mission Science Network
FOROFFICIALUSEONLY
Enterprise Segments
Common Metadata
Repository
Contains information on the quality, type,
and location of the data
Tiered Storage
Search, Discovery, and
Access
Data collection, preparation,
and Ingest
13

14. Mission Science Network
FOROFFICIALUSEONLY
OneStop to Mission Data Management System
Data Managers
Inventory Manager
M/D
Event
Log
OneStop
UI
Public
Thin
Portals
OneStop API/
Search
Engine
External
Systems
Co
M
ET
E
M
M
A
Collection Metadata Manager
Collection
M/D
CEdit CoMET Metaserver
Rubrics DocuComp
Common
Ingest
Data
Streams
Data
Operators
Disk
CloudTape
Data and Granule
Information
Collection
Information
Discovery
Services
Areas with thick lines being built by OneStop - Simplified View, Detailed View Next Slide
14
workflows
Storage and Order
Fulfillment
Index

15. Mission Science Network
FOROFFICIALUSEONLY
NOAA – Big Data Partnership
NOAA ACIO
Ed Kearns

16. Mission Science Network
FOROFFICIALUSEONLY
AMS Annual Meeting, Austin, TX Jan 10, 2018
Making Data Available on the Cloud for
Decision Support Applications through
NOAA's Big Data Project
E. J. Kearns*, S. Glass*, O. Brown^, J. Brannock^,
A. Simonson*, and J. O'Neil*
NOAA Big Data Project
*NOAA ^CICS-NC
NOAA's Big Data Project
16

17. Mission Science Network
FOROFFICIALUSEONLY
Big Data Project Basics
•Experiment with Industry, via Cooperative Research and Development
Agreements
▪ 5 separate, identical agreements – extending to Apr 2019
• Industry provides access to NOAA’s open data to all
▪ Data remain open, are not to be sold
▪ Collaborators monetize services based on data
• Uses open data to combine 3 powerful resources:
1. NOAA’s science and subject matter expertise
2. Industry’s data storage and cloud expertise
3. Industry’s user-focused tools and processing
17

18. Mission Science Network
FOROFFICIALUSEONLY
NOAA Big Data Project
One-way transfer out of federal
systems. Only a trusted user
inside security boundary
Distributing a single copy of data
can support all users.
How to best provide cloud access to NOAA data?
18

19. Mission Science Network
FOROFFICIALUSEONLY
Industry Builds What is Needed to Support its Users
Uses
Decision
Support Tools
Cloud Platforms
NOAA Data and Expertise
3rd Parties
Information Consumers
19

20. Mission Science Network
FOROFFICIALUSEONLY
Big Data Project Collaborators’ Offerings
● AWS
○ https://aws.amazon.com/noaa-big-data/
● Google Cloud Platform
○ https://cloud.google.com/bigquery/public-data/
○ https://explorer.earthengine.google.com/#index
● IBM
○ https://noaa-crada.mybluemix.net/
● Microsoft
○ No public services to date
● Open Commons Consortium
○ http://edc.occ-data.org/
35+ NOAA datasets are now available

21. Mission Science Network
FOROFFICIALUSEONLY
Over 35 NOAA Datasets are now
1. Aerosols - AVHRR AOT - Climate Data Records
2. CFSv2 - Climate Forecast System
3. Cloud Properties - PATMOS Climate Data Record
4. Cooling Degree Day weekly summary
5. Daily Summaries/Global Historical Climate Network (GHCN-D)
6. Essential Fish Habitat
7. Geostationary Infrared Brightness Temperatures - GRIDSAT B1 - Climate Data Records
8. Global Surface Summary of the Day (GSOD)
9. GOES -16
10. Hurricane Tracks – Int’l Best Track Archive for Climate Stewardship (IBTrACS)
11. Int’l Comprehensive Ocean-Atmosphere Data Set (ICOADS)
12. Land and Ocean Surface - ETOPO1
13. Monthly Global Historical Climate Network (GHCN-M)
14. National Water Model - forecasts
15. National Water Model - reanalysis
21

22. Mission Science Network
FOROFFICIALUSEONLY
16. Near Ocean Surface Atm Properties - Climate Data Record
17. Near-Surface Weather - Realtime Mesoscale Analysis RTMA
18. NEXRAD L2
19. Nighttime Lights - VIIRS Day/Night Band - Monthly
20. Nighttime Lights - DMSP
21. NMFS Observer
22. NMFS Trawl
23. Normalized Difference Vegetation Index - AVHRR NDVI - Climate Data Records
24. Ocean Surface Fluxes - Climate Data Records
25. Pathfinder SST
26. Precipitation - PERSIANN - Climate Data Records
27. RAP Rapid Refresh
28. Sea Level Pressure Reanalysis
29. Sea Surface Temperature - OISST - Climate Data Record
30. Sea Surface Temperature - WHOI V2 - Climate Data Records
Over 35 NOAA Datasets are now
22

23. Mission Science Network
FOROFFICIALUSEONLY
31. SPC Storm Reports
32. Surface Reflectance - AVHRR Albedo - Climate Data Records
33. Surface Temperature Reanalysis
34. Vegetation - Leaf Area Index and FAPAR - Climate Data Records
35. Water Vapor Reanalysis
36. Weather Forecasts - GFS
23
Over 35 NOAA Datasets are now

24. Mission Science Network
FOROFFICIALUSEONLY
CICS – GraphDB Proof of Concept
Cooperative Institute for Climate and Satellites
Jim Baird
Linda Copely

25. Mission Science Network
FOROFFICIALUSEONLY
Graph Database Proof-of-Concept
Federated Archive Search Tool (FAST)
A cross-dataset search and discovery tool proof-of-concept
based on ‘Big Data’ graph database technology.
OBJECTIVES:
- Demonstrate use of disparate datasets in a graph database
(VIIRS SDR, Storm Events).
- Develop and implement use cases for granule-level search
and discovery.
- Demonstrate granule-level search and discovery capabilities.
25

26. Mission Science Network
FOROFFICIALUSEONLY
FAST Topography
26

27. Mission Science Network
FOROFFICIALUSEONLY
Federated Archive Search Tool (FAST)
Proof-of concept for granule-level search tool across multiple datasets (Satellite, In-Situ, CDR).
Neo4j Spatial plug-in
FIPS State/County/Features database
NCDC Storm Events database
VIIRS SDR
time geo
event
qual
loc
Graph Database Proof-of-Concept
Neo4j Graph database
27

28. Mission Science Network
FOROFFICIALUSEONLY
Graph Database Proof-of-Concept
VIIRS Geography
Polygon: Lat/Lon
VIIRS Time Scale
85 second granules
Storm Events Geography
FIPS Codes
Lat/Lon (sparsely)
Point
Line
Place
Storm Events Time Scale
Discrete point
Time range
28

29. Mission Science Network
FOROFFICIALUSEONLY
Graph Database Proof-of-Concept
VIIRS Attributes
Descriptive
Day/Night Flag
Percent Cloud Cover
Percent Land (vs. Ocean)
Quality
Percent Error
Percent Missing
Percent NA
Status
Summary Quality
Storm Events Attributes
Event Type
Damage Type
Damage Amount
Deaths
Injuries
Reporting Source
Wind Speed
29

30. Mission Science Network
FOROFFICIALUSEONLY