The Big Data Project is an innovative approach to publishing NOAA’s vast data resources and positioning them near cost-efficient high performance computing, analytic, and storage services provided by the private sector. This collaboration combines three powerful resources - NOAA’s tremendous volume of high quality environmental data and advanced data products, private industry’s vast infrastructure and technical capacity, and the American economy’s innovation and energy - to create a sustainable, market-driven ecosystem that lowers the cost barrier to data publication. This project will create a new economic space for growth and job creation while providing the public far greater access to the data created with its tax dollars.

1. The
NOAA
Big Data Project
Overview
Amy Gaskins, Project Director
February 2016

2. What is the Big Data Project?
• An innovative approach to publishing NOAA’s vast data
resources and positioning them near cost-efficient high
performance computing, analytic, and storage services
provided by the private sector
• High-quality environmental data at low cost
• Increased public access to taxpayer-funded data
• Opportunities for new data products and services

3. Why Start Now?
• NOAA’s projected data
growth is exponential
• By 2020, we’ll have 160
petabytes of archived
data
• The rate of data capture
will only continue to
increase

4. Why a CRADA?
• Existing NOAA infrastructure and funding cannot support
growing amount of data or level of external demand
• CRADA is a low-risk mechanism for creating and testing a
smaller version of the full market ecosystem
• Provides an iterative approach to dissemination without
disrupting NOAA operations
• Allows for lessons learned and real-time modifications to the
dissemination process

5. Why Cloud Services?

6. The BDP
Dissemination Model
Traditional Data Request Big Data Project

7. Important CRADA
Rules of the Road
• Valid for three years with annual renewal option
• Collaborators and/or NOAA may choose to terminate with
30 days’ notice
• Collaborators have non-exclusive access to NOAA data
• All NOAA data is up for discussion (excluding ITAR-restricted
and national security sensitive data)
• Collaborators must provide users with equal access to
NOAA data on equal terms

8. The
partnership
model…
• Allows access to the entire
historical archive of large
datasets
• Uses a market to provide
choices to industry without
NOAA interference
• Increases the reach of
NOAA data
• Frees up NOAA personnel
from repeating data extracts

9. Reducing Operational Time at
The Climate Corporation

10. Reducing Operational Time at
The Climate Corporation

11. Reducing Operational Time at
The Climate Corporation
• Project timelines at TCC are now several weeks shorter
• TCC can evaluate new forecasting models on larger datasets
• TCC reduces project cost because they only pay for
temporary cloud storage and compute cycles
• NOAA improves archive through resolution of data issues

12. Types of NOAA Data

13. NEXRAD
Doppler Radar

14. NEXRAD
Doppler Radar
• Original resolution (Level II) available
now
• Precipitation and atmospheric
movement (wind)
• Tornado visualization
• Cold fronts
• Thunderstorm gusts
• Rainfall rate/hydrological forecasting

15. VIIRS
Infrared Satellite
• Covers the entire globe every 14 hours
• Enables short-term monitoring of:
• Wildfires
• Drought
• Vegetation
• Ocean temperature

16. VIIRS
Infrared Satellite
• Assists in long-term assessment of:
• Emerging global storm patterns
• Global economic activity
• Arctic and ocean ice

17. IOOS
Ocean Observation
• 10,000+ oceanographic
data sets
• Buoys
• High-frequency
radar
• Water level gauges
• Gliders
• Animal telemetry
• Coastal and estuary
stations

18. Questions?

19. Contact Information
Email: Amy.Gaskins@noaa.gov
Twitter: @AmyVGaskins
LinkedIn: www.linkedin.com/in/amygaskins