This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) and netCDF (network Common Data Form) are two widely used data formats for storing and manipulating scientific data. The NetCDF format also supports temporal data by using multidimensional arrays. The basic structure of data in this format and how to work with it will be covered in the context of standardized data structures and conventions. This slide will demonstrate the tools and techniques for ingesting HDF and netCDF data efficiently in ArcGIS, as well as some common workflows to employ the visualization capabilities of ArcGIS for effective animation and analysis of your data.
Handwritten Text Recognition for manuscripts and early printed texts
Working with HDF and netCDF Data in ArcGIS
1. Working with HDF and netCDF Data in ArcGIS
Tools and Case Studies
Nawajish Noman
Jeff Donze
HDF and HDF-EOS Workshop XIV, September 28-30, 2010, Champaign, IL
HDF28-
2. Outline
• HDF and netCDF in ArcGIS
• Time in ArcGIS
• Performing Analysis
• Use Cases and Applications
• Script Tools
•F t
Future Directions
Di
ti
4. Scientific Data and ESRI
• Direct support - NetCDF and HDF
• THREDDS – a data server technology for multidimensional array
data, integrated use by our customers
• Examples using ESRI technology
• National Climate Data Center
• National Weather Service
• National Center for
Atmospheric Research
• Air Force Weather
• Australian Navy
• Australian Bur.of Met.
• UK Met Office
7. HDF4 and HDF5 Support in ArcGIS
http://help.arcgis.com/en/arcgisdesktop/10.0/help/
8. What is NetCDF?
NetCDF?
• NetCDF (network Common Data Form)
network
A platform independent format for representing multi-dimensional
multiarrayarray-orientated scientific data.
• Self Describing: a netCDF file includes information about the data it
contains.
• Di
Direct A
t Access: a small subset of a large dataset may be accessed
ll b t f l
d t
t
b
d
efficiently, without first reading through all the preceding data.
• Sh
Sharable: one writer and multiple readers may simultaneously
bl
it
d
lti l
d
i lt
l
access the same netCDF file.
NetCDF is relatively new to the GIS community but widely used by scientific
communities for many years.
9. Storing Data in a netCDF File
Time = 1 to 3
Y = 1 to 4
X = 1 to 4
10. NetCDF Support in ArcGIS
• ArcGIS reads/writes netCDF since version 9.2
• An array based data structure for storing
multidimensional data.
T
• N-dimensional coordinates systems
• X, Y, Z, time, and other dimensions
• Variables – support for multiple variables
• Temperature, humidity, pressure, salinity, etc
Temperature,
• Geometry – implicit or explicit
• Regular grid (implicit)
(implicit)
• Irregular grid
• Points
Y
Z
X
12. Ingesting netCDF data in ArcGIS
• NetCDF data is accessed as
Raster
• Feature
• Table
•
• Di
Direct read
d
• Exports GIS data to netCDF
13. CF Convention
Climate and Forecast (CF) Convention
(CF)
http://cf-pcmdi.llnl.gov/
htt // f
di ll l
/
Initially developed for
• Climate and forecast data
• Atmosphere, surface and ocean model-generated data
model• Also for observational datasets
• The CF conventions generalize and extend the COARDS (Cooperative
cean/A
Ocean/Atmosphere Research Data Service) convention.
• CF i now th most widely used conventions f geospatial netCDF
is
the
t id l
d
ti
for
ti l tCDF
data. It has the best coordinate system handling.
14. NetCDF and Coordinate Systems
• Geographic Coordinate Systems (GCS)
X dimension units: degrees_east
• Y dimension units: degrees_north
•
• Projected Coordinate Systems (PCS)
X dimension standard_name: projection_x_coordinate
standard_name:
• Y dimension standard_name: projection_y_coordinate
standard_name:
• V i bl has a grid_mapping attribute.
Variable h
id
i
tt ib t
• CF 1.4 conventions currently supports twelve predefined coordinate
systems (Appendix F: Grid Mappings)
Mappings)
•
• Undefined
• If not GCS or PCS
• ArcGIS writes (and recognizes) PE String as a variable attribute.
15. NetCDF Tools
Toolbox: Multidimension Tools
Make NetCDF Raster Layer
• Make NetCDF Feature Layer
• Make NetCDF Table View
•
Raster to NetCDF
• Feature to NetCDF
• Table to NetCDF
•
•
Select by Dimension
y
18. Using NetCDF Data
Behaves the same as any layer or table
• Display
• Same display tools for raster and feature layers will work on netCDF
raster and netCDF feature layers.
• Graphing
• Driven by the table just like any other chart.
• Animation
• Multidimensional data can be animated through a dimension (e.g.
time, pressure, elevation)
• Analysis Tools
• A netCDF layer or table will work just like any other raster layer,
feature layer or table (e g create buffers around netCDF points
layer,
table. (e.g.
points,
reproject rasters, query tables, etc.)
rasters,
22. Temporal GIS Patterns
Dynamic
Discrete
Stationary
Change
something that moves
something that
“just happens”
“j t h
”
stands still but
records changes
change or growth
• Planes
• Vehicles
• Animals
• Satellites
• Storms
• Crimes
• Weather Stations
• Population
• Lightning
• Accidents
• Traffic Sensors
• Distribution
• Fire Perimeter
23. GIS Integration of Time
New Ways to Manage Visualize & Analyze Geography
Manage,
Multi
Dimensional
Data (netCDF)
(netCDF)
Visualize Change
Vi
li Ch
T
Real Time
Sensor Network
y
x
Files
DBMS
Mobile
Stationary
T1
Simulation
Modeling
24. Time is now built-in to ArcGIS
built-
•
Simple Temporal Mapping
•
Unified experience for Time
•
•
•
Configure time properties on the layer
Use Time Slider to visualize temporal data
is ali e
Share temporal visualization
TimeTime-enabled Map Services
Export videos or images
Generate temporal map books
using ArcPy scripting
• L
Layer and map packages
d
k
•
•
•
25. Visualizing temporal data using graphs
• Create a graph using a layer or table
• Create an animation in the usual way, attaching the
layer or table to a time layer track
• When the animation is played,
p y ,
the graph will animate
28. Spatial and Temporal Analysis
• Several hundreds analytical tools available for raster, features,
and table
• Temporal Modeling
p
g
• Looping and iteration in ModelBuilder and Python
29. Generate Rainfall Statistics
• Calculates specified statistics for all time steps
• Outputs a raster catalog
• Optionally outputs a netCDF file
30. Generate Rainfall Statistics Table
• Calculates statistics for all time steps
• Outputs a table
• Optionally creates a graph
32. Use Cases and Applications
•
•
•
•
•
Hydrography and METOC Branch, Royal Australian Navy
Applied Science Associated, Inc.
Inc.
The Nature Conservancy
The University of Washington
The University of Southern Mississippi
33. Observations and Model Forecasts
• Near real-time observations
real•
•
•
•
•
Weather satellite imagery
Sea Surface Temperature
Significant Wave Height (altimeter)
Ocean Winds
Moisture and Precipitation
• Numerical model forecasts
• Fixed domain Global, Regional, Tropical and Local atmospheric models
(ACCESS)
• Fixed domain Global, Regional, and Local wave models (WaveWatch 3)
(WaveWatch
• Global Ocean Model (BLUElink)
(BLUElink)
highg
p
• Relocatable high-resolution atmospheric and ocean models
Source: Hydrography and METOC Branch, Royal Australian Navy.
34. Climatology – CCMP Winds
Source: Hydrography and METOC Branch, Royal Australian Navy.
35. Climatology – CCMP Winds
Source: Hydrography and METOC Branch, Royal Australian Navy.
36. Ocean Model Forecast – Custom Display
Source: Hydrography and METOC Branch, Royal Australian Navy.
37. Web Services - GHRSST
Source: Hydrography and METOC Branch, Royal Australian Navy.
38. Environmental Data Connector (EDC)
XML
DAP
THREDDS
Features
NetCDF
Java App,
Unidata
Rasters
ArcGIS
Extension
Source: Applied Science Associated, Inc.
39. EDC Screen and Time Slider Toolbar
Synchronize and animate time-varying data in
http://www.pfeg.noaa.gov/products/EDC/
http://www.asascience.com/TimeSlider/index.htm
Source: Applied Science Associated, Inc.
44. Community Developed Tools
• Geoprocessing Resource Center
http://resources.arcgis.com/geoprocessing/
• Marine Geospatial
Ecology Tools
• Developed at Duke Univ.
• Over 180 tools for import
management, and
analysis of marine data
• Australian Navy tools
(not publicly available)
45. Script Tools of interest
• Python is used to build custom tools for specific tasks or
datasets
46. NEXRAD Geoprocessing Tools
•
•
•
•
•
•
Currently 6 geoprocessing script tools
Designed to work with NEXRAD netCDF file
Can be easily modified for other datasets
Customized tools for various workflows
Simplify repetitive work
Automate GIS processes
51. Things to Consider…
• Embrace the Common Data Model (netCDF, HDF etc.)
(netCDF,
• Use Data and metadata standards (OGC, CF etc)
• Make your data “spatial” (by specifying geographic or a
p j
projected coordinate system)
y
)
• Clearly define workflow and requirements
• Create sample tools where possible
52. Tools Under Consideration
• Temporal Statistics
• Get Variable Statistics
Analysis
•
•
•
•
Clip
p
Extract By Variable
Extract By Dimension
Append By Dimension
Data Management
53. Future directions
• Multidimension data management
• Temporal analysis tools
• Additional support for HDF5 using netCDF 4.x library
• What do you need?
55. Sharing Temporal Maps & Data with ArcGIS 10
•
Publish time-aware maps
time-
•
Export videos or images, layer and map packages
•
Vi
li data
Visualize d t
Access via REST API
Web API
• FLEX
• JavaScript
• Silverlight
• Time Slider web control
•
•
1979
56. How to store temporal data?
• DATE is a special field type specific to time
• GeoDatabase provides DATE
• If at all possible – use DATE type
• DATE field should be indexed for faster query performance
• Numeric and String fields
•
•
•
•
YYYY
YYYYMM
YYYYMMDD
YYYYMMDDhhmmss
YYYY/MM/DD
YYYY/MM/DD hh:mm:ss
• Time dimension in a netCDF variable
YYYY-MM-DD
YYYY-MMYYYY-MM-DD hh:mm:ss
YYYY-MM-
57. Animation examples
Data source: Cavalieri, D., C. Parkinson, P. Gloerson,
and H.J. Zwally. 1996, updated 2005. Sea ice
concentrations from Nimbus-7 SMMR and DMSP SSM/I
Nimbuspassive microwave data, June to September 2001.
Boulder, CO, USA: National Snow and Ice Data Center.
Sea ice concentration
Solar Insolation
Data provided courtesy of Declan Butler - http://declanbutler.info/blog/
Stream flow analysis
Avian Influenza