SlideShare ist ein Scribd-Unternehmen logo
1 von 38
HDF and netCDF Data Support in ArcGIS

Nawajish Noman
Jeff Donze

HDF and HDF-EOS Workshop XV, April 17-19, 2012, Riverdale, MD
Outline

•
•
•
•
•

HDF and netCDF in ArcGIS
Time in ArcGIS
Performing Analysis
Python Tools
Future Directions
HDF and netCDF in ArcGIS
Scientific Data and Esri

• Direct support - NetCDF and HDF

• THREDDS/OPeNDAP – a framework for scientific data networking,
integrated use by our customers
• Examples using Esri technology
• National Climate Data Center
• National Weather Service
• National Center for
Atmospheric Research
• U. S. Navy (NAVO)
• Air Force Weather
• Australian Navy
• Australian Bur.of Met.
• UK Met Office
HDF4 and HDF5 Support in ArcGIS
Raster Concepts

• Raster Format (e.g. img, tif, etc.)
– driver level support / storage format and layout / read/write of pixels and metadata

• Raster Type (e.g. GeoEye-1)
– implies Raster Format support and are format and sensor specific
– intelligent use of metadata and other sensor specific parameters
– defines georeferencing and well known processing chains

• Raster Product (e.g. Panchromatic, Multispectral, Pansharpened )
– templates which make it easy to work with well defined end user products
– multiple per sensor
– e.g. Panchromatic, Multispectral, Pansharpened

• Raster Product Definition (e.g. Natural Color, False Color)
– defines “how you want your Mosaic Dataset to look” regardless of multiple source sensors
and band combinations
– uses metadata such as wavelength information to map bands
HDF Raster Support
Raster Concept

ArcGIS 10.1 Support

Raster Format

 HDF4
• read: open a HDF subdataset as a Raster Dataset
• write: APIs available but not exposed in UI
 HDF5
• read: open a HDF subdataset as a Raster Dataset
• write: not supported at this time

Raster Type

 HDF4, HDF5
• direct ingest of one or many HDF subdatasets into a
Mosaic Dataset using the Raster Dataset Raster
Type or the Table Raster Type

* Esri interested in discussing other Raster Types
Raster Product

* Esri interested in discussing other Raster Products

Raster Product Definition

* Esri interested in discussing other Raster Product
Definitions
HDF Raster Support
• 10.1 Raster Format and Raster Type support implies…
– ArcGIS Desktop
• direct use as a Raster Dataset or Mosaic Dataset
• use via conversion (i.e. convert to another format)
• feature rich use in the applications
– Visualization and Mapping
– Geoprocessing

– ArcGIS Server
• publishing as dynamic image services
• caching and publishing as tile services (i.e. basemaps)
• OGC (WCS, WMS, WMTS)
Displaying MODIS LST Data
HDFView

ArcGIS
NetCDF Support in ArcGIS

• ArcGIS reads/writes netCDF since version 9.2

• An array based data structure for storing
multidimensional data.

T

• N-dimensional coordinates systems
• X, Y, Z, time, and other dimensions

• Variables – support for multiple variables
• Temperature, humidity, pressure, salinity, etc

• Geometry – implicit or explicit
• Regular grid (implicit)
• Irregular grid
• Points

Y

Z
X
Gridded Data

Regular Grid

Irregular Grid
Ingesting netCDF data in ArcGIS

• NetCDF data is accessed as
•

Raster
• Feature
• Table

• Direct read
• Exports GIS data to netCDF
CF Convention
Climate and Forecast (CF) Convention
http://cf-pcmdi.llnl.gov/

Initially developed for
• Climate and forecast data
• Atmosphere, surface and ocean model-generated data
• Also for observational datasets
• The CF conventions generalize and extend the COARDS (Cooperative
Ocean/Atmosphere Research Data Service) convention.
• CF is now the most widely used conventions for geospatial netCDF
data. It has the best coordinate system handling.
NetCDF and Coordinate Systems

• Geographic Coordinate Systems (GCS)
•

X dimension units: degrees_east
• Y dimension units: degrees_north

• Projected Coordinate Systems (PCS)
•

X dimension standard_name: projection_x_coordinate
• Y dimension standard_name: projection_y_coordinate
• Variable has a grid_mapping attribute.
• CF 1.6 conventions currently supports thirteen predefined
coordinate systems (Appendix F: Grid Mappings)

• Undefined
• If not GCS or PCS
• ArcGIS writes (and recognizes) PE String as a variable attribute.
NetCDF Tools

Toolbox: Multidimension Tools
•

Make NetCDF Raster Layer
• Make NetCDF Feature Layer
• Make NetCDF Table View
•

Raster to NetCDF
• Feature to NetCDF
• Table to NetCDF
•

Select by Dimension
NetCDF Layer/Table Properties

Raster
Feature

Table
Changing Time Slice
143

342

341

441

131

231

331

431

121

Time = 1

241

221

321

211

311

232

332

433

223

323

423

213

313

413

432

122

222

322

422

112

212

312

412

421

111

333

442

132
141

233

113

242

443

123

142

343

133

Y

243

411

X

Time
Using NetCDF Data
Behaves the same as any layer or table
• Display
• Same display tools for raster and feature layers will work on netCDF
raster and netCDF feature layers.

• Graphing
• Driven by the table just like any other chart.

• Animation
• Multidimensional data can be animated through a dimension (e.g.
time, pressure, elevation)

• Analysis Tools
• A netCDF layer or table will work just like any other raster layer,
feature layer, or table. (e.g. create buffers around netCDF points,
reproject rasters, query tables, etc.)
Time in ArcGIS
Time is now built-in to ArcGIS

•

Simple Temporal Mapping

•

Unified experience for Time
•

Configure time properties on the layer
• Use Time Slider to visualize temporal data
•

Share temporal visualization
•
•
•
•

Time-enabled Map Services
Export videos or images
Generate temporal map books
using ArcPy scripting
Layer and map packages
Animation examples

1979
Performing Analysis
Spatial and Temporal Analysis

• Several hundreds analytical tools available for raster, features,
and table
• Temporal Modeling
• Looping and iteration in ModelBuilder and Python
Generate Rainfall Statistics

• Calculates specified statistics for all time steps
• Outputs a raster catalog
• Optionally outputs a netCDF file
Generate Rainfall Statistics Table

• Calculates statistics for all time steps
• Outputs a table
• Optionally creates a graph
Python Tools
Community Developed Tools

• Geoprocessing Resource Center
http://resources.arcgis.com/geoprocessing/

• Marine Geospatial
Ecology Tools (MGET)
• Developed at Duke Univ.
• Over 180 tools for import
management, and
analysis of marine data

• Australian Navy tools
(not publicly available)
Sample Script Tools

• Python is used to build custom tools for specific tasks or
datasets
NEXRAD Geoprocessing Tools

•
•
•
•
•
•

Currently 6 geoprocessing script tools
Designed to work with NEXRAD netCDF file
Can be easily modified for other datasets
Customized tools for various workflows
Simplify repetitive work
Automate GIS processes
Extract NEXRAD Rainfall To Points

• Extracts the cell values for all time steps
• Outputs a feature class
New NetCDF Tools (under development)

• Download NetCDF File (OPeNDAP, WCS)
• Clip
• Extract By Variable
• Extract By Dimension
• Append By Dimension
• Variable Statistics
• Temporal Statistics
Download NetCDF File (WCS/OPeNDAP)
Dependencies on 3rd Party Utilities
• netcdf4-python
•

This module can read and write files in both the new netCDF 4 and
the old netCDF 3 format, and can create files that are readable by
HDF5 clients.

• Pydap
•

Pydap is a pure Python library implementing the Data Access
Protocol, also known as DODS or OPeNDAP. OWSLib

• OWSLib (OGC Web Service utility library)
•

Package for working with OGC map, feature, and coverage services.
OWSLib provides a common API for accessing service metadata and
wrappers for GetCapabilities, GetMap, and GetFeature requests.
Future Directions
HDF and Swath
Scientific Data Workshop and Future Initiatives…
• Esri hosted a workshop in February 2012
• To understand the future need for scientific data support in ArcGIS
• Ongoing efforts - require close collaboration with all of you
• Some of the future initiatives are:
• Continue to support netCDF classic and netCDF4
•
•
•
•

Provide better support for HDF5
Provide tool to consume data served using THREDDS/OPeNDAP
Continue to support the evolving CF convention
Support a strong developer experience for netCDF and HDF using
Python

• What else?
Things to Consider…

• Embrace the Common Data Model (netCDF, HDF etc.)
• Use Data and metadata standards (OGC, CF etc)
• Provide “mechanism” so that we can access scientific data
using a single set of APIs….
• and can expect data to be CF complainant
• Make your data “spatial” (by specifying geographic or a
projected coordinate system)
• Clearly define workflow and requirements
• Create sample tools where possible
Questions?
Nawajish Noman
Team Lead
nnoman@esri.com

Jeff Donze
Esri Federal Business Development
jdonze@esri.com

Weitere ähnliche Inhalte

Was ist angesagt?

Ground truth survey and spatial data analysis (using ILWIS)
Ground truth survey and spatial data analysis (using ILWIS)Ground truth survey and spatial data analysis (using ILWIS)
Ground truth survey and spatial data analysis (using ILWIS)
Serge Rafanoharana
 

Was ist angesagt? (20)

Future of GIS, Moving to the Enterprise Platform
Future of GIS, Moving to the Enterprise PlatformFuture of GIS, Moving to the Enterprise Platform
Future of GIS, Moving to the Enterprise Platform
 
Apache HBase™
Apache HBase™Apache HBase™
Apache HBase™
 
GIS file types
GIS file typesGIS file types
GIS file types
 
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangApache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
 
Land Information System Of Nepal (LIS)
Land Information System Of Nepal (LIS)Land Information System Of Nepal (LIS)
Land Information System Of Nepal (LIS)
 
Meta data
Meta dataMeta data
Meta data
 
Lecture 1b introduction to arc gis
Lecture 1b  introduction to arc gisLecture 1b  introduction to arc gis
Lecture 1b introduction to arc gis
 
Using deep learning in remote sensing
Using deep learning in remote sensingUsing deep learning in remote sensing
Using deep learning in remote sensing
 
Land Information System
Land Information SystemLand Information System
Land Information System
 
Concepts Of Land Information Systems Rua 2006
Concepts Of Land Information Systems Rua 2006Concepts Of Land Information Systems Rua 2006
Concepts Of Land Information Systems Rua 2006
 
Parquet and AVRO
Parquet and AVROParquet and AVRO
Parquet and AVRO
 
Ground truth survey and spatial data analysis (using ILWIS)
Ground truth survey and spatial data analysis (using ILWIS)Ground truth survey and spatial data analysis (using ILWIS)
Ground truth survey and spatial data analysis (using ILWIS)
 
Spark Compute as a Service at Paypal with Prabhu Kasinathan
Spark Compute as a Service at Paypal with Prabhu KasinathanSpark Compute as a Service at Paypal with Prabhu Kasinathan
Spark Compute as a Service at Paypal with Prabhu Kasinathan
 
Geographic information system
Geographic information system Geographic information system
Geographic information system
 
Geospatial Technology for Sustainable Development
Geospatial Technology for Sustainable DevelopmentGeospatial Technology for Sustainable Development
Geospatial Technology for Sustainable Development
 
Netezza vs Teradata vs Exadata
Netezza vs Teradata vs ExadataNetezza vs Teradata vs Exadata
Netezza vs Teradata vs Exadata
 
Applications of Arc GIS
Applications of Arc GISApplications of Arc GIS
Applications of Arc GIS
 
QGIS Module 4
QGIS Module 4QGIS Module 4
QGIS Module 4
 
LRMIS Punjab
LRMIS PunjabLRMIS Punjab
LRMIS Punjab
 
GIS Modeling
GIS ModelingGIS Modeling
GIS Modeling
 

Andere mochten auch

Andere mochten auch (20)

Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesWorking with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
 
NetCDF and HDF5
NetCDF and HDF5NetCDF and HDF5
NetCDF and HDF5
 
Using HDF5 and Python: The H5py module
Using HDF5 and Python: The H5py moduleUsing HDF5 and Python: The H5py module
Using HDF5 and Python: The H5py module
 
Whats New in ArcGIS 10.1
Whats New in ArcGIS 10.1Whats New in ArcGIS 10.1
Whats New in ArcGIS 10.1
 
Relationship chart gordon henry kraft:john cheney
Relationship chart gordon henry kraft:john cheneyRelationship chart gordon henry kraft:john cheney
Relationship chart gordon henry kraft:john cheney
 
GIS as a Platform by Sam Viana (Esri Inc)
GIS as a Platform by Sam Viana (Esri Inc)GIS as a Platform by Sam Viana (Esri Inc)
GIS as a Platform by Sam Viana (Esri Inc)
 
Status of HDF-EOS, Related Software and Tools
 Status of HDF-EOS, Related Software and Tools Status of HDF-EOS, Related Software and Tools
Status of HDF-EOS, Related Software and Tools
 
Granules Are Forever
Granules Are ForeverGranules Are Forever
Granules Are Forever
 
Images of HDF5
Images of HDF5Images of HDF5
Images of HDF5
 
HDF Tools Tutorial
HDF Tools TutorialHDF Tools Tutorial
HDF Tools Tutorial
 
HDF & HDF-EOS Data & Support at NSIDC
HDF & HDF-EOS Data & Support at NSIDCHDF & HDF-EOS Data & Support at NSIDC
HDF & HDF-EOS Data & Support at NSIDC
 
HDF Group Support for NPP/NPOESS/JPSS
HDF Group Support for NPP/NPOESS/JPSSHDF Group Support for NPP/NPOESS/JPSS
HDF Group Support for NPP/NPOESS/JPSS
 
HDF Tools Updates and Discussions
HDF Tools Updates and DiscussionsHDF Tools Updates and Discussions
HDF Tools Updates and Discussions
 
HDF OPeNDAP Project Update and Demo
HDF OPeNDAP Project Update and DemoHDF OPeNDAP Project Update and Demo
HDF OPeNDAP Project Update and Demo
 
HDF4 Mapping Project Update
HDF4 Mapping Project UpdateHDF4 Mapping Project Update
HDF4 Mapping Project Update
 
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewHDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
 
Connecting HDF with ISO Metadata Standards
Connecting HDF with ISO Metadata StandardsConnecting HDF with ISO Metadata Standards
Connecting HDF with ISO Metadata Standards
 
Earth Science Data and Information System (ESDIS) Project Update
Earth Science Data and Information System (ESDIS) Project UpdateEarth Science Data and Information System (ESDIS) Project Update
Earth Science Data and Information System (ESDIS) Project Update
 
Bridging ICESat and ICESat-2 Standard Data Products
Bridging ICESat and ICESat-2 Standard Data ProductsBridging ICESat and ICESat-2 Standard Data Products
Bridging ICESat and ICESat-2 Standard Data Products
 
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs ProjectsGES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
 

Ähnlich wie HDF and netCDF Data Support in ArcGIS

Ähnlich wie HDF and netCDF Data Support in ArcGIS (20)

HDF Update
HDF UpdateHDF Update
HDF Update
 
Multidimensional Scientific Data in ArcGIS
Multidimensional Scientific Data in ArcGISMultidimensional Scientific Data in ArcGIS
Multidimensional Scientific Data in ArcGIS
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
 
HDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's GuideHDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's Guide
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
Tim Pugh-SPEDDEXES 2014
Tim Pugh-SPEDDEXES 2014Tim Pugh-SPEDDEXES 2014
Tim Pugh-SPEDDEXES 2014
 
Ozri 2013 Brisbane, Australia - Geodatabase Efficiencies
Ozri 2013 Brisbane, Australia - Geodatabase EfficienciesOzri 2013 Brisbane, Australia - Geodatabase Efficiencies
Ozri 2013 Brisbane, Australia - Geodatabase Efficiencies
 
Introduction to GIS
Introduction to GISIntroduction to GIS
Introduction to GIS
 
Reading HDF family of formats via NetCDF-Java / CDM
Reading HDF family of formats via NetCDF-Java / CDMReading HDF family of formats via NetCDF-Java / CDM
Reading HDF family of formats via NetCDF-Java / CDM
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
Big Data and NoSQL for Database and BI Pros
Big Data and NoSQL for Database and BI ProsBig Data and NoSQL for Database and BI Pros
Big Data and NoSQL for Database and BI Pros
 
Apache Spark Fundamentals
Apache Spark FundamentalsApache Spark Fundamentals
Apache Spark Fundamentals
 
Map reducecloudtech
Map reducecloudtechMap reducecloudtech
Map reducecloudtech
 
Introduction to Google Earth Engine .pptx
Introduction to Google Earth Engine .pptxIntroduction to Google Earth Engine .pptx
Introduction to Google Earth Engine .pptx
 
Scala and spark
Scala and sparkScala and spark
Scala and spark
 
NUIG LOSD tools
NUIG LOSD toolsNUIG LOSD tools
NUIG LOSD tools
 
HDF5 and The HDF Group
HDF5 and The HDF GroupHDF5 and The HDF Group
HDF5 and The HDF Group
 
Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding Hadoop
 
view_hdf
view_hdfview_hdf
view_hdf
 

Mehr von The HDF-EOS Tools and Information Center

Mehr von The HDF-EOS Tools and Information Center (20)

Cloud-Optimized HDF5 Files
Cloud-Optimized HDF5 FilesCloud-Optimized HDF5 Files
Cloud-Optimized HDF5 Files
 
Accessing HDF5 data in the cloud with HSDS
Accessing HDF5 data in the cloud with HSDSAccessing HDF5 data in the cloud with HSDS
Accessing HDF5 data in the cloud with HSDS
 
The State of HDF
The State of HDFThe State of HDF
The State of HDF
 
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance FeaturesHighly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
 
Creating Cloud-Optimized HDF5 Files
Creating Cloud-Optimized HDF5 FilesCreating Cloud-Optimized HDF5 Files
Creating Cloud-Optimized HDF5 Files
 
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance DiscussionHDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
 
Hyrax: Serving Data from S3
Hyrax: Serving Data from S3Hyrax: Serving Data from S3
Hyrax: Serving Data from S3
 
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLABAccessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
 
HDF - Current status and Future Directions
HDF - Current status and Future DirectionsHDF - Current status and Future Directions
HDF - Current status and Future Directions
 
HDFEOS.org User Analsys, Updates, and Future
HDFEOS.org User Analsys, Updates, and FutureHDFEOS.org User Analsys, Updates, and Future
HDFEOS.org User Analsys, Updates, and Future
 
HDF - Current status and Future Directions
HDF - Current status and Future Directions HDF - Current status and Future Directions
HDF - Current status and Future Directions
 
H5Coro: The Cloud-Optimized Read-Only Library
H5Coro: The Cloud-Optimized Read-Only LibraryH5Coro: The Cloud-Optimized Read-Only Library
H5Coro: The Cloud-Optimized Read-Only Library
 
MATLAB Modernization on HDF5 1.10
MATLAB Modernization on HDF5 1.10MATLAB Modernization on HDF5 1.10
MATLAB Modernization on HDF5 1.10
 
HDF for the Cloud - Serverless HDF
HDF for the Cloud - Serverless HDFHDF for the Cloud - Serverless HDF
HDF for the Cloud - Serverless HDF
 
HDF5 <-> Zarr
HDF5 <-> ZarrHDF5 <-> Zarr
HDF5 <-> Zarr
 
HDF for the Cloud - New HDF Server Features
HDF for the Cloud - New HDF Server FeaturesHDF for the Cloud - New HDF Server Features
HDF for the Cloud - New HDF Server Features
 
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
 
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
 
HDF5 and Ecosystem: What Is New?
HDF5 and Ecosystem: What Is New?HDF5 and Ecosystem: What Is New?
HDF5 and Ecosystem: What Is New?
 
HDF5 Roadmap 2019-2020
HDF5 Roadmap 2019-2020HDF5 Roadmap 2019-2020
HDF5 Roadmap 2019-2020
 

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 

HDF and netCDF Data Support in ArcGIS

  • 1. HDF and netCDF Data Support in ArcGIS Nawajish Noman Jeff Donze HDF and HDF-EOS Workshop XV, April 17-19, 2012, Riverdale, MD
  • 2. Outline • • • • • HDF and netCDF in ArcGIS Time in ArcGIS Performing Analysis Python Tools Future Directions
  • 3. HDF and netCDF in ArcGIS
  • 4. Scientific Data and Esri • Direct support - NetCDF and HDF • THREDDS/OPeNDAP – a framework for scientific data networking, integrated use by our customers • Examples using Esri technology • National Climate Data Center • National Weather Service • National Center for Atmospheric Research • U. S. Navy (NAVO) • Air Force Weather • Australian Navy • Australian Bur.of Met. • UK Met Office
  • 5. HDF4 and HDF5 Support in ArcGIS
  • 6. Raster Concepts • Raster Format (e.g. img, tif, etc.) – driver level support / storage format and layout / read/write of pixels and metadata • Raster Type (e.g. GeoEye-1) – implies Raster Format support and are format and sensor specific – intelligent use of metadata and other sensor specific parameters – defines georeferencing and well known processing chains • Raster Product (e.g. Panchromatic, Multispectral, Pansharpened ) – templates which make it easy to work with well defined end user products – multiple per sensor – e.g. Panchromatic, Multispectral, Pansharpened • Raster Product Definition (e.g. Natural Color, False Color) – defines “how you want your Mosaic Dataset to look” regardless of multiple source sensors and band combinations – uses metadata such as wavelength information to map bands
  • 7. HDF Raster Support Raster Concept ArcGIS 10.1 Support Raster Format  HDF4 • read: open a HDF subdataset as a Raster Dataset • write: APIs available but not exposed in UI  HDF5 • read: open a HDF subdataset as a Raster Dataset • write: not supported at this time Raster Type  HDF4, HDF5 • direct ingest of one or many HDF subdatasets into a Mosaic Dataset using the Raster Dataset Raster Type or the Table Raster Type * Esri interested in discussing other Raster Types Raster Product * Esri interested in discussing other Raster Products Raster Product Definition * Esri interested in discussing other Raster Product Definitions
  • 8. HDF Raster Support • 10.1 Raster Format and Raster Type support implies… – ArcGIS Desktop • direct use as a Raster Dataset or Mosaic Dataset • use via conversion (i.e. convert to another format) • feature rich use in the applications – Visualization and Mapping – Geoprocessing – ArcGIS Server • publishing as dynamic image services • caching and publishing as tile services (i.e. basemaps) • OGC (WCS, WMS, WMTS)
  • 9. Displaying MODIS LST Data HDFView ArcGIS
  • 10. NetCDF Support in ArcGIS • ArcGIS reads/writes netCDF since version 9.2 • An array based data structure for storing multidimensional data. T • N-dimensional coordinates systems • X, Y, Z, time, and other dimensions • Variables – support for multiple variables • Temperature, humidity, pressure, salinity, etc • Geometry – implicit or explicit • Regular grid (implicit) • Irregular grid • Points Y Z X
  • 12. Ingesting netCDF data in ArcGIS • NetCDF data is accessed as • Raster • Feature • Table • Direct read • Exports GIS data to netCDF
  • 13. CF Convention Climate and Forecast (CF) Convention http://cf-pcmdi.llnl.gov/ Initially developed for • Climate and forecast data • Atmosphere, surface and ocean model-generated data • Also for observational datasets • The CF conventions generalize and extend the COARDS (Cooperative Ocean/Atmosphere Research Data Service) convention. • CF is now the most widely used conventions for geospatial netCDF data. It has the best coordinate system handling.
  • 14. NetCDF and Coordinate Systems • Geographic Coordinate Systems (GCS) • X dimension units: degrees_east • Y dimension units: degrees_north • Projected Coordinate Systems (PCS) • X dimension standard_name: projection_x_coordinate • Y dimension standard_name: projection_y_coordinate • Variable has a grid_mapping attribute. • CF 1.6 conventions currently supports thirteen predefined coordinate systems (Appendix F: Grid Mappings) • Undefined • If not GCS or PCS • ArcGIS writes (and recognizes) PE String as a variable attribute.
  • 15. NetCDF Tools Toolbox: Multidimension Tools • Make NetCDF Raster Layer • Make NetCDF Feature Layer • Make NetCDF Table View • Raster to NetCDF • Feature to NetCDF • Table to NetCDF • Select by Dimension
  • 17. Changing Time Slice 143 342 341 441 131 231 331 431 121 Time = 1 241 221 321 211 311 232 332 433 223 323 423 213 313 413 432 122 222 322 422 112 212 312 412 421 111 333 442 132 141 233 113 242 443 123 142 343 133 Y 243 411 X Time
  • 18. Using NetCDF Data Behaves the same as any layer or table • Display • Same display tools for raster and feature layers will work on netCDF raster and netCDF feature layers. • Graphing • Driven by the table just like any other chart. • Animation • Multidimensional data can be animated through a dimension (e.g. time, pressure, elevation) • Analysis Tools • A netCDF layer or table will work just like any other raster layer, feature layer, or table. (e.g. create buffers around netCDF points, reproject rasters, query tables, etc.)
  • 20. Time is now built-in to ArcGIS • Simple Temporal Mapping • Unified experience for Time • Configure time properties on the layer • Use Time Slider to visualize temporal data • Share temporal visualization • • • • Time-enabled Map Services Export videos or images Generate temporal map books using ArcPy scripting Layer and map packages
  • 23. Spatial and Temporal Analysis • Several hundreds analytical tools available for raster, features, and table • Temporal Modeling • Looping and iteration in ModelBuilder and Python
  • 24. Generate Rainfall Statistics • Calculates specified statistics for all time steps • Outputs a raster catalog • Optionally outputs a netCDF file
  • 25. Generate Rainfall Statistics Table • Calculates statistics for all time steps • Outputs a table • Optionally creates a graph
  • 27. Community Developed Tools • Geoprocessing Resource Center http://resources.arcgis.com/geoprocessing/ • Marine Geospatial Ecology Tools (MGET) • Developed at Duke Univ. • Over 180 tools for import management, and analysis of marine data • Australian Navy tools (not publicly available)
  • 28. Sample Script Tools • Python is used to build custom tools for specific tasks or datasets
  • 29. NEXRAD Geoprocessing Tools • • • • • • Currently 6 geoprocessing script tools Designed to work with NEXRAD netCDF file Can be easily modified for other datasets Customized tools for various workflows Simplify repetitive work Automate GIS processes
  • 30. Extract NEXRAD Rainfall To Points • Extracts the cell values for all time steps • Outputs a feature class
  • 31. New NetCDF Tools (under development) • Download NetCDF File (OPeNDAP, WCS) • Clip • Extract By Variable • Extract By Dimension • Append By Dimension • Variable Statistics • Temporal Statistics
  • 32. Download NetCDF File (WCS/OPeNDAP)
  • 33. Dependencies on 3rd Party Utilities • netcdf4-python • This module can read and write files in both the new netCDF 4 and the old netCDF 3 format, and can create files that are readable by HDF5 clients. • Pydap • Pydap is a pure Python library implementing the Data Access Protocol, also known as DODS or OPeNDAP. OWSLib • OWSLib (OGC Web Service utility library) • Package for working with OGC map, feature, and coverage services. OWSLib provides a common API for accessing service metadata and wrappers for GetCapabilities, GetMap, and GetFeature requests.
  • 36. Scientific Data Workshop and Future Initiatives… • Esri hosted a workshop in February 2012 • To understand the future need for scientific data support in ArcGIS • Ongoing efforts - require close collaboration with all of you • Some of the future initiatives are: • Continue to support netCDF classic and netCDF4 • • • • Provide better support for HDF5 Provide tool to consume data served using THREDDS/OPeNDAP Continue to support the evolving CF convention Support a strong developer experience for netCDF and HDF using Python • What else?
  • 37. Things to Consider… • Embrace the Common Data Model (netCDF, HDF etc.) • Use Data and metadata standards (OGC, CF etc) • Provide “mechanism” so that we can access scientific data using a single set of APIs…. • and can expect data to be CF complainant • Make your data “spatial” (by specifying geographic or a projected coordinate system) • Clearly define workflow and requirements • Create sample tools where possible
  • 38. Questions? Nawajish Noman Team Lead nnoman@esri.com Jeff Donze Esri Federal Business Development jdonze@esri.com

Hinweis der Redaktion

  1. - Land Surface Temperature (LST)-The global monthly composite and average daytime and nighttime land-surface temperatures (LST) at grids with equal latitude and longitude bin sizes of 0.25 degree.- The LST data were generated from the daily MODIS LST product (MOD11B1) at5km spatial resolution.- MODIS (or Moderate Resolution Imaging Spectroradiometer) is a key instrument aboard the Terra (EOS AM) and Aqua (EOS PM) satellites.- http://modis.gsfc.nasa.gov/about/Data Source:Zhengming Wan (wan@icess.ucsb.edu) Institute for Computational Earth System Science University of California, Santa Barbara, CA 93106