HDF and HDF-EOS Workshop XXV (2022)

1. SESIP-0722-JL
Accessing Cloud Data and
Services Using EDL, Pydap,
MATLAB
HDF-EOS Workshop XXV / ESIP 2022 Summer
This work was supported by NASA/GSFC under Raytheon Technologies contract number 80GSFC21CA001.
This document does not contain technology or Technical Data controlled under either the U.S. International Traffic
in Arms Regulations or the U.S. Export Administration Regulations.
H. Joe Lee
EED-3 / The HDF Group
Software Engineer
hyoklee@hdfgroup.org

2. SESIP-0722-JL
2
A Single Sign-On Solution for User
registration and profile management for all
EOSDIS system components and data
services:
• Data
• Tools
• Services
The Earthdata Login (EDL)

3. SESIP-0722-JL
3
• Data: Cloud Data on Amazon Web
Service (AWS) Simple Storage Service
(S3)
• Service: Open-source Project for a
Network Data Access Protocol
(OPeNDAP)
• Goal: Help users to access S3 data and
OPeNDAP service by providing code
examples on hdfeos.org site.
Our Focus: Data and Service

4. SESIP-0722-JL
4
• Data: I can’t access HDF5* on S3 using
MATLAB.
• Service: I can’t read data from OPeNDAP
server using Pydap client.
Some Problems Users
Encountered
*Hierarchical Data Format version 5

5. SESIP-0722-JL
5
• Use two URLs*:
– URL1 =
'https://urs.earthdata.nasa.gov/oauth/authori
ze/…’ # For EDL session
– URL2 =
‘https://acdisc.gesdisc.eosdis.nasa.gov/open
dap/…’ # For OPeNDAP data
Pydap Solution
:
*Uniform Resource Locator

6. SESIP-0722-JL
6
• Get a temporary AWS token using EDL.
• Copy & paste the token to MATLAB code.
• Run MATLAB EC2* instance on us-west-2
region to access data on S3.
S3 Solution for MATLAB Users
:
*Elastic Compute Cloud

7. SESIP-0722-JL
7
• Pydap:
http://hdfeos.org/forums/showthread.php?
t=821#4
• MATLAB:
https://hdfeos.org/zoo/PODAAC/ATM_SU
RFACE_TEMP_HUM_WIND_PRES_day
_mean_ECCO_V4r4_nc_c.m
Examples Provided on
HDFEOS.org
:

8. SESIP-0722-JL
8
• S3: Token is valid for 1 hour.
– This is the limit imposed by AWS on role
chaining
• MATLAB: netCDF* API** doesn’t work.
Limitation
:
*Network Common Data Form
**Application Programming Interface

9. SESIP-0722-JL
9
• S3: Token is valid for 1 hour
– Run multiple MATLAB instances in parallel.
– Code must handle expired tokens and
request new ones as needed for sessions
that exceed the 1-hour limit.
• MATLAB: netCDF API doesn’t work
– Use MATLAB HDF5 API for netCDF-4*.
– Ask MathWorks to support S3 access.
Possible Solutions
:
*Network Common Data Form version 4

10. SESIP-0722-JL
10
This work was supported by NASA/GSFC under
Raytheon Technologies contract number
80GSFC21CA001.