HDF and HDF-EOS Workshop XXV (2022)

1. ESIP-0722 JG
Hyrax: Serving Data from S3
Summer ESIP 2022
This work was supported by NASA/GSFC under Raytheon Technologies contract number 80GSFC21CA001.
This document does not contain technology or Technical Data controlled under either the U.S. International Traffic
in Arms Regulations or the U.S. Export Administration Regulations.
James Gallagher
Software Engineer/NASA EED-3 contractor
jgallagher@opendap.org

2. ESIP-0722 JG
2
• Serve existing files that are stored on S3*
• There’s no need to alter the files – just
copy them to S3
• Works for HDF5** and netCDF4***
What you can do
* Simple Storage Service
** Hierarchical Data Format, version 5
*** Network Common Data Format, version 4

3. ESIP-0722 JG
3
• Where the server will run
• Where you will store the ancillary
metadata files the server needs
– These files are not the big data files
– They provide a road map to the interior
structure of those data files
• The URLs* of the data files you want to
serve
What you need to know
* Universal Resource Locators

4. ESIP-0722 JG
4
• Use the command line tool (get_dmrpp)
to build a DMR++* file
• Use get_dmrpp’s default setting and see
if the way it represents your data is
acceptable (try it and see)
• Customize the configuration if needed
• Write a script to process your collection of
files
How to make the ancillary files
*Dataset Metadata Response Plus Plus

5. ESIP-0722 JG
5
• Get the Hyrax server Docker container
• Start the container – it contains both the
Hyrax data server and the get_dmrpp
command
• Run get_dmrpp inside the container – a
recipe follows
To run get_dmrpp

6. ESIP-0722 JG
6
Go to the directory where you want to store the ancillary information
Start the docker container that has the Hyrax server and the get_dmrpp tool
docker run --hostname hyrax
--publish 8080:8080
--volume $(pwd):/usr/share/hyrax
--env AWS_ACCESS_KEY_ID
--env AWS_SECRET_ACCESS_KEY
--name hyrax
opendap/hyrax:snapshot
If your computer uses the new Apple Silicon processor (M1), add
--platform linux/amd64
The get_dmrpp Recipe

7. ESIP-0722 JG
7
Run any command in the Docker container
docker exec –interactive --tty hyrax /bin/bash
Run get_dmrpp in the Docker container
docker exec –interactive --tty hyrax
/bin/bash -c “cd /usr/share/hyrax; get_dmrpp …”
Where
get_dmrpp
-b .
-u https://url.for.your/data/file.nc
-o file.nc.dmrpp
s3://bucket/objectname
get_dmrpp, continued

8. ESIP-0722 JG
8
Run any command in the Docker container
docker exec –interactive --tty hyrax /bin/bash
Run get_dmrpp in the Docker container
get_dmrpp
-b .
-u https://cloudydap.s3.amazonaws.com/samples/1A.GPM.GMI.COUNT2014v3.20160105-
S230545-E003816.010538.V03B.h5
-o 1A.GPM.GMI.COUNT2014v3.20160105-S230545-E003816.010538.V03B.h5.dmrpp
s3://cloudydap/samples/1A.GPM.GMI.COUNT2014v3.20160105-S230545-E003816.010538.V03B.h5
This will build a DMR++ document for the named granule in
cloudydap S3 bucket. The DMR++ will use the URL
https://cloudydap.s3... to read data values.
get_dmrpp, example

9. ESIP-0722 JG
9
• Look at the default output and decide how
it should change
• Look at the HDF5 Handler documentation
for Hyrax – the same options can be used
with get_dmrpp
• Put those optional ‘keys’ in a file and pass
name of that file to get_dmrpp using the
command’s -s option
How to customize the DMR++

10. ESIP-0722 JG
10
• How to build & deploy DMR++ files for Hyrax
• Customization options for HDF5
• If you’d like more information or a
demonstration, see me after the session
More information
Some useful Docker commands
docker rm -f $(docker ps -aq) # remove all containers
docker rmi -f $(docker images -q) # remove all images

11. ESIP-0722 JG
11
This work was supported by NASA/GSFC under
Raytheon Technologies contract number
80GSFC21CA001.