1. The HDF Group
Tools to Improve the Usability
of NASA HDF Data
Kent Yang and Joe Lee
The HDF Group
April 17, 2012
HDF/HDF-EOS Workshop XV
1
www.hdfgroup.org

2. Tutorial Outline
Introduction
5 Tools
1.
H4CF Conversion Toolkit (h4cf)
2.
h4toh5 Conversion Tool (h4toh5)
3.
HDF-EOS5 to netCDF-4 Converter (eos52nc4)
4.
HDF-EOS5 Augmentation Tool (aug_eos5)
5.
HDF-EOS2 Dumper (eos2dump)
Demo
April 17, 2012
HDF/HDF-EOS Workshop XV
2
www.hdfgroup.org

3. Who Said Users Don’t Matter?
April 17, 2012
HDF/HDF-EOS Workshop XV
3
www.hdfgroup.org

4. Usability of NASA HDF Data
MATLAB
Panoply
Panoply
April 17, 2012
IDL
NASA
HDF/HDF-EOS
Data
IDV
IDV
AA school teacher’s view
school teacher’s expectation
HDF/HDF-EOS Workshop XV
4
www.hdfgroup.org

5. The 5 Tools
Improve the usability of NASA Data
Supported by The HDF Group
Funded by NASA / Raytheon
http://hdfeos.org/software/tool.php
April 17, 2012
HDF/HDF-EOS Workshop XV
5
www.hdfgroup.org

6. 5 Tools that Improve Usability
Panoply / IDV
NETCDF-3
h4cf
h4toh5
eos2
dump
eos2
hdf4
ASCII
eos5
NETCDF-4
April 17, 2012
eos52nc4
aug_eos5
HDF/HDF-EOS Workshop XV
MATLAB
IDL
NCL
6
www.hdfgroup.org

7. H4CF Conversion Toolkit
Panoply / IDV
NETCDF-3
h4cf
eos2
hdf4
Follow
NETCDF-4
CF convention
Both Library and Tools
http://hdfeos.org/software/h4cflib
April 17, 2012
HDF/HDF-EOS Workshop XV
7
www.hdfgroup.org

8. H4CF Conversion Toolkit
Two ways to build:
1. Use configure script.
2. Edit Makefile
template.
April 17, 2012
HDF/HDF-EOS Workshop XV
8
www.hdfgroup.org

9. H4CF Conversion Library
• Variables (datasets) list
• Attributes (metadata) list
• All Vars & Attrs follow CF
conventions
• Develop your own CF application!
April 17, 2012
HDF/HDF-EOS Workshop XV
9
www.hdfgroup.org

10. H4CF Library APIs Example
// open the example HDF file
h4cf_open("geo.hdf");
// HDF file attributes are obtained in a C++
STL map object
const map<string, string> file_attrs =
h4cf_get_file_attrs();
// HDF file variables are obtained in a C++
STL list object
const list<var*> pvars = h4cf_get_vars();
// close the HDF file
h4cf_close();
April 17, 2012
HDF/HDF-EOS Workshop XV
10
www.hdfgroup.org

11. H4CF Conversion Tools
2 Tools based on H4CF APIs:
cdldumper
h4tonccf
April 17, 2012
HDF/HDF-EOS Workshop XV
11
www.hdfgroup.org

12. cdldumper
• Usage: cdldumper filename.hdf
• Dump file in Common
Data Language.
• Useful for quick check and debugging purpose.
April 17, 2012
HDF/HDF-EOS Workshop XV
12
www.hdfgroup.org

13. cdldumper Demo
Number of attributes=1
_FillValue=255
April 17, 2012
HDF/HDF-EOS Workshop XV
13
www.hdfgroup.org

14. cdldumper Demo
Number of attributes=4
_FillValue=255
%./cdldumper -H AMSR_E_L3_5DaySnow_V09_20050126.hdf
orig_name
netCDF AMSR_E_L3_5DaySnow_V09_20050126.hdf {
long_name
dimensions:
coordinates
XDim_Northern_Hemisphere = 721, XDim_Southern_…
variables:
ubyte
Flags_NorthernPentad(YDim_Northern_Hemis…
Flags_NorthernPentad:_FillValue = 255;
Flags_NorthernPentad:origname = "/Northern…
Flags_NorthernPentad:long_name = "/Northern…
Flags_NorthernPentad:coordinates = "latitude…
April 17, 2012
HDF/HDF-EOS Workshop XV
14
www.hdfgroup.org

15. h4tonccf
• Usage: h4tonccf filename.hdf [filename.nc]
• generates either netcdf-3 or netcdf-4 file.
• follows netcdf
April 17, 2012
classic model.
HDF/HDF-EOS Workshop XV
15
www.hdfgroup.org

16. h4tonccf Demo
April 17, 2012
HDF/HDF-EOS Workshop XV
16
www.hdfgroup.org

17. Don’t Follow CF Conventions Blindly
[1] CF: scale * value + offset
[2] MODIS: scale * (value – offset)
Scale = 0.01
Value = 5000
Offset = -15000.0
[1] 0.01 * 5000 + (-15000) = -14950
[2] 0.01 * (5000 – (-15000)) = 200
April 17, 2012
HDF/HDF-EOS Workshop XV
17
www.hdfgroup.org

18. -14950 Kelvin?
April 17, 2012
HDF/HDF-EOS Workshop XV
18
www.hdfgroup.org

19. MODIS Scale / Offset Correction
April 18, 2012
HDF/HDF-EOS Workshop XV
19
www.hdfgroup.org

20. Converted File Size Comparison
656M
Netcdf-3
128M
Netcdf-4
72M
HDF-EOS2
April 17, 2012
HDF/HDF-EOS Workshop XV
9X
20
www.hdfgroup.org

21. Limitation – No Support Yet
MISR
(SOM/LAMAZ)
MERRA
AVHRR
LISTO
QuikSCAT/SeaWind
MOPITT L2
etc.
April 17, 2012
HDF/HDF-EOS Workshop XV
21
www.hdfgroup.org

22. h4toh5 Conversion Tool
Convert any hdf4 to hdf5.
http://hdfgroup.org/h4toh5
h4toh5
NETCDF-4
April 17, 2012
eos2
Usage: h4toh5 –eos –nc4 in.hdf out.nc
Static Binary w/ –eos & -nc4:
http://hdfeos.org/software/h4toh5/bin/
HDF/HDF-EOS Workshop XV
22
www.hdfgroup.org

23. h4tonccf vs. h4toh5
h4tonccf
Handle
non-eos2 hdf4
Follow
CF conventions
Output
netcdf classic
Output
netcdf
enhanced
April 17, 2012
Yes
h4toh5
w/ -eos –nc4
No
Yes
No
Yes
No
No
Yes
HDF/HDF-EOS Workshop XV
23
www.hdfgroup.org

24. eos52nc4
•
•
•
•
•
converts hdf-eos5 to netcdf-4.
follows netcdf enhanced model.
does not follow CF conventions.
Usage: eos52nc4 test.he5 test.nc
http://hdfeos.org/software/convert_hdfeos5.php
eos5
NETCDF-4
eos52nc4
April 17, 2012
HDF/HDF-EOS Workshop XV
24
www.hdfgroup.org

25. aug_eos5
• augments hdf-eos5 to netcdf-4.
• Both HDF-EOS5 and NetCDF4 can access it.
• Usage: aug_eos5 [-i|-f map] test.h5
• modifies the input file directly.
• does not follow CF conventions.
• http://hdfeos.org/software/aug_hdfeos5.php
eos5
NETCDF-4
aug_eos5
April 17, 2012
HDF/HDF-EOS Workshop XV
25
www.hdfgroup.org

26. Augmentation Concept
aug_eos5
HDF-EOS5
API
NETCDF-4
API
hdf-eos5
augmentation
netcdf4
hdf5
HDF5 API
April 17, 2012
HDF/HDF-EOS Workshop XV
26
www.hdfgroup.org

27. Why NetCDF-4 Doesn’t Work
[nLevels][nTimes]
hdf-eos5
April 17, 2012
HDF/HDF-EOS Workshop XV
NETCDF-4
API
27
www.hdfgroup.org

28. What aug_eos5 Does
[nLevels][nTimes]
hdf-eos5
April 17, 2012
HDF/HDF-EOS Workshop XV
NETCDF-4
API
28
www.hdfgroup.org

29. Options in aug_eos5
1. Default option
• -i option
2.-f map_file option
April 17, 2012
HDF/HDF-EOS Workshop XV
29
www.hdfgroup.org

30. Default option in aug_eos5
Grid
• XDim, YDim:
calculated
• Other dims: default fill value
Swath / ZA
• All dims: default fill value
April 17, 2012
HDF/HDF-EOS Workshop XV
30
www.hdfgroup.org

31. Better -i option in aug_eos5
• Grid
• XDim, YDim: calculated
• Other dims:
1,2,…,n
• Swath / ZA
• All dims:
April 17, 2012
1,2,…,n
HDF/HDF-EOS Workshop XV
31
www.hdfgroup.org

32. Best -f option in aug_eos5
• Needs an input map file.
• Input file specifies 3
rules:
1. What to be filled in
2. How to fill in
3. Where to get the values
• Most useful and flexible option
April 17, 2012
HDF/HDF-EOS Workshop XV
32
www.hdfgroup.org

33. aug_eos5 Input File Format
Each line in input file contains
1. How: 0,1,2
2. What: dim name
3. Where: data file or existing variable in file.
4. (Optional) Multiple grid/swath/za case only:
•
April 17, 2012
grid/swath/za name:<name>
HDF/HDF-EOS Workshop XV
33
www.hdfgroup.org

34. aug_eos5 Input File Example
Copy the data
Read file data.txt
How
0
1
2
What
Where
nTimes
Time
nLevels
data.txt
nCandidates
Fill 1,2,…, n
95% cases – Copy the data
April 17, 2012
HDF/HDF-EOS Workshop XV
34
www.hdfgroup.org

35. Multiple Swath Case
How
Multiple
What
Where
swath name: BrO
0 nTimes
Time
1 nLevels
data1.txt
2 nCandidates
swath name: BrO column
2 nTimes
1 nLevels
data2.txt
April 17, 2012
HDF/HDF-EOS Workshop XV
35
www.hdfgroup.org

36. aug_eos5 vs eos52nc4
aug_eos5
hdfeos-5
Yes
No
netcdf-4
April 17, 2012
eos52nc4
Yes
Yes
HDF/HDF-EOS Workshop XV
36
www.hdfgroup.org

37. HIRDLS/MLS Team Use aug_eos5
So You Can!
April 17, 2012
HDF/HDF-EOS Workshop XV
37
www.hdfgroup.org

38. HDF-EOS2 Dumper Tool
Useful for tools that can’t handle some projections.
eos2
eos2
dump
ASCII
http://hdfeos.org/zoo
MATLAB
IDL
NCL
April 17, 2012
HDF/HDF-EOS Workshop XV
38
www.hdfgroup.org

39. Important Tool for Zoo Maintenance
April 17, 2012
HDF/HDF-EOS Workshop XV
39
www.hdfgroup.org

40. Why Tools Need HDF-OES2 Dumper
HDF-EOS2
dumper
Lat / Lon
in
ASCII
Lat / Lon
LaRC MISR
SOM Projection
HDF-EOS2
April 17, 2012
MATLAB
IDL
NCL
HDF/HDF-EOS Workshop XV
40
www.hdfgroup.org

41. HDF-EOS2 Dumper Options
Usage: eos2dump options filename.hdf
What:
1=lat / 2 =lon
Format: m=5 values per line
April 17, 2012
HDF/HDF-EOS Workshop XV
41
www.hdfgroup.org

42. eos2dump Latitude Example
90
80
eos2dump –c filename.hdf > latitude.txt 70
60
50
40
What:
30
Format: one value per line (default)
20
10
0
-10
-20
-30
1
lat
April 17, 2012
HDF/HDF-EOS Workshop XV
42
www.hdfgroup.org

43. eos2dump Longitude Example
eos2dump –c2
m filename.hdf > longitude.txt
What: lon
Format: 5
values per line
MATLAB cannot read it.
180 175 170 165 160
155 150 145 140 135
130 125 120 115 110
April 17, 2012
HDF/HDF-EOS Workshop XV
43
www.hdfgroup.org

44. Got Tools?
April 17, 2012
HDF/HDF-EOS Workshop XV
44
www.hdfgroup.org

45. Enjoy the NASA HDF Data!
Panoply / IDV
NETCDF-3
h4cf
h4toh5
eos2
NASA HDF
hdf4
data users
eos2
dump
ASCII
eos5
NETCDF-4
April 17, 2012
eos52nc4
aug_eos5
HDF/HDF-EOS Workshop XV
MATLAB
IDL
NCL
45
www.hdfgroup.org

46. … and observe the Earth!
April 17, 2012
HDF/HDF-EOS Workshop XV
46
www.hdfgroup.org

47. The HDF Group
Thank you !
April 17, 2012
HDF/HDF-EOS Workshop XV
47
www.hdfgroup.org

48. Acknowledgements
This work was supported by Subcontract number
114820 under Raytheon Contract number
NNG10HP02C, funded by the National Aeronautics
and Space Administration (NASA) and by
cooperative agreement number NNX08AO77A from
the NASA. Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the authors and do not necessarily reflect
the views of Raytheon or the National Aeronautics
and Space Administration.
April 17, 2012
HDF/HDF-EOS Workshop XV
48
www.hdfgroup.org

49. The HDF Group
Questions/comments?
April 17, 2012
HDF/HDF-EOS Workshop XV
49
www.hdfgroup.org