The document presents guidelines for standardizing the HDF-EOS file format for the Aura satellite mission. It proposes organizing data fields and attributes in a consistent way, using standard dimension and field names, and reporting data on a common pressure grid. This will make files easier to understand, cross-platform compatible, and simplify software development. The guidelines cover topics such as data ordering, units, and attribute names. Standardizing these aspects aims to reduce mismatches between instruments. [/SUMMARY]

1. HDF-EOS Aura File Format Guidelines
HDF-EOS Aura File Format
Guidelines
Cheryl Craig
September 20, 2000
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
1

2. HDF-EOS Aura File Format Guidelines
Preliminary
• Aura Data System Working Group will
meet in October to discuss and hopefully
adopt
• Authors from each instrument team are
in agreement
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
2

3. HDF-EOS Aura File Format Guidelines
What Is Aura?
• Third in series of EOS atmospheric chemistry
satellites (1:45 PM equator crossing time)
• Formerly known as CHEM
• To be launched June 2003
• Level 2 data products include:
– Vertical profiles of temperature, pressure, chemical
and aerosol species
– Column amounts of chemical species
– Cloud properties
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
3

4. HDF-EOS Aura File Format Guidelines
Aura Instruments
• HIRDLS (High Resolution Dynamics Limb Sounder)
– Limb infrared sounder
– University of CO and Oxford
• MLS (Microwave Limb Sounder)
– Limb microwave sounder
– JPL
• OMI (Ozone Monitoring Instrument)
– Nadir wide-field-imaging spectrometer
– Netherlands, Finland and US
• TES (Tropospheric Emission Spectrometer)
– Nadir and limb infrared-imaging spectrometer
– JPL
All instruments have world-wide co-investigators
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
4

5. HDF-EOS Aura File Format Guidelines
Why Guidelines for Aura Instruments?
• HDF-EOS Swath allows a lot of flexibility
• Ease cross-platform use of Aura files
– Ease development of software
– Easier to understand
• Intended to standardize Level 2 files
(may apply to Level 3 and above as well)
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
5

6. HDF-EOS Aura File Format Guidelines
Areas of Potential Mismatch
•
•
•
•
•
•
Organization of data fields and attributes
Dimension names
Geolocation names and dimension ordering
Data field names and dimension ordering
Units for data fields
Attribute names, values and units
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
6

7. HDF-EOS Aura File Format Guidelines
Guidelines List
• Instrument teams will use the same major
version of HDF-EOS
• Aura Level 2 files will use HDF-EOS Swath
• HDF-EOS structure names will strictly adhere
to Valids list
• Data will be reported on a pressure grid and be
ordered from ground to space
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
7

8. HDF-EOS Aura File Format Guidelines
Guidelines List (continued)
• Data fields will be ordered with pressure
coordinate being the fastest incrementing
coordinate
• Data fields will be in units specified
• HDF fill value and MissingValue will have the
same value
• Instrument data fields unique to particular
instrument will be prefixed with instrument
name
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
8

9. HDF-EOS Aura File Format Guidelines
Organization of Data Fields and Attributes
Type 1
Type 2
Swathname: Instrument Name
Swathname: Meas. Species Name
Dimensions:
Dimensions:
nTimes
nLevels etc.
Geolocation Fields:
nTimes
nLevels etc.
Geolocation Fields:
Time
Latitude etc.
Data Fields:
Time
Latitude etc.
Data Fields:
Temperature
O3 etc.
Global Attributes:
L2gpValue
L2gpPrecision
Global Attributes:
Instrument Name
Process Level etc.
Instrument Name
Process Level etc.
HDF-EOS V3.x Note – Two types may be merged into one using links for
Data Field Names
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
9

10. HDF-EOS Aura File Format Guidelines
Standard Names
•
•
•
•
•
Adhere to strict names (spacing and capitalization
required)
Dimension names
Geolocation Fields – Used to hold descriptor data
(includes SolarZenithAngle, OrbitAscendingFlag, etc.)
Data Fields – Contains measured and derived data
Global Attributes – Includes InstrumentName and
ProcessLevel
Data Field Attributes – only if use HDF-EOSV3 (includes
MissingValue, ScaleFactor, Units, etc.)
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
10

11. HDF-EOS Aura File Format Guidelines
HDF-EOS V2.x Versus V3.x
Aura will decide at future date. At January, 2000 DSWG, indicated
desire to used V3.x if packages like IDL will be available in V3.x
HDF-EOS V2.x
HDF-EOS V3.x
PROS
• Currently used by Terra
and Aqua teams
• Currently available
CONS
• Based on HD4 –
supported by HDF group
in maintenance only mode
PROS
• Contains new functionality
– local data attributes
– more than one unlimited
dimension
– compression with unlimited
dimensions
• Based on HDF5
CONS
• IDL support (unknown?)
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
11

12. HDF-EOS Aura File Format Guidelines
Conclusions
• Preliminary standardization complete – meeting in
October to discuss and adopt
• HDF-EOS version decision to be decided at a future
date
• Some decisions postponed until prototyping efforts
complete
• Standard may evolve as instrument teams get better
understanding of their data storage needs
• URL of document:
http://www.eos.ucar.edu/hirdls/HDFEOS_Aura_File_Format_Guidelines.pdf
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
12

13. HDF-EOS Aura File Format Guidelines
Valids from Document
Swath Name:
Instrument Name
- or Measured Field Name (see Data Field Names for list of names)
Dimensions:
These are the actual dimensions of the Geolocation and Data field quantities.
nTimes
nLevels
nWavel
nXtrack
Number of times (profiles) in data set (this may be unlimited)
Number of pressure levels
Number of wavelengths
Number of pixels in the across track direction (OMI specific)
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
13

14. HDF-EOS Aura File Format Guidelines
Geolocation Fields:
This is data which describe the scientific measured quantities. They provide information to aid in describing the data’s “location”.
Instrument
H M O
X
X X
X
X
X
X
X
X
X
X
HDF-EOS Name
Time
Latitude
Latitude
Longitude
Longitude
Pressure+
Dimension
(nTimes)
(nTimes)
(nXtrack,nTimes)
(nTimes)
(nXtrack,nTimes)
(nLevels)
Altitude
SecondsInDay
(nLevels,nTimes)
(nTimes)
Suggested units
seconds from Jan 1, 1993
degrees
degrees
degrees (-180 to 180)
degrees (-180 to 180)
hPa (ordered from ground to
space)
meters
seconds
SolarZenithAngle
LocalSolarTime
SpacecraftLatitude
SpacecraftLongitude
SpacecraftAltitude
OrbitAscendingFlag
OrbitGeodeticAngle
LineOfSightAngle
SolarAzimuthAngle
ViewingZenithAngle
ViewingAzimuthAngle
SpacecraftAzimuth
SpacecraftZenith
HIRDLSScienceScanMode
(nTimes)
(nTimes)
(nTimes)
(nTimes)
(nTimes)
(nTimes)
(nTimes)
(nTimes)
(nXtrack,nTimes)
(nXtrack,nTimes)
(nXtrack,nTimes)
(nTimes)
(nTimes)
(nTimes)
degrees
hours
degrees
degrees (-180 to 180)
meters
logical
degrees
degrees (East of North)
degrees (East of North)
degrees
degrees (East of North)
degrees (East of North)
degrees
short integer
X
X
X
X
X
X
HIRDLSScanUpFlag
(nTimes)
logical
X
Cheryl Craig
X
X
T
X
X
Notes
time in TAI units
Geodetic Latitude
Geodetic Latitude
X
X
Pressure will be a superset of
the UARS pressure levels
X
Seconds from midnight of
day listed in global attributes
X
X
X
X
X
X
X
X
A
A
A
A
if true then orbit is ascending
X
X
X
X
X
A
A
A
X
HIRDLS Science Scan Mode
identifier
HIRDLS Scan Up identifier
HDF-EOS Workshop IV
Sept 20, 2000
14

15. HDF-EOS Aura File Format Guidelines
Geolocation Fields (continued):
HDF-EOS Name
HIRDLSScanElevationAtNominalAltitude
HIRDLSScanAzimuthAtNominal Altitude
HIRDLSTangentHeightAtNominalAltitude
HIRDLSViewDirectionAtNominalAltitude
TESDate
Dimension
(nTimes)
(nTimes)
(nTimes)
(nTimes)
(nTimes)
Suggested units
degrees
degrees
meters
degrees (East of North)
TESSequenceID
TESBoresightAngle
TESFiltersUsed
TESPixelsUsedFlag
(nTimes)
(nTimes)
(nTimes)
(nTimes)
Instrument
H M O
X
X
X
X
sequence within current survey
degrees
bit mask
bit mask
T
Notes
A
TES survey may cover
multiple days
X
A
A
A
X – Field in standard file
A – Data items which will be carried in an ancillary file that is created only once per set of standard products
+ nLevels for TES is currently set to 14 levels (based on UARS standard) but may be changed to 85 levels.
nLevels for HIRDLS is set to 145 (1000. * 10 ** (-i/24) i=0,144)
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
15

16. HDF-EOS Aura File Format Guidelines
Data Field Names:
This is the actual scientific data.
IMPORTANT NOTE – Array ordering is Fortran ordering, the first dimension is the most rapidly incrementing. For C, reverse the array indices.
HDF-EOS Name
L2gpValue
Dimension
(nLevels,nTimes)
Suggested units
varies
L2gpPrecision
(nLevels,nTimes)
varies
Temperature
O3
O3
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nXtrack,nT
imes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
K
parts per volume
parts per volume
X
X
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
X
X
X
X
X
X
X
X
X
H2O
ClONO2
N2O5
N2O
NO2
CH4
HNO3
CFC11
CFC12
OH
HO2
CO
HCN
NO
HCl
H
M O
X
T
X
S
S
X
X
X
S
S
S
S
S
S
S
S
Cheryl Craig
Notes
MLS Generic Data Field Name (species
identified by SwathName)
MLS Generic Precision Name (species
identified by SwathName)
OMI imager data
X
Sp
Sp
Sp
X
X
X
X
Sp
X
Sp
TES special product, tropospheric control
TES limb only
TES limb only
TES limb only
HDF-EOS Workshop IV
Sept 20, 2000
16

17. HDF-EOS Aura File Format Guidelines
Data Field Names (continued):
HDF-EOS Name
HOCl
ClO
BrO
SO2
H2O2
HDO
C2H6
C2H2
HCOOH
CH3OH
PAN
CH3C(O)CH3
C2H4
H02N02
NH3
CCl4
CCl3F
CCl2F2
CHCl2F
CHClF2
COS
H2S
SF6
XX.XXMicronAerosolExtinction
yyyPrecision
yyyNormChiSq
Dimension
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nTimes)
Suggested units
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
parts per volume
extinction (1/km)
H
M O
S
S
S
T
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
Sp
X
X
X
Cheryl Craig
Notes
TES special product, volcanic plumes
Aerosol at XX.XX
Random Errors for species yyy
Normalized Chi2 for species yyy
HDF-EOS Workshop IV
Sept 20, 2000
17

18. HDF-EOS Aura File Format Guidelines
Data Field Names (continued):
HDF-EOS Name
CirrusIceContent
CloudTopHeight
CloudTopPressure
CloudTopPressure
ColumnAmountO3
ColumnAmountNO2
ColumnAmountSO2
ColumnAmountBrO
ColumnAmountHCHO
ColumnAmountOCLO
CloudFraction
AerosolOpticalThickness
Suggested units
g/m3
meters
hPa
hPa
Dobson units
molec/cm2
molec/cm2
molec/cm2
molec/cm2
molec/cm2
dimensionless
dimensionless
SurfaceTemperature
Dimension
(nLevels,nTimes)
(nTimes)
(nXtrack,nTimes)
(nTimes)
(nXtrack,nTimes)
(nXtrack,nTimes)
(nXtrack,nTimes)
(nXtrack,nTimes)
(nXtrack,nTimes)
(nXtrack,nTimes)
(nXtrack,nTimes)
(nWavel,
nXtrack,nTimes)
(nTimes)
H
M O
S
T
K
X
RandomError
LandSurfaceEmissivity
SystematicError
TotalColumnDensity
TotalColumnDensityError
EstRandomErrors
EstSystematicErrors
EstFractionalExplainedVariance
EstCorrelationLength
Convergence
CompConvergenceToEstimate
(nLevels,nTimes)
(nTimes)
(nLevels,nTimes)
(nTimes)
(nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nLevels,nTimes)
(nTimes)
(nTimes)
range: 0-1
X
A
X
X
X
X
X
X
X
X
A
Notes
X
X
X
S
range: 0-1
moles/cm2
moles/cm2
range: 0-1
range: 0-1
range: 0-1
0-1000 km
Cheryl Craig
(OMI imager data)
X
X
X
X
X
X
X
X
X
(May be cloud top for limb or may be
omitted for limb)
TES may still split into Trop/Strat
TES may still split into Trop/Strat
(Chi2) final retrieval – not normalized
Comparison between actual Chi2 and
estimated Chi2 – not normalized
HDF-EOS Workshop IV
Sept 20, 2000
18

19. HDF-EOS Aura File Format Guidelines
Data Field Names (continued):
HDF-EOS Name
ConvergenceFlag
nIterations
Dimension
(nTimes)
(nTimes)
Suggested units
0-1 (Boolean)
0-8 small int.
nonConvergenceFlag
(nTimes)
0-1 (Boolean)
OutofExpectedRange
AverageRMSError
MaxRMSError
CloudCover
ProcessingTime
(nTimes)
(nTimes)
(nTimes)
(nTimes)
(nTimes)
H
M O
T
A
A
X
X
X
X
A
A
0-1 (Boolean)
Notes
Final retrieval converged
Number of model iterations for final
retrieval
Did not converge at least during an
intermediate step
Identified through retrieval or L1B
X – Field in standard file
S – Swathname (MLS will identify field with the swathname rather than the fieldname)
A – Data items which will be carried in an ancillary file that is created only once per set of standard products
Sp – TES special data product
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
19

20. HDF-EOS Aura File Format Guidelines
F5 Note
Global Attributes:
This is information which helps to describe this particular data set.
Attribute Name
InstrumentName
ProcessLevel
GranuleMonth
GranuleDay
GranuleYear
Version
VerticalCoordinate
Attribute Type
character
HDFE_NATIVE_SHORT
HDFE_NATIVE_INT
HDFE_NATIVE_INT
HDFE_NATIVE_INT
HDFE_NATIVE_CHAR
HDFE_NATIVE_CHAR
Attribute Values
“HIRDLS”, “MLS”, “TES” or “OMI”
2, 3, etc.
1-12
1-31
“Pressure”, “Altitude”, “Potential Temperature”
Data Field Attributes
This is information which helps to describe the individual data fields. Data Field Attributes are a feature which can be useful in annotating plots as
well as describing the data product to input routines. It is a feature which was cumbersome to implement and retrieve in HDF-EOS V2.x, but is a
subroutine call in HDF-EOS V3.x. For this reason, it is proposed that we only use data field attributes if the Aura team adopts HDF-EOS V3.x as
the version to use.
Data Field Attributes:
Attribute Name
MissingValue
ScaleFactor
Offset
Title
Units
Attribute type
Same type as Data Field
HDFE_NATIVE_FLOAT
HDFE_NATIVE_FLOAT
HDFE_NATIVE_CHAR
HDFE_NATIVE_CHAR
Attribute Description
Contains the value for missing data
Factor for scaling data (if applicable)
Value to add to the data (if applicable)
For labeling a plot or axis
Labeling units (for labeling color bars, converting between units, etc).
Values for Units: "m", "km", "hPa", "Pa", "mAtm-cm", "ppbv", “K”, “micron”, “degree”, “second”, “hour”, “m-1”, “dimensionless”, etc.
Cheryl Craig
HDF-EOS Workshop IV
Sept 20, 2000
20