Climate data can provide a great deal of information about the atmospheric environment that impacts almost all aspects of human endeavour. This module explains the importance of climate data, its storage, security, applications and other aspects, in a nutshell.

1. 1
CLIMATE DATA MANAGEMENT
By
Prof. A. Balasubramanian
CENTRE FOR ADVANCED STUDIES IN EARTH SCIENCE
UNIVERSITY OF MYSORE
MYSORE-6

2. 2
Introduction:
Climate data can provide a great deal of
information about the atmospheric environment
that impacts almost all aspects of human
endeavour.
Most of the long-term climatological analyses
depend on a number of non-climatic factors.

3. 3
These factors include changes in:
instruments,
observing practices,
station locations,
formulae used to calculate means, and
station environment.
The whole world depends on climatic data for
present and future developments.

4. 4
If we measure rainfall, in order for the data to
be useful for future users, we also need to
document where and how the measurements
were made.
Meteorological data users other than the
climatological community, working in fields
like agrometeorology, engineering or
aeronautics, also benefit from good data.
Meteorological data are influenced by a wide
variety of observational practices.

5. 5
Data depend on the instrument, its exposure,
recording procedures and many other factors.
There is a need to keep a record of all these
metadata to make the best possible use of the
data.
The twentieth century saw the routine
exchange of weather data in digital form and
many meteorological and related data centres
took the opportunity to directly capture and
store these in their databases.

6. 6
Automatic methods of collecting and processing
meteorological data was started in the late
1950s. Today, the management of climate
records requires a systematic approach that
encompasses paper records,
microfilm/microfiche records and digital
records, where the latter include image files as
well as the traditional alphanumeric
representation.

7. 7
The Internet is already delivering greatly
improved data access capabilities and,
providing security issues are managed, we can
expect major opportunities for data managers in
the next five to ten years.
STATION IDENTIFIERS AND GEOGRAPHICAL
DATA
Data can always be associated with some place.
To do so, the station has to be identified by
names and codes and to be located into the
geographical network.

8. 8
It is also important to clearly identify when data
started to be collected and by whom.
Station Identifiers -Name: station names
usually refer to the city or village where the
data are collected .
Type of station:
Synoptical, aeronautical, agrometeorological ,
etc.

9. 9
Geographical Data : Climate data are associated
with geographical locations. Latitude and
longitude and Elevation above Mean Sea Level.
Local Environment, Coordinates and elevation.
Local land use/land cover:
At different scales, it is recommended to keep
track of several attributes.
At the mesoscale (1 km to 30 km) it is
important to account in the metadata for:
o Proximity and size of large water surfaces

10. 10
o Urbanized areas
o Mountain ranges.
At toposcale ("local" scale, 100 m to 2 km)
observations are influenced by:
o Terrain slope, both steepness and direction
o Forests, crops and other roughness .
o Nearby obstacles such as trees or houses (at
airports: airplanes)
o Proximity to irrigation.

11. 11
Landuse/ landcover classification that covers
most cases.
o Artificial surfaces: continuous urban cover;
discontinuous urban cover; industrial and
commercial areas; transportation
infrastructures; harbour areas; airports; mines,
dumps and areas under construction; artificial
green areas (non agricultural).

12. 12
o Agricultural surfaces: non irrigated crops;
irrigated crops; rice fields and other inundated
crops; grasslands; mixed crops; agricultural-
forest systems
o Natural vegetation and open areas: deciduous
forests; evergreen forests; mixed forest; shrub
vegetation; mixed shrub and forest; natural
grasslands and prairies

13. 13
o Wetlands:
swamp areas; peat lands; marshes; inter tidal
flat areas .
o Water bodies:
rivers and other natural water courses;
artificial water courses;
lakes and lagoons;
dams; estuaries; seas and oceans

14. 14
Type of instruments, Instrument exposure
Depending on each meteorological element,
some additional instrument features are very
important:
o Temperature and humidity: screen (type and
size) and ventilation.
Wind direction: time and method of azimuth
alignment.

15. 15
Wind speed: response time of anemometer and
recording chain, and how these where
determined.
o Precipitation: gauge rim diameter, rim height
above ground, presence of overflow storage,
presence of a nipher screen or other airflow-
modifying feature, presence of heating or other
means to deal with solid precipitation.
Global radiation: wavelength range transmitted
by the dome.

16. 16
Sunshine: thresholds for automatic sunshine
recorders.
Evaporation: any coverage applied to
evaporation pan.
Data recording and transmission:
When a meteorological element is measured
with an instrument, data have to be recorded
and usually transmitted to the data management
section of the organization for checking and
archival.

17. 17
Data Conversions:
consider data conversion algorithms.
Data Processing:
It is very important to keep information on how
the data are to be processed, validated and
transmitted to the regional or central office from
every single station.
Units: All units of measurement , analysis,
processing.

18. 18
Calculations:
Calculations other than those made on-site by
the observers, such as time averaging (daily,
monthly and so on) of elements, can also be
performed at stations or at regional and central
meteorological offices.

19. 19
Climate data management
1. User requirements and supporting priority
needs
It is essential to take into account the needs of
the existing, and to the extent that it is
predictable, future data users.
The Climatic data management requires
awareness of the needs of the end users.

20. 20
At present, the key demand factors for data
managers are coming from climate prediction,
climate change, agriculture and other primary
industries, health, disaster/ emergency
management, energy, natural resource
management (including water), sustainability,
urban planning and design, finance and
insurance. The quality of climate data is
greatly influenced by how well observation
networks and systems are managed.

21. 21
The data manager will need to operate as an
effective intermediary between the observation
manager and data user. In responding to user
needs, some of the key issues to consider in
prioritizing new/additional observations are:
National social, economic and environmental
priorities;
Data-poor regions; Poorly-observed parameters;
Regions sensitive to change; and Measurements
with inadequate temporal resolution.

22. 22
Climate Data Management Systems:
Desirable properties- Database model
Any climate database will be based on some
underlying model of the data.
Meteorological data will be accessed
‘synoptically’, and retrievals will be of the
form “get all data for some given locations or
area and for some defined and relatively short
period”.

23. 23
By contrast, climate applications typically
involve retrieving data for one or a few stations,
but for a long period.
Consequently, one broad approach to storing
climate data involves storing (for daily data) all
data associated with a given station and day
together.
A similar approach can be taken for hourly and
monthly data.

24. 24
Key entry capabilities :
The data entry system should be free of
annoying defects which may slow up the data
entry operator. Ideally the forms presented on
the screen will be customizable so as to
optimize the efficiency of data entry. The
system should also, as far as possible, validate
data as they are entered - catching errors and,
where possible, suggesting alternative values.

25. 25
Electronic input options:
As mentioned above, it is desirable for a CDMS
to have the ability to represent the full content
of the relevant WMO standard message formats
– SYNOP, CLIMAT, etc).
An associated beneficial feature is the ability to
decode these message formats directly into the
climate database.

26. 26
Scope of quality checks on observation
values:
Checks should be applied to determine the
quality of an observation.
Data extraction:
Ideally data can be retrieved both from a GUI
interface and from a command-line interface, as
appropriate.

27. 27
Ideally, GUI retrieval facilities should be
provided for the vast majority of users, with
query language facilities used only by a small
range of knowledgeable users who have a need
to do non-standard retrievals.
Output Options:
The system should also support a wide range of
output options.

28. 28
Options should be provided to give access to
listings of data, tabular summaries, statistical
analyses (simple and complex), and graphical
presentations.
Security issues:
The main goals of a security policy and
associated activities are to prevent loss of, or
damage to, the CDMS and to keep data
management facilities in the best possible
condition.

29. 29
The archives and database environment must be
secured and protected against fire, humidity,
etc;
Only protected applications permitted to a small
group of people have the right to handle data
manipulations (i.e. insert, update, delete).
All changes to data tables should have an audit
trail and the controls on access to this trail
should be in place;

30. 30
• There should be a policy of not sharing
passwords as well as not writing down
passwords anywhere.
Passwords should be changed regularly and this
applies to all users from the database
administrator to the user who handles data
manipulation applications.
The archive database system must run behind a
firewall.

31. 31
• Backups must be made at such intervals-
daily and weekly full backups;
A recovery plan should be drawn up
Database management and monitoring :
The aim of database management is to ensure
the integrity of the database at all times, and to
ensure that the database contains all the data
and metadata needed to deliver the objectives of
the organization, both now and into the future.

32. 32
Typical monitoring reports would include the
number and type of stations in the database, the
quantity of data in the database grouped by
stations and by observation element types.
Documentation management :
Documentation of the processes involved in
managing and using the database is essential,
both to record the design and as operational
instructions and guidelines for the managers,
users and developers of the database.

33. 33
Metadata documentation and management :
In order that meteorological data be useful for
future users, it is essential that an adequate set
of metadata be available.
Data acquisition, entry, storage and
archiving
Data collection should be as close to the source
as possible.

34. 34
AWSs, including stations which have some
manual observations, and partially automatic
weather stations should collect their climate
data and error messages on site and transfer
these electronically to the CDMS, possibly via
another database system.
Manually observed data should be collected and
captured on-site and transferred as soon as
possible to the CDMS.

35. 35
Storage and archiving of hard copy records:
All paper records should be stored in a
controlled environment to avoid deterioration
and possible destruction by temperature and
humidity extremes, insects, pests, fire, flood,
accidents or vandalism.
But before archiving, the records should be
captured in microfilm or, preferably, in
electronic image form through a digital camera
or scanner.

36. 36
Storage and archiving of digital information:
An important job of the data manager is to
estimate data storage requirements, including
estimating future growth.
In the case of having not enough mass storage
to keep all original raw data, the oldest data
could be rolled out of the database to a slow
mass storage archiving system.

37. 37
This would generally be a tape robot system but
nowadays it is often a DVD robot system or
something similar.
Managing original records and data rescue
Data rescue is to be comprehensively covered .
Data exchange :
Exchange of data between organisations is
essential for climatology.

38. 38
This may cover both the storage and use of data
(and metadata) among countries in the database
and the transmission of data to global and
regional data centres.
Change management issues :
The kinds of changes that need to be managed
include: Changes to observation networks and
systems; Changes to observational practices;

39. 39
The introduction of new data types; and
Changes in the algorithms that compute derived
data.
Scalability:
Typical issues will be the need to add extra data
types, or to cope with large volumes of extra
data.

40. 40
System Architecture and technology
Data models in use by CDMSs
The Element Model
An Element Model represents data in tables,
having, in each row, different values of one
element observed at one station at different
times.
Advantages: It is easy to add new elements; the
data model remains the same even if a new
element is added.

41. 41
Disadvantages:
Performance for real-time applications may be
poor; many operations on the database can be
more complex than would otherwise be the
case.
The Observation Model
An Observation Model represents data in tables
having, in each row, the values of different
elements observed at one station at a given
time.

42. 42
For example daily data could be stored in a
Daily table.
Each row would correspond to a specific station
at a specific time.
Each column of a specific row would store the
values of the different elements observed.
Advantages:
High performance for real-time applications;
optimisation of data storage.

43. 43
Disadvantages:
Need to update the table structure if a new
element that has not been included during the
database design stage has to be added.
The Value Model
A Value Model will represent the data values in
tables having, in each row, only one value of
one element observed at one station at a
specific time. For example, daily data could be
stored in a Daily table.

44. 44
Each row would correspond to a specific station
for a specific element and at a specific time.
Advantages:
It is easy to add new elements, the model is
adaptable to a large range of data types.
Disadvantages:
Optimization of data storage will not be done
well, so this approach is not suitable for tables
with huge amounts of data; also shares the
disadvantages of the Element model.

45. 45
Computer hardware and software
considerations
Complete inventory and description of the
hardware and software components currently
available: computers, network, operating
systems, DBMSs, applications in use, etc.
Current telecommunication possibilities in
the country and/or the region: International
and national telecommunication lines available:
Internet, GTS, telephone, radio, etc.

46. 46
Functional interactions: Internal, national,
and international? Drawing up functional
schemas between the different stakeholders
within a data management operation including
internal, national and international levels are
especially useful.
Operating System: Which OS?
DBMS:
Which DBMS?
Day to day operation

47. 47
Process and responsibility:
Each process should be described and should be
under the supervision of an identified person.