Producing INSPIRE Compliant Data Sets

INSPIRE Data Models
Finnish Meteorological Insitute
Finnish Meteorological Institute
Roope Tervo

Finnish Meteorological Institute opened its data in 2013.
Basically everything that FMI has property rights was opened.
Data is provided in freely in machine readable format.
29.4.2015 INSPIRE Data Models | Finnish Meteorological Insitute | Roope Tervo 2
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data

FMI Open Data Portal follows INSPIRE requirements.
29.4.2015 3
FMI Open Data Portal
Meta data
Data
Models
Services
The very same data portal works as Open Data and
INSPIRE portal.
INSPIRE Data Models | Finnish Meteorological Insitute | Roope Tervo

Catalog Service
(CSW)
o Based on GeoNetwork
29.4.2015 4
View Service (WMS)
o Based on GeoServer
o Only the most common layers
published

Download Service
(WFS 2.0)
o Web Feature Service (WFS) 2.0
Simple Profile
o Based on stored queries
o Predefined data sets with
possibility for additional
parameters (i.e. time and
area)
o In-house production
29.4.2015 5INSPIRE Data Models | Finnish Meteorological Insitute | Roope Tervo

Data set Description Time
Interval
Estimated
publish date
Weather
Observations
Temperature, Wind,
Humidity, Ground
Temperature…
10 min Open,
older data to be
added
Sun Radiation UV, Short and Long
Term Radiation…
1 min Open
Marine
Observations
Waves, Sea
Temperature, Sea
Level…
1 h Open
Weather Radars Precipitation Rate,
Precipitation Amount…
5 min Open,
older data to be
added
Lightning Thunder Strikes in
Finland
5 min Open
29.4.2015 INSPIRE Data Models | Finnish Meteorological Insitute | Roope Tervo 6
Example of Data Sets

29.4.2015 7
Data set Description Time Interval Estimated
publish date
Real Time
Observations
Real Time Observations from
specific location(s)
AWS 2010 –
Soundings 1959 –
Flashes 1998 –
Sea Level 1971 –
Waves 2005 –
Open
older data will
be added
Climatological
Observations
Dayly and monthly
temperature mean and
extreme values from weather
stations
1959 - Open
Climatological
Observations
Monthly temperature and
precipitation rate mean
values interpolated to grid
1961 - Open
Climatological
Reference
Climatological Reference.
Temperature, humidity,
pressure, precipitation
amount and snow depth.
Reference seasons:
1971-2000 1981-
2010
Open

29.4.2015 8
publish date
Weather forecast
model HIRLAM RCR
Point forecasts and grid
data
Latest model
run
(4 times a day)
0…54 h
Open
Sea forecast models Sea level point
forecasts, Wave (WAM)
and current (HBM) as
grid data
Latest model
run
(4 times a day)
0...54 h
Open
Environmental
Monitoring Facilities
Weather observation
stations, radars…
2015
Aviation
Observations
METAR 30 min open
Ground & mast
observations
Special observations
from ground and masts
2015

29.4.2015 9
publish date
Air Quality
Observations
Air Quality Observations 1h 2015-2016
Silam Model Dispersion Model for Air
Quality, Forest Fire and
Pollen
Latest model
run (once a day)
0…96h
2015
HELMI Ice Model Ice forecast model Latest model
run
(4 times a day)
0...54 h
open
Soundings Temperature, Humidity,
Pressure, Wind from
ground to 25 km height
2 times a day 2015

Data Models
o Observations and point
forecasts as GML
o The same data is published in:
o MultiPointCoverage
o MeasurementTimeSeries
o SimpleFeature
o Gridded data is provided in
appropriate binary format (Grib,
NetCDF, GeoTiff…)
o WFS members contains the
metadata ‘envelope’ with a link
to a actual data

Data Models
gmlcov:MultiPointCoverage
29.4.2015 11
gml:rangeSet
gml:doubleOrNilReasonTupleList
The data is listed for every
point defined in domain set.
gml:domainSet
gmlcov:simpleMultiPoint
The coverage is
defined as a list of
points in 4
dimensional grid (lat,
lon, height, time).
gmlcov:rangeType
The parameters
listed in range set
are defined in
separate element.

Cons
- Not intuitive
- No natural
structure of XML
 XSLT and
Xpath don’t work
Pros
+ Compact
+ Efficient
+ Small file size
+ Works for many
data types
29.4.2015 12
Data Models

Data Models
wml2:MeasurementTimeseries
29.4.2015 13
wml2:MeasurementTimeseries
One member contains time
series for one parameter
and one location

Cons
- Lots of repetition
- Large file size
- Heavy for DOM-
based parsers
- Don’t work i.e. for
thunder strikes
Pros
+ Intuitive
+ Easy to use
+ XSLT & XPath
works
29.4.2015 14
Data Models
wml2:MeasurementTime
series

Data Models
SimpleFeature
29.4.2015 15
SimpleFeature
One member contains one
time, one parameter and
one location

Cons
- Lots of repetition
- Very large file size
- Heavy for DOM-
based parsers
Pros
+ Intuitive
+ Easy to use
+ XSLT & XPath
works
+ Ready client
support
29.4.2015 16
Data Models
SimpleFeature

29.4.2015 17
Data Type Data Format
Observations wml2:MeasurementTimeseries
SimpleFeature
Point Forecasts wml2:MeasurementTimeseries
SimpleFeature
Lighting Observations gmlcov:MultiPointCoverage
SimpleFeature
Grid Forecasts XML Envelope + Grib2/NetCDF
Radar Images GeoTiff / PNG images
METAR IWXXM

29.4.2015 18
Data Models File size Comparison
81.7
52.9
1.81.3 1.2 0.2
0
10
20
30
40
50
60
70
80
90
Document Size
[MB]
Compressed
DocumentSize
[MB]

29.4.2015 19
Data Models Popularity
Comparison
80
19.8
0.2
0
10
20
30
40
50
60
70
80
90
Downloads[%]

Intranet
DMZ
Server 1
GS 1 GS 2 GS 3
Server 2
GS 1 GS 2 GS 3
Server 3
GS 1 GS 2 GS 3
Load Balancer
Configuration
GeoServer
Data
(NFS)
Configuration
(NFS)
Database

Intranet
DMZ
Backend
(WFS)
Load Balancer
Data
(NFS)
Configuration
(NFS)
Database
Frontend Frontend
Backend
(WFS)
Backend
(binary data)
Backend
(binary data)

Open Data Service
Cluster
S1 S2 S3
Client Data Service
Cluster
S1 S2 S3
Load Balancer
Configuration
Data
(NFS)
Configuration
(NFS)
Database

Weather Data – Volumes
29.4.2015 23
• In-situ weather measurement 1 TB
• Weather radar data 50 TB
• NWP model gridded data (FMI)
• HIRLAM 120 TB
• AROME 230 TB
• Satellite image data (FMI)
• Globsnow 32 TB
• Other 50 TB
• Climate model data (FMI) 29 TB
• Other models
• SILAM 100 TB
• Tuuliatlas 21 TB
• Other 300 TB

Producing INSPIRE Data Products
Observations
Database BS Data Server
BS Data Server
WFS Plugin

Point Forecasts
File
System
BS Data Server
BS Data Server
WFS Plugin

Grid Forecasts 1/2
File
System
BS Data Server
BS Data Server
WFS Plugin

Grid Forecasts 2/2
File
System
BS Data Server
BS Data Server
Download
Plugin

Radar Images 1/2
PostGIS
DB
BS Data Server
BS Data Server
WFS Plugin

Radar Images 2/2
PostGIS
DB
GeoServer

INSPIRE Data Sets
How to define a data set?
o All weather observations from
Finland?
 Would cause over 50 000 000
Observations (XML file size ~37 G)
o All observations from one
observation station?
 Would cause over 200 data sets
o Even one year’s observations cause
too large data set to handle

INSPIRE Data Sets
Meteorological data is a constant
flow of observations
FMI has one data set per data
type, i.e. one for ground weather,
observations, one for Hirlam
weather forecasts, etc…
Every data set have predefined
area and time range.

INSPIRE Data Sets
It is notable that data set
response depends on time it’s
requested
 Unique identifiers are not
reasonable

And a little over
300 000 data
downloads
per day
(3,7 req/s)
At the moment
about 7200
registered users
29.4.2015 33
Some Experiences

Practically
no client
supports
complex
features
Although standards
are followed, there’s
a gap between
provided data model
and clients’
capabilities
Some Experiences

GeoServer is
modified to support
stored queries in
WFS 2.0 (released
in version 2.7)
FMI is going to open
the same data as
simple features to
support clients
Some Experiences

Industry is
happy to use
standardized
services
Amateur and
freelancer coders
would prefer simple
JSON API
Some Experiences

So far maybe
even more
professional
interest than
private
Quite many
expected a user
interface to load data
to i.e. to Excel
instead of machine
readable interface
Some Experiences

…but suites quite
well for exchanging
(subsets of) data.
Data format is too
verbose for clients to
use directly…
Some Experiences

For now,
very few have
been interested in
forecast models
as a grid data
Point forecasts,
observations and
radar images are the
most interesting data
types
Some Experiences

www.fmi.fi
http://www.slideshare.net/tervo/
https://en.ilmatieteenlaitos.fi/open-data

Producing INSPIRE Compliant Data Sets

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to Producing INSPIRE Compliant Data Sets

Similar to Producing INSPIRE Compliant Data Sets (20)

More from Roope Tervo

More from Roope Tervo (10)

Recently uploaded

Recently uploaded (20)

Producing INSPIRE Compliant Data Sets