Processing data in GeoServer with WPS and SQL views

Processing Data In GeoServer
With WPS And SQL Views
Ing. Andrea Aime, GeoSolutions
Ing. Alessio Fabiani, GeoSolutions
FOSS4G-Europe 2014, Bremen
14th-17th July 2014

FOSS4G Europe 2014, Bremen
14th-17th July 2014
Some historical perspective
 Year 2008:
 Publish maps on the net
 Nice styling, maybe time/elevation based selector, some
little extra filtering
 Maybe some editing, some PDF printing
 Year 2013:
 All of the above, but…
 We can hardly make a new application without some
data processing in it

14th-17th July 2014
Web Processing Service
 Wikipedia introduces OGC WPS as:
 [A service] designed to standardize the
way that GIS calculations are made
available to the Internet.
 WPS can describe any calculation
including all of its inputs and outputs,
and trigger its execution
 The specific processes served up by a WPS
implementation are defined by the owner of that
implementation.
 Although WPS was designed to work with spatially
referenced data, it can be used with any kind of data.

14th-17th July 2014
An Example
 Buffer a L shaped
geometry with
distance “2”
 Get the result back
as GML

14th-17th July 2014
GeoServer WPS integration
WPS
Remote
WCS
Remote
WFS
HTTP
server
WPS
client
All GeoServer
Layers
WMS
clientWMS
GeoServer
UI

14th-17th July 2014
Rendering transformations
 On-the-fly data transformations inside rendering
chain
 Calling WPS processes from SLD docs
 Optimized for performance

14th-17th July 2014
• GeoServer ships with lots of built-in processes
• Basic geometry manipulation
• Aggregation options missing from WFS
• Raster and vector clipping
• And more!
• Ability to chain processes to build more complex
functionality
• Open API to manage inputs and outputs
• See all of it at http://demo.geo-
solutions.it/share/foss4g2011/wps_aaime_foss4g20
11.pdf
More On GeoServer WPS

When the WPS does not cut it…

14th-17th July 2014
• Never under-estimate the processing power of
your BDMS:
• Designed to efficiently juggle large quantities of data
• Efficient spatial primitives (at least, in PostGIS)
• Doesn’t get more local to your data than this!
• Passing params down?
• Parametric SQL views!
Spatial DBMS!

14th-17th July 2014
Parametric SQL views
WMS/WFS client
GeoServer
Spatial database
&viewparams=low:2000000;high:5000000
Expanded Query

Enough theory
Let’s get down to business!

14th-17th July 2014
Mapping Tuna Catches
• Multiple
Filtering
• Aggregation
• Joining
quartely
stats
against the
grid

14th-17th July 2014
Filtering, joining and aggregation
Some example control regexps:
• Y_INTERV:
• Default: 1
• Regex: ^(d)+$
• OP
• Default: sum
• Regex: ^[avg|sum]$
SELECT (T.TS_VALUE / %Y_INTERV%) AS TS_VALUE,
T.CD_TA_OCEANAREA, G.GEOMETRY
FROM
(SELECT CD_TA_OCEANAREA,
OP%(TS_VALUE) AS TS_VALUE
FROM FIGIS.TS_FI_TA
WHERE FIC_ITEM IN (%FIC_ITEM%)
AND CD_GEAR IN (%CD_GEAR%)
AND YR_TA IN (%YR_TA%)
AND QTR_TA IN (%QTR_TA%)
GROUP BY CD_TA_OCEANAREA
) t
LEFT OUTER JOIN FIGIS_GIS.GRID_G5 g
ON T.CD_TA_OCEANAREA = g.CD_OAREA
ORDER BY T.CD_TA_OCEANAREA

14th-17th July 2014
• Animator tool
• GetMap + variying parameters + frame control =>
animated GIF!
• http://docs.geoserver.org/stable/en/user/tutorials/animr
eflector.html
Animation

14th-17th July 2014
• …/geoserver/wms/animate?request=GetMap&…
&format=image/gif;subtype=animated
&aparam=viewparams:YR_TA
&avalues=2000,2001,2002,2003,…
Animator

What’s the wheater like today?

14th-17th July 2014
The Situation
• MetOc data is extremely dynamic
• In-Situ sensors acquire data in near real-time
• Meteorological and Oceanographic model runs
multiple times a day
• Remote Sensing data is acquired at fast pace
• Meteo Radar data is acquired at fast pace
• Large amount of data is produced to model fast
changing environmental variables
• We must shorten to the small possible extent the
time and resources for preprocessing!

14th-17th July 2014
Solutions: On-the-fly contouring

14th-17th July 2014
Solutions: On-the-fly Wind-Barbs

14th-17th July 2014
Solutions: On-the-fly Currents

14th-17th July 2014
Solutions: On-the-fly Storm Tracks

14th-17th July 2014
• New addition to the MapStore project
• Requirements
• Download large amounts of data
• Generic data filtering
• Clip on polygon/bbox/circle, both vector and raster
• Reproject to target CRS
• Mail notification when download is ready (or fails)
• Work in a cluster
• Solution: new WPS processes, asynch WPS calls
and WPS customization for state sharing
Advanced Clip and Ship

14th-17th July 2014
MapStore GUI
Buffer process called synchronously when buffer size
changes

14th-17th July 2014
MapStore GUI
Tracking download status (asynch WPS)

14th-17th July 2014
Download service architecture
WPS
All GeoServer
Layers
MapStore
WMS
GetCapabilities
List of layers
Buffer
DownloadEstimator
Download
GetStatus
Fetch data
Status database
Store and
retrieve
status
Mail server
Custom
process
manager
Start/
Complete/
Failed
notifications

14th-17th July 2014
• The Integrated Decision Aid (IDA) has been developed at the Centre for Maritime
Research and Experimentation (CMRE) in order to provide naval exercise
planners with an effective tool to assess risk to marine mammals from acoustic
transmissions.
• Assessment of potential hazard is performed through sound propagation
modeling available in the interface.
Intro
Sound
Propagation
Model

14th-17th July 2014
Sound Propagation Model
• External Octave Matlab Process
• Logs the asynchronous process status to the DB
as a Feature
Octave Matlab Script
for SPM
WFS
Follows the process
status through the
WFSWFSLog
Process allows
to insert and
update the
status of SPM
into the DB
Dynamically builds an external Octave
command line using the input parameters
through a FreeMarker Template (FTL)

14th-17th July 2014
• The list of model runs is
available on the bottom Data
Grid.
• It is possible to get
information on the
input parameters and
execution from
the WFS
Sound Propagation Model

14th-17th July 2014
• Advanced Raster Algebra using Jiffle
• Wraps Jiffle (http://code.google.com/p/jiffle/) in order to
execute complex Raster Algebra scripts against the input
layers
Raster Algebra Processes
Jiffle is a scripting language for creating and analysing raster images. Rather than
having to write and test lots of JAI or Java AWT boiler-plate code to access and
manipulate images, Jiffle lets you concentrate on the interesting bit: your algorithm.
Jiffle is being developed as part of the JAITools project.
Update 16 January 2013: Jiffle sources have moved to GitHub

14th-17th July 2014
• Advanced Raster Algebra using Jiffle
• Gets an OGC Filter containing logical operations between
raster layers and produces a binary light layer

14th-17th July 2014
• Raster Algebra detailed info
• Input parameters and outcomes are logged into WFS
• WPS Raster Stats Process provides statistics on the
coverages
Raster
Statistics and
Area in sq Km
Raster Algebra
Process user
inputs

Script me plenty
An example using GeoScript

14th-17th July 2014
Sample scripting application
http://localhost:8080/geoserver/www/wps.html

14th-17th July 2014
Land property distrib. in the polygon

14th-17th July 2014
A look at the data dir
1. Data
2. WPS process scripted in Jython
3. The GUI

14th-17th July 2014
The Jython script
http://geoscript.org/py/index.html

14th-17th July 2014
• The Destination project computes
the risk of accidents involving
dangerous goods (chemicals,
petrol, gases and so on)
Intro
Road segments
and stats about
car accidents
Human and environmental «targets»
Involved area,
depending on
type of good and
amount of
damage

14th-17th July 2014
• Road network of good part of northern Italy
• Road divided into segments
• 100m portions (500k of them)
• 500m aggregation (120k of them)
• 1 km square cells (few hundreds)
• 51 buffer distances (depending on good,
scenario, level of damage)
• Several types of targets: schools, malls,
hospitals, populated areas, superficial and
underground acquifers, crops, woods, ….
Large Data Volume

14th-17th July 2014
The road arc risk formula (s)
• Adding togheter the risk caused by the different
• Arc own propension to accidentds
• Types of goods
• Human and enviromental targets
• The system allows to compute partial views of the
formula, either by selection of targets/goods or by
computing portions of it
• Has a number of coefficients that can be hand-
tuned by the caller

14th-17th July 2014
The results, visually
• Rendering
transformation
• Read the
arcs/polys from
the DB, compute
their risk based
on the chosen
formula,
scenario, targets,
and coefficients

14th-17th July 2014
How to compute it efficiently?
• Using SQL Views? No, the possible aggregations
variants are too many
• Using a pure Java process? No, too much data to
transfer from the DBMS
• Fully on the fly? No, too much data involved
•  Pre-compute all buffers and locate all involved
targets before hand (pre-cooked per buffer risk)
•  Use a process that builds a final aggregation
query on the fly (dynamic sql views)

14th-17th July 2014
Efficient rendering tx
Risk
process
Queries
database
Arcs/Buffer
areas db
Map
renderer
Build overall
query, replace
params
Compute risk for
a batch of arcs
Raw arcs
Arcs + risk
• Compute risk
on the fly in
the viewing
area
• Batch
requests to
the BDMS to
minimize
round-trip
overhead

14th-17th July 2014
Efficient cross layer filtering
• Show only targets
involved in the
scenario under
study, e.g., the ones
crossing the buffer
areas where there is
significant risk
• Limit query to the
current bbox

14th-17th July 2014
Efficient cross layer filtering
SELECT v_geo_popolazione_residente_pl.*
FROM v_geo_popolazione_residente_pl
WHERE v_geo_popolazione_residente_pl.fk_bersaglio_umano_pl in
(
SELECT distinct bersaglio.fk_bersaglio_umano_pl
FROM v_geo_popolazione_residente_pl bersaglio
join siig_geo_ln_arco_1 on
st_dwithin(bersaglio.geometria,
siig_geo_ln_arco_1.geometria,
%distanzaumano%)
WHERE siig_geo_ln_arco_1.geometria &&
st_makeenvelope(%bounds%, 32632)
)
• This is a job for a parametric sql view

14th-17th July 2014
The End
Questions?
andrea.aime@geo-solutions.it
simone.giannecchini@geo-solutions.it

Processing data in GeoServer with WPS and SQL views

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (12)

Mehr von GeoSolutions

Mehr von GeoSolutions (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Processing data in GeoServer with WPS and SQL views