3. GeoSolutions
Founded in Italy in 2006
Expertise
• Image Processing, GeoSpatial Data Fusion
• Java, Java Enterprise, C++, Python
• JPEG2000, JPIP, Advanced 2D visualization
Supporting/Developing FOSS4G projects
GeoTools, GeoServer, MapStore
GeoBatch, GeoNetwork
Clients
FAO (CIOK, FIGIS, NRL, FORESTRY, ESTG), JRC, ARPAT, Nato
CMRE, DLR, EUMETSAT, WFP
ITT-VIS, E-GEOS, GEOSMART, OpenGeo, SINERGIS, City of Prato, City of
Florence, County of Florence, County of Prato, CSI-
Piemonte, NWGEO, IGEA, AMBRERO, LIBEROLOGICO, Astrium UK, EOMAP, City of
Wien, County of Bozen, City of Bozen, etc…
http://www.geo-solutions.it
8. RESTful Configuration
Programmatic configuration of layers via REST
calls
Workspaces, Data stores / coverage stores
Layers and Styles, Service configurations
Freemarker templates (incoming)
Exposing internal configuration to remote clients
Ajax - JavaScript friendly
Various client libraries available in different
languages (Java, Python, Ruby, …).
Example, geoserver-manager:
https://github.com/geosolutions-it/geoserver-manager
9. Security: Authentication
Pluggable user sources, available out of the box:
LDAP, DBMS
Pluggable authentication mechanisms, available
out of the box:
BASIC/DIGEST HTTP, CAS
Possible to integrate with other mechanisms and
in-house solutions
Available since 2.2.0, before only basic HTTP auth
+ simple text file for users
10. WMS
Dissemination of Maps
Fusing raster and vector data seamlessly
Rule/scale driven rendering
WMS 1.1.1 and 1.3 support
SLD
Basic support for SLD 1.1 and SE 1.1
Full support for SLD 1.0
CSS extension for compact styling
Many rendering extensions available
12. WMS: TIME and ELEVATION
TIME = 20100512T0000000Z
ELEVATION = 0.0
FeatureType Editor
13. WMS: Rendering Transformations
On-the-fly data transformations
Calling spatial analysis processes from SLD docs
Optimized for performance
Examples: on the fly contour lines, heat maps,
point clustering, point interpolation, GCP based
image rectification
14. GeoWebCache Integration
GeoServer
GeoWebCache
Persistent raster/KML
tile cache
Direct calls to GeoServer
rendering engine
Support for layers modified
through WFT-T
Support for various tile
protocols
GMap, Gearth
OpenLayers, VEarth,
Bing
Speed-up factor 10/100
Disk quota support
17. WFS
Dissemination and filtering of vector data
WFS 1.0, 1.1 and 2.0 (since 2.2.0)
Transaction and paging available in all versions
Simplified filtering via CQL
Formats:
GML 2, 3.1 and 3.2
CSV, Excel spreadsheet, GeoRSS, GeoJSON
Shapefile (zipped)
Any other format supported by ogr2ogr (configurable)
More output formats can be generated by XSLT
18. Complex Feature*
Application/community schemas
Complex Features
Attributes as sub-features
Attributes as list of features
Tree-like structure
Mixing in a single tree heterogeneous data sources
19. WCS
Raster data dissemination
Raw raster data useful for analysis, no maps!
Support for TIME and ELEVATION (via ImageMosaic plugin)
WCS 1.0 and 1.1.1
Output formats
GeoTiff, ArcGrid
GDAL based formats under discussion
Extensions
ELEVATION as band management
WCS 2.0.X Implementation Ongoing
20. WPS
WPS 1.0
Official Extension
Raster and Vector data support
High performance processes (raster/vector
statistics, raster/vector format conversions and
more), asynchronos call support
Integrated WPS
Direct access to data sources
Automatic publishing of results as new layers
Embedding processes into SLD styles (rendering
transformation, since 2.2.0)
21. Database configuration backend (2.3)
Pluggable configuration backends
In-memory implementation + XML storage (current
one)
Database based implementation (as a community
module)
Pluggable, add your own (any takers for a NoSQL
elastic implementation?)
In-memory
GeoServer config and catalog
DBMS NoSQL?
22. GWC clustering (2.3)
Improved clustering for GWC in 1.4.x:
Metastore removed
Disk quota can work off a central DBMS
Distributed locks, avoid concurrent computation of
same tile at the same time
Active/active clustering
of GWC now possible
GWC1
Tile storage Disk quota
GWC2
Load balancer
23. CSW 2.0.2 (2.3)
Wow, catalogue services in GeoServer!
Catalog Service for the Web 2.0.2
Pluggable record backend
Pluggable record type support
Not a replacement for a full-fledged GeoNetwork
(not at the moment, at least)
Dublin core
support
GeoServer
config
backend
CSW protocol handler
ebRIM
support
ISO
support
Simple DC
backend
Vendor
specific
bridge
Record type
plugins
Record sources
24. WCS 2.0 EO / WMS EO (2.3)
WCS 2.0 implementation with extensions:
Range subsetting
Scaling and interpolation
CRS (reprojection)
GeoTiff & NetCDF encoding
Earth Observation profile support
Temporal series
Exposing mosaic structure
EO metadata describing sensors
NetCDF support as both input and output
Sponsors
DLR (German spatial agency)
EUMETSAT (European operational satellite agency for
monitoring weather, climate and the environment)
26. Work Plan – The great plan
Goals
Better support EO and MetOc data in GeoServer
Implement WCS 2.0.1 in GeoServer
Implement WCS-EO in GeoServer
Implement WMS-EO in GeoServer
Support serving NetCDF Data from GeoServer
Improve ability to manage Multidimensional data via REST in
GeoServer
Activities
WCS 2.0.1
WMS-EO
WCS-EO
Ingestion with GeoBatch
Work on multidimensional input formats
PostgisRaster support
27. Work Plan – WMS-EO
Extending LayerGroup
concept
Support same style on
both raster and vector
data
Support custom
dimensions
Alter map on the fly to
support band
combination
Wizard to configure EO
layer groups
How this fits in
GeoServer’s world
GSIP 84
28. Work Plan – WMS EO
WMS-EO Quirks
Root layer must respond with a specific layer rather than the
composition of the children
Kind of a default visualization for a certain Dataset
E.g. when I send a GetMap for the root layer I could get back the browse image
default layer
Same styles for both Raster as well as Vector data
E.g. yellow can be use for both flags as well as outlines
Peculiar behavior for Band layer ( raw data )
E.g. multiple bands at different wavelengths
Can request either 1 (grayscale image) or 3 (RGB image)
Different combinations are prohibited
Peculiar Behavior for GetFeatureInfo
29. Work Plan – WCS 2.0
Build the basics
Core service
KVP binding
XML binding
Implement the GetCoverage extensions
CRS
Scaling
Interpolation
Range subsetting
GeoTiff
GML
NetCDF
Add the output format extensions
GeoTIFF
GML Grid
Ensure CITE tests compliancy
31. Work Plan – WCS-EO
Build on top of a working WCS 2.0 with full extensions
WCS 2.0
CRS extension
Range subsetting extension
Interpolation extension
Scaling extension
GeoTiff extension
NetCDF extension
Add support for the WCS-EO extras
Listing coverage datasets in the capabilities documents (based on image
mosaic contents, which will have to be marked as “exposed” so that we
show their inner structure for EO)
Support describe coverage dataset
Support returning results for an entire dataset in GetCoverage
32. Work Plan – WCS-EO
Add support for downloading the original file in case of no
subsetting/reprojection/scaling/format change
Add support to GeoTools readers to signal they are returning us an
original file
Use that information to download the original file directly
Add support for WCS EO metadata in readers
Associate each file with EO metadata
Include such information in DescribeCoverage/DescribeEODataset
Ensure CITE tests compliancy
33. Work Plan – Additional Format Support
NetCDF support
Improve existing NetCDF/CF input format, support CF convention and
make sure the samples provided by DLR/EUMETSAT can be read
Expose NetCDF internal data as a set of 2D slices
Write new NetCDF/CF output format for GeoServer
34. Work Plan – Additional Format Support
ImageMosaic NetCDF integration
Allow the ImageMosaic to handle NetCDF files
Expose NetCDF internal structure (times, elevations)
Make ImageMosaic handle slices of the NetCDF file as granules
Mosaic
NetCDF 1 NetCDF 2 NetCDF … N
…
35. Work Plan – Additional Format Support
ImageMosaic
improvements
Mark returned coverage as
single granule when a
single, whole file has been
used to generate the output
Expose the inner structure to
support DataSet time and
stitched mosaic notions
Allow manipulation of the index
to support REST config
extensions
36. Work Plan – REST Interface
Improvements
Add REST support to expose a image mosaic internal structure
Dimensions
Granules
Dimensions: list, edit, create, remove
/workspaces/<ws>/coveragestores/<cs>/coverages/<mosaic>/dimensions
/workspaces/<ws>/coveragestores/<cs>/coverages/<mosaic>/dimensions/<dimension>[.format]
TODO paging and query of dimension domain to overcome WMS Dimensions limitations
Granules: list, edit, create, remove
/workspaces/<ws>/coveragestores/<cs>/coverages/<mosaic>/index
/workspaces/<ws>/coveragestores/<cs>/coverages/<mosaic>/index/pageN
/workspaces/<ws>/coveragestores/<cs>/coverages/<mosaic>/index/pageN/granuleM
38. Current Status - Overview
WMS EO
Layer group extension complete
GeoServer support for static custom dimensions added
Custom dimension support added to image mosaic, GeoServer support
for dynamic dimensions complete
WCS 2.0
Basic implementation completed
GetCoverage extensions completed: crs, interpolation, scaling, range
subsetting
GeoTiff request parameters and compliant output format
GML grid output format
WCS-EO
Basic extensions are in already
NetCDF reader support
Initial work on low level readers
Enhancement to ImageMosaic
Support for Custom dimensions
39. Current Status – WMS-EO
• Created GeoServer improvement proposal
• GSIP 84
• Discussed the proposal on the mailing list:
• See here and here
• LayerGroup Enhancements
• Added Type property
• Single: the layer group as today, a single exposed layer with a name
• Named tree: the layer group retains a Name in the layer tree, but also
exposes its nested layers in the capabilities document
• Container tree: the layer group is exposed in the tree, but does not have a
Name element, showing structure but making it impossible to get all the layers
at once
• Earth Observation tree: created to manage the earth observation
requirements
• Added Root Layer and Root Layer Style, used only when Type EO is
selected
• Added methods renderingLayers() and renderingStyles()
40. Current Status – WMS-EO
Updated GeoServer web administration interface to edit
new LayerGroup properties.
41. Current Status – WMS-EO
Added Support for custom dimensions to GeoServer
“Dynamic dimensions”, dimensions whose existence the reader gets to
know about at runtime
42. Current Status – WMS-EO
• Updated GetCapabilities and GetMap request handlers for
WMS 1.1.1 and WMS 1.3.0
Nested layer
Nested layer
Named container
43. Current Status – WCS 2.0
GeoTools
EMF models
XML Parsing
Geoserver
Service architecture: changes in input params
OWS2 allows repeated keys in KVP
http://osgeo-org.1560.n6.nabble.com/WCS-2-0-multiple-kvp-
keys-with-different-values-here-they-come-td5021302.html
Created WCS 2.0.1 service
Community module
Request thread http://osgeo-org.1560.n6.nabble.com/New-
community-module-WCS-2-0-EO-td5015927.html
Implementation here
44. Current Status – WCS 2.0
Geoserver
Implemented operations
GetCapabilities (KVP GET + XML POST)
DescribeCoverage (KVP GET + XML POST)
GetCoverage (KVP GET + XML POST)
Implemented Handling exceptions according to OWS2.0
Some differences from v1:
Referenced schemas
Introduced support to HTTP response codes
So far we did not focus on support for Time and
Elevation
45. Current Status – WCS 2.0
CRS Extensions is in
Interpolation Extension is in
Range Subsetting Extension is in
Scaling Extension is in
GeoTiff Extension is in
GML Output is in
46. Current Status – CITE Tests
After some fixing on CITE test definition for WCS 2.0, 101 tests are
passing and 14 are failing.
The base repository used for tests is
https://github.com/geoserver/geoserver-cite-tools
Patch for the CITE tests is at https://github.com/geoserver/geoserver-
cite-tools/blob/master/20121213_tests_wcs_200_tags_r5_ctl.patch
Patch info at https://github.com/geoserver/geoserver-cite-
tools/blob/master/20121213_tests_wcs_200_tags_r5_ctl.readme
Tickets already opened at http://cite.opengeospatial.org/issues
First open ticket is at
https://portal.opengeospatial.org/services/srv_public_issues.php?call=
viewIssue&issue_id=722 -- the latest ones have not been
acknowledged yet.
47. Current Status – NetCDF as Input
Multidimensional Datasets management
I/O, metadata management and ND APIs
Low level - 2D Slices Readers (ImageIO-Ext):
ImageIO readers, format specific ImageReaders (NetCDF)
(ImageIO Direct readers)
High level - Coverage API (GeoTools):
CoverageStore, CoverageAccess,
CoverageSource, Driver (GeoTools ND plugins)
SpatioTemporal ImageReaders (GeoTools smart
readers)
Build spatioTemporal metadata
Expose SliceDescriptors describing 2D Slices
(temporal extent, vertical extent, imageIndex, envelope)
48. Current Status – NetCDF as Input
Spatiotemporal Metadata
http://demo.geo-solutions.it/share/ImageIOMetadataDoc.pdf
Collecting domains information
CoordinateReferenceSystems
RectifiedGrid
Boundings
Bands
Simplifications to be done
spatioTemporal metadata
sampleDimensions (Bands)
ND dimensions to 2D slice mapping
49. Current Status – NetCDF as Input
ImageIO-Ext NetCDF low level access libraries/plugins
https://github.com/geosolutions-it/imageio-ext.git
https://github.com/geosolutions-it/imageio-ext/tree/1.1.x/library/netcdf-core
https://github.com/geosolutions-it/imageio-ext/tree/1.1.x/plugin/unidata-netcdf/netcdf
2D slices of ND datasets accessed through direct imageIndex-es
Index to slice mapping: COARDS convention: t,z,y,x
global and variable’s attributes are exposed (needed to build metadata)
Currently based on 4.0.41 (consider switching to 4.3)
GeoTools multidimensional rasters and NetCDF Driver
https://github.com/simboss/geotools/tree/master-multidimraster
/modules/unsupported/coverage-experiment
/modules/unsupported/coverage-experiment/netcdf
SpatioTemporalImageReaders
Wraps ImageIO-Ext direct readers to provide SliceDescriptors and populate
spatioTemporal metadata (use Direct reader to access exposed datasets
attributes and dimensions)
NetCDFSpatioTemporalImageReaders
Parse NetCDF attributes and metadata to setup SpatioTemporal metadata and SliceDescriptors
50. Current Status – Things to clarify
Coverage names are defined as NCName in OWS2
Thread about prefixed names escaping: http://osgeo-
org.1560.n6.nabble.com/Surprises-in-WCS-2-0-coverage-names-td5020561.html
ExceptionReport version when a service/version pair can
not be inferred from the request
Thread: http://osgeo-org.1560.n6.nabble.com/Arbitrary-choice-when-sending-back-service-
exceptions-td5022804.html
Doubts with AxesLabels VS AxisIdentifers for subsetting
Doubts with AxesLabels VS AxisIdentifers for scaling
Missing Array-Type Range Type
Still a little unclear on Time management (potential PICNIC)
Not sure how to manage non equally-spaced Elevation
levels
Using the Bands with multivalued domain the past, now hard to do this way
51. Current Status – Blockers
WCS Extensions are moving targets
did a first implementation, we’ll have to revisit once they settle down
CITE tests are available for WCS 2.0
2.0.1 version with tests for extensions is in heavy development by OGC
right now
Patches for CITE tests should be ack by OGC
Some fixes in CITE tests code are needed
Some OWS2 requirements are not being incorporated
HTTP return codes
ExceptionReport schema version
Missing WCS extensions
RangeSubset for other types of Field?
53. Next Step - Overview
WMS EO
Create community module for WMS-EO specific extensions:
GetMap for bands merging
Complete wizard for WMS EO layer group creation
WCS 2.0.1
Improve testing
Review extensions implementations once spec settles down
Test against CITE tests for WCS 2.0.1 once they are ready
WCS EO
Build support for datasets (by marking mosaics as dataset sources)
Amend GetCapabilities output
Create DescrieEODataset and its output
Add support for datasets in GetCoverage, as well as extracting a
particular coverage out of them
54. Next Step - Overview
REST configuration improvements
Add support for programmatically manipulating mosaics
Expose these capabilities via REST
Additional input/output formats
Complete work on NetCDF input, expose inner structure as a set of 2D
slices
Extend mosaic to integrate NetCDF slices as granules
Prepare NetCDF output
Extend mosaic to support PostGIS rasters
Ingestion work
Moving WCS 2.0 community module into standard
extension
55. Next Step – Possible Issues
WMS-EO
Band Coverages:
1 Dimension – Greyscale rendering
3 Dimensions – RGB rendering
Other number of Dimensions - WMS-EO server should throw an error and
this behaviour differs from WMS standard
Rendering will require additional processing: WMS service must recognize
Band layers and render them accordingly to WMS-EO specification.
Root Layer must have a Time Dimension
This Time Dimension will be inherited by all contained Layers.
Outline Layers
a Time Dimension parameter will be required for getMap and getFeatureInfo
request. It could be a time range
getFeatureInfo requests must return product information and acquisition time
56. Next Step – Possible Issues
Uncertainty of WCS extensions future modifications and
timing
Uncertainty of WCS 2.0.1 CITE tests completion and
availability
Uncertainty of WCS EO CITE tests completion and
availability
Refactoring to support PostGIS rasters, possible
community issues on overlap with JDBC image mosaic
Window to make large changes in GeoServer core closes
January 21 2013, we can continue work on trunk but it will
have to go though proposals for backport or delayed to
2.4.x. series (due September 2013)
57. Next Step – Possible Issues
NetCDF input support
Entirely new approach requires deep modification to GeoTools-
ImageMosaic/GeoServer
Potential performances/memory issues may happen
(with respect to using GeoTiff based imageMosaics VS using NetCDf
based ImageMosaics)
Vertical dimensions specifications
Nightmare in some cases with compound CRS
58. Conclusions
WCS 2.0.X specs look way better then 1.X
Nice usage of extensions
Clear model, Clear way of thinking
Still some grey areas
So far good, steady progress has been made
Minor issues in adapting GeoTools and GeoServer to the specs
Idiosyncrasies with CITE tests
Some delays from the Community
Also positive feedback from the Community
Challenging Tasks ahead of us
Multidimensional data
WCS-EO
Linking WMS-EO to data
TIME and ELEVATION
Ingestion
60. Rendering: real world units
<Rule>
<LineSymbolizer uom="http://www.opengeospatial.org/se/units/metre">
<Stroke>
<CssParameter name="stroke-width">
<ogc:Literal>5</ogc:Literal>
</CssParameter>
</Stroke>
</LineSymbolizer>
</Rule>
61. WMS: PNG8 with alpha
Support for paletted PNG with alpha transparency
Best of both worlds: compact but good looking
Good quality, yet usable in interactive setups
62. Virtual services
Expose different OGC services per workspace
Styles and layer groups per workspace
Have different
administrators per
workspace
multi-tenancy
63. WFS: 2.0 and XSLT
WFS 2.0
GML 3.2
Paging (back-ported to other versions)
Joins (scalar, temporal, spatial) between
feature types
Stored queries
XSLT output
format: GML
2.X/3.X
Custom
XSL
sheet
XLST
engine
txt
Custom xml
HTML
KML
64. CSW 2.0.2 (2.3)
Current implementation
Demo backend with Dublin Core record support,
passes CITE certifications tests
ISO + Dublin core backend reporting layers in the
GeoServer configuration, in development
ebRIM (Earth Observation profile) implementation
plus proxy to a in-house, vendor specific catalog
(proxy front-end model)
Currently a community module
will be graduated to extension once the ISO backend
over the GeoServer own config is completed
65. Work Plan – The great plan
Milestones
M1, Dec 2012
WCS 2.0
WMS-EO
M2, End of Feb 2013
Additional Formats
REST Improvements
M3, End of March 2013
WCS-EO
Ingestion
M4, End of June 2013
Support
66. Work Plan – WMS-EO
Abstract Model
Exposing a meaningful (EO-Wise) complex structure
Raw data, products, browse images, footprints
Heavy usage of WMS Layer Inheritance (with some quirks)
67. Work Plan – WMS-EO
Extending the LayerGroup GeoServer concept
Show the nested layers in the capabilities document
Allow the root of the group to be represented by a separate layer (the
overview one)
The above is a set of API, GUI and REST config changes, so a
GeoServer proposal is needed
Add the notion of custom dimensions in raster data
Needed to support the “eoproduct_bands” layer
Modify the GeoServer API to support custom dimensions (was almost
ready in this respect)
Modify the grid coverage readers API to allow new dimensions to be
exposed
Allow “dynamic” dimensions to be exposed (dimensions that are
configured by the user)
Adapt the GUI to allow new dimensions to be configured
68. Work Plan – WMS EO
Support for band merging
Add extension point to GeoServer to change the layer/styles on the
fly requires a proposal
Throw exceptions if outside of the supported cases (one band for
gray, three bands for RGB)
Wizard- base configuration
Configure easily a EO layer group from a mosaic and a set of layers
Automate creating footprints
Automate creating bands
Cherry pick geophysical and mask layers
Automatically associate the predefined styles to the mask layers and
footprints
Tricky Stuff
Allow using the same style name on both mask layers and footprint
layers
Develop a new filter function to choose which symbolization to use based
on the layer nature
69. Current Status – WMS-EO
Additional Domains support on GeoTools
GEOT-4347
AdditionalDomainAttributes for indexer.properties
See example here
Reader support getDynamicParameters
Custom mosaic dimensions <-> dynamic read parameter descriptors
Currently only Strings are supported
WMS Service Enhancements
Get metadataNames and metadataValues from the reader
Get dynamicParameters from the reader
Combine them to send the request to the coverage reader
70. Current Status – WCS 2.0
Support for CRS extension (both XML and KVP):
Specify subsettingCRS (for crop in world coordinates)
Specify outputCrs (for reprojection of output)
71. Current Status – WCS 2.0
Support for interpolation extension (nearest-neighbor,
linear, cubic, both XML and KVP):
We don’t supported different interpolations for Lat,Lon
We don’t support
Lost area
Barycentric
Quadratrci
We will strive to support Nearest for Time dimension
72. Current Status – WCS 2.0
Range subsetting support (both XML and KVP):
Based on SWE:DataRecord
No support yet for bands referring to arrays of values, like
Wavelength
73. Current Status – WCS 2.0
Scaling support, uniform factor, different factor by axis, to
specified size, to specified extent (both XML and KVP)
74. Current Status – WCS 2.0
GeoTiff extension, setting parameters for the generated
GeoTiff (both XML and KVP)
Band interleaving not supported
Floating Point predictor not supported
75. Current Status – WCS 2.0
GML grid as output format
Grid linear scan from Top-Left
Single field per Band
We might improve names soon
Notice the choice of axes
Names for the grid
i,j,k as per columns,rows ordering
76. Current Status – NetCDF as Input
NetCDF Sample datasets have been analyzed
http://demo.geo-solutions.it/share/DLRsampledatasetsReport.pdf
Polyphemus (t,z,lon,lat grid)
Gome2 (lon, lat grid)
BlackForest (lon, lat grid) – small dataset
Eumetsat ASCAT (complex datasets on swath grids) – geolocated, not
georectified
Additional NetCDF Samples from MetOc world have been
collected
Data which needs to be analyzed and added to the report
OSTM Jason2 – 1D dataset (time)
Eumetsat IASI – (pressure level, surface emissivity, wavelenght, surface
temperature, cloud formation and irregular lon,lat grids) – need
interpolation
Eumetsat Hyperspectral+sounding IASI – (huge dataset)
78. Current Status – CITE Tests
First open ticket is at
https://portal.opengeosp
atial.org/services/srv_pu
blic_issues.php?call=vie
wIssue&issue_id=722
We opened 4 more
tickets, but they have
not been acknowledged
yet
79. Current Status – WCS 2.0
Unit/integration test coverage: 83% as reported by
Cobertura
80. WPS: asynchronous calls
Asynchronous WPS support for long running
processes
WPS protocol
handler WPS executorClient
Execute
Internal submit
Status poll
Update status
Update status
Write results
Status poll
Status?
Results