GRASS GIS has contained remote sensing tools for decades, catering to the needs of users since the first generations of Landsat satellites. In this talk we will present the current efforts of integrating modern data sources and modern approaches into GRASS GIS. Tools exist for pre-processing recent very high resolution images, object-based image analysis (OBIA), Lidar data handling, etc. At the same time, efforts have gone into ensuring the scalability of tools for huge data sets. The presentation will provide a short brief on the general state of GRASS GIS development, go on to an overview of the remote sensing tools available, to end with a use case on how to use GRASS GIS for time series processing in a high performance cluster/grid computing environment.
GRASS GIS has been existing for over 30 years by now and provides a very large and diverse set of state-of-the-art tools for the analysis of spatial data. Less known by many, remote sensing tools have been part of it almost from the beginning. GRASS GIS provides a series of imagery analysis tools for pre-processing (radiometric correction, cloud detection, pansharpening, etc), creating derived indices (vegetation indices, texture analysis, principal components, fourier transform, etc), classifying (management of training zones, different classifiers, validation tools), and producing other derived products such as evapotranspiration and energy balance models. Next to these tools for satellite images, other tools exist for the handling of aerial photography for creation of orthophotos, and for the import and analysis of Lidar data.
In addition to these tools, efforts have gone into integrating current state-of-the-art methods such as object-based image analysis and machine learning. A complete toolchain exists to segment images using different algorithms, to create superpixels, to collect statistics characterizing the resulting objects, and to apply machine learning algorithms for classification. New modules also include unsupervised segmentation parameter optimization and active learning. Options for pixel-based classification have also been enlarged to a host of machine learning algorithms.
A specific aspect of treating the rapidly increasing amounts of satellite data is the scalability of tools. Through its modular structure, GRASS GIS also allows to easily parallelize certain operations, thus opening the door to the use of cluster/grid computing environments.
Speakers
- Markus Neteler
- Moritz Lennert
- Markus Metz
Source + video registration:
https://fosdem.org/2018/schedule/event/geo_grass
GRASS GIS as high-performance remote sensing toolbox
1. GRASS GIS in the sky
GRASS GIS as high-
performance remote
sensing toolbox
Markus Neteler, Markus Metz, Moritz Lennert
https://grass.osgeo.org/
2. FOSDEM 2018 GRASS GIS in the sky 2/20
GRASS GIS Intro
r3.flow
Skyview factor
from LiDAR
g.gui.tplot
● Geographic Resources Analysis Support System
● Open Source GIS
● developed since 1984, since 1999 GNU GPL
● Portable code (many operating systems, 32/64bit)
● Your GIS backbone – linkable to:
3. FOSDEM 2018 GRASS GIS in the sky 3/20
GRASS GIS and Python
Using GRASS GIS from “outside” through “grass-session”
pip install git+https://github.com/zarch/grass-session.git
# in future, the stable "grass-session" release
# will be available with:
# pip install grass-session
Finally easy use of GRASS GIS
as a processing backend in Python!
Combine with GDAL, OTB, ...
4. FOSDEM 2018 GRASS GIS in the sky 4/20
GRASS GIS 7.4 Release
● New: Get demo data at start screen
● GUI: data catalog improved
● Most of ortho-rectification brought back
● r.in.gdal + r.external: provide support for import of
raster maps exceeding 90N or 90S or with an EW
extent larger than 360 degrees
● r.out.gdal: possibility to create overviews which
enhances the compatibility with other GIS software
packages
● v.clip added for easy vector clipping
● … (480 fixes and improvements with respect to
7.2.0)
https://trac.osgeo.org/grass/wiki/Grass7/NewFeatures74
5. FOSDEM 2018 GRASS GIS in the sky 5/20
GRASS GIS 7.4 Release
● Atmospheric correction updated with new
satellites
● MODIS product processing made easy
6. FOSDEM 2018 GRASS GIS in the sky 6/20
Remote sensing in GRASS GIS :
a long history
● Imagery submodule already
available for GRASS 1.1 in 1986
● i.* modules fully integrated in
version 3.0 in 1988
● Steady improvement and additions
ever since
● From simple text terminal to X-
monitors to a modern GUI
● Memory efficient modules
7. FOSDEM 2018 GRASS GIS in the sky 7/20
Preprocessing
i.landsat.toar
i.pansharpen
i.atcorr
i.landsat.acca
i.topo.corr
Classification
i.maxlik
i.smapr.learn.ml
r.kappa
g.gui.iclass
● Pixel-based tools for satellite
and aerial imagery
● Most state-of-the-art methods
implemented
● Complete toolchain from pre-
processing to classification
● Many highly specialized tools
Transformation, edge
detection, index extraction
i.pca
i.fft
i.cca i.oifi.tasscap
i.wavelet
i.zci.edge
i.cva
Remote sensing in GRASS GIS :
pixel-based techniques
Specialized modules
i.evapo.*
i.biomassi.eb.*
i.albedo
i.gravityi.feotio2
i.water
i.lswt
8. FOSDEM 2018 GRASS GIS in the sky 8/20
● Complete toolchain from segmentation
to classification
● Including
– unsupervised segmentation
parameter optimization
– high performance object statistics
calculation
– module-level parallelization
● Recently created module for SLIC
superpixel creation
source : http://dx.doi.org/10.3390/rs9040358
Remote sensing in GRASS GIS :
object-based image analysis
9. FOSDEM 2018 GRASS GIS in the sky 9/20
● Suite of LiDAR data tools
● Suite for creation of orthophotos
● Current developments :
– convolutional neural networks
– cutlines for semantically sensitive tiling
– etc, etc
● Constantly growing list of extensions
● Permanent work on performance
improvements
Remote sensing in GRASS GIS :
plus so much more !
10. FOSDEM 2018 GRASS GIS in the sky 10/20
High-performance computing
Example: NDVI time series
Harmonic
Analysis of
Time Series
(HANTS)
11. FOSDEM 2018 GRASS GIS in the sky 11/20
High-performance computing
Components
– master with job/queue manager
– compute nodes
disk space per CPU core
RAM per CPU core
ideally one SSD per node
master
(compute node)
General HPC layout
compute
node
compute
node
compute
node Parallelization :
several GRASS commands
running at the same time
12. FOSDEM 2018 GRASS GIS in the sky 12/20
High-performance computing
Chunks for parallel processing
Temporal processing
● spatial chunks
need to be mosaiked at the end
or
● temporal chunks
need overlap
t
1 2
3 4
2
1
overlap
13. FOSDEM 2018 GRASS GIS in the sky 13/20
High-performance computing
Spatial chunks
computational region
● North, South, West, East
● rows, columns
pre-defined regions,
one for each chunk (tile)
Alternative
● create X tiles from one raster map (r.tile)
1 2
3 4
14. FOSDEM 2018 GRASS GIS in the sky 14/20
High-performance computing
Chunks for parallel processing
Spatial processing
● each time step as one chunk
or
● spatial chunks
not recommended
→ spatial discontinuities
t
16. FOSDEM 2018 GRASS GIS in the sky 16/20
High-performance computing
temporary GRASS GIS session
GISRC: name of file with GRASS variables
GISRC contents:
GISDBASE: name
LOCATION_NAME: name
MAPSET: name
LOCATION_NAME: sub-directory of GISDBASE
MAPSET: sub-directory of LOCATION_NAME
export GISRC=/path/to/tmpgisrc
Ideally all on a SSD scratch disk
script 2
Arguments : first and last time step
1. create temporary GISRC, mapset
with unique names, using a pre-defined
mapset template
2. run script 1 in temporary mapset
3. copy results, one for each time step, from
temporary mapset to final mapset
4. delete temporary GISRC, mapset
rm -rf $GISRC /path/to/temp_mapset
directories
http://grass.osgeo.org/wiki
17. FOSDEM 2018 GRASS GIS in the sky 17/20
High-performance computing
job/queue manager
– Select / create a queue
– submit a job to a queue
– start a job when hardware
resources are available
– redirect stdout and stderr to
unique files
job with task(s), here task = script 2
job 1 : running
job 2 : running
job 3 : running
job 4 : running
job 5 : waiting
job 6 : waiting
job 7 : waiting
job 8 : waiting
waiting : no hardware resources
available
18. FOSDEM 2018 GRASS GIS in the sky 18/20
High-performance computing
Collect results
– Copy results to one common
GRASS mapset
→ this is the I/O bottleneck
try nice / ionice
cluster file systems
job 1
results
job 2 job 3 job 4
job 5
job 6
job 7
job 8job 9job 10job 11
job 12
job 13
job 14
19. FOSDEM 2018 GRASS GIS in the sky 19/20
High-performance computing
MODIS Land Surface Temperature
temporal + spatial processing
20. FOSDEM 2018 GRASS GIS in the sky 20/20
High-performance computing
Most important
Have a good admin
that fixes the system
after you broke it
Neteler M, Metz M, 2011 – 2018, pers. com.