22 - CSIRO - Water Data Management-Sep-17

1. Water data management platforms
Modern Tools & Techniques for Water Resources Assessments & Management
Amit Parashar
17 September 2014 New Delhi
LAND & WATER FLAGSHIP

2. Contents
• CSIRO
• Data management challenges
• Data sharing platforms
• Hydrological geofabric
• Cloud computing
• Demo

3. Top 1% of global research
institutions in 14 of 22 research
fields
Top 0.1% in 4 research fields
Darwin
Alice Springs
Geraldton
2 sites
Atherton
Townsville
2 sites
Rockhampton
Toowoomba
Gatton
Myall Vale
Narrabri
Mopra
Parkes
Griffith
Belmont
Geelong
Hobart
Sandy Bay
Wodonga
Newcastle
Armidale
2 sites
Perth
3 sites
Adelaide
2 sites Sydney 5 sites
Canberra 7 sites
Murchison
Cairns
Irymple
Melbourne 5 sites
CSIRO: Who we are
Werribee 2 sites
Brisbane
6 sites
Bribie
Island
People
Locations
Flagships
Budget
6000
58
9
$1B+

4. Challenges in data management
• Lots of investment in water data
collection (collecting, finding,
accessing and formatting of data)
• Multiple agencies collecting data –
different ways of managing data
• No single point of truth (overlaps
between agencies)
• Fragmented data silos
• Difficult to aggregate quality
assured data set as input to
research and inform policy

5. Challenges in data management
• Supporting information/data requests again and again and again
(usually for the same data) Need scaleable mechanism to share
data
• Water management pressures require increasing complex and
integrated assessments to support decision makers.
• As India moves towards IWRM & Basin Level Planning, water data
managers will increasingly need to support a variety of
jurisdictions, agriculture, urban, environment, energy
• Community pressure for access to data (increased transparency)

6. Discover Access Extract, Transform, Load Understand
Use
Time and effort
Research Focus
To enable more efficient and effective management of
water by improving the availability, accessibility and
usability of existing and new water information products
and services.
Action
Knowledge
Information
Data
Action
Knowledge
Information
Data

7. Spatial Information Service
Stack – Data Sharing Platforms
Discovery/Access/Integrate

8. Why?
Provide data uniformly and we use it diversely?
GIS Reports Research
Desktop simulations
and modelling
Cloud computing and many others...

9. The Spatial Information Services Stack
Present day

10. Integration – WaterML 2.0
• Framework exists, we need a
mechanism of sharing data between
each other
• Under Australia’s 2007 Water Act,
BoM collects observations of storage
level and stream flow from over 200
providers across the nation.
• CSIRO has led the development of
WaterML 2 standard specifically for
quantity and is currently extending it
for water quality as well.
• Time series data, allows near real
time model-data integration

11. Geofabric – towards a single
point of truth

12. Why?
• No consistent national scale water map of Australia
• Differences between States, regions, different resolutions
• Classic examples:
• 5 different catchment boundaries for the same catchment
• Stream network does not match the DEM
• Stream network where the gauges are not on the streams
• Makes it difficult to compare and do basin level and national scale
assessments

13. Hydrological Geofabric of Australia
• The Hydrological Geofabric
provides:
• A consistent spatial framework with
a historical gazetteer (location
names);
• A specialised GIS that registers
relationships between features from
the hydrological system (rivers,
lakes, reservoirs, dams, aquifers,
drains and monitoring points)
• It also stores the agreed boundaries
of basins, drainage divisions,
catchments, aquifer and priority
aquatic ecosystems.

14. User oriented & model driven products

15. Clouds – A watershed moment

16. Trends in technology

17. Cloud infrastructure

18. Software as a Service to support Research
• Provide modelling services through cloud computing
• Opportunistically applied to catchment modelling in Koshi
• eWater Source is the river system modelling software used

19. Source Modelling Service
Data
Centre
Compute
Node e.g.
local
machine
Source
Modelling
Service
Compute
Node e.g.
Azure
Compute
Node e.g.
Amazon
External Modeller
uploading data and running
models via the Web
CSIRO Scientist updating model
science and functionality
With the Source Modelling Service,
complex model runs and analysis can be
undertaken from anywhere in the world
and scaled to handle increasingly
complex problems through use of
commercial Cloud Providers

20. Remote Sensing Cloud
• Typically time consuming to find
data and to process
• Very large data sets
• Require a subset of data often,
(x,y,t)
• Earth Observations Data Cube
• Spatially regular
• Calibrated images (cloud cover etc)
• Long time series (Landsat)
• Openly accessible using cloud
technologies
Calibrated “Cubed”
Data in AG-DC

21. Bringing it all together
DEMO

22. Scenario – Water Sharing in the Ringarooma
Moorina
Gauge: 6
cumecs e-flow
Water Storage

23. Sensor Data Feeds

24. Managing flow

25. Sensor Cloud
Conceptual
Architecture
ACTION

KNOWLEDGE

INFORMATION

DATA
Sensor Cloud with real-time with eWater Source cloud
modelling platform
DATA
Sensor Networks
(real-time data)
Data Providers
(spatial, historical)


APIs & Web Services

Apps (DSS)

26. Sensor Cloud - benefits
• Lots of projects under development in India:
Hydrology projects – Sensor data, State Data, Central Data, Water Storage data,
Climate data, River flow data, Ground water levels, Water Quality data &
Historical data sets
• Don’t need expensive clusters in-house
• Easier to access and process parts of large data sets and much
easier to share data and model outputs
• Bring them all together in to sensor cloud to support community
engagement, government to citizen and government to
government engagement

27. Next steps?
• Open data policy (real time and archived); Standards to support
data exchange & standards based technology platforms (software
and hardware)
• Capacity building to support river basin planning
• Build a community of practice, time, effort and focus. These are not
easy things to implement. Focus efforts on an operational scenario.

28. Land & Water Flagship
Amit Parashar
t +91 8130443332
E amit.parashar@csiro.au
w www.csiro.au/
Thank you
LAND & WATER FLAGSHIP