BigDataOcean brings a digital revolution to the maritime industry by creating a large maritime big data infrastructure that enables collaborative, data-driven intelligence. BigDataOcean will allow analytics based on diverse data resources, coming from public and private providers. In this webinar, Spyros Mouzakitis and Giannis Tsapelas will present a demo of the BigDataOcean platform and discuss the challenges and lessons learned so far.
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
BDVe Webinar Series - Big Data Ocean - Rocking the boat with Big Data
1. Rocking the boat with big
data
www.bigdataocean.eu
Dr. Spiros Mouzakitis (NTUA)
Deputy Project Manager
GiannisTsapelas(NTUA)
Senior Researcher
2. Around 80% of global trade by volume is carried
by sEA
….and around 70% of global
trade by value
3. CHALLEN
GESo Undeveloped sharing of Blue Data between
enterprises and entities of the maritime domain
and other domains
o Lack of agreed standards and formats
o Huge potential from cross-sectorial blue data
applications – still unexploited
o Out-of-the-box Big Data solution that enables
advanced queries and analytics on cross-sector
data - missing
4. Develop a Maritime Big Data platform
that delivers out-of-the-box, value-added data and
analytic services for maritime applications
by exploiting cross-sector data streams
4
8. BigDataOcean end-user applications
Vessel Fault
Detection,
Predictive
Maintenance
and Fuel
Consumption
Reduction
Maritime
security and
anomaly
detection
Oil spill
modeling
Wave power
generation
8BDV PPP Technical Committee Meeting
9. Case 1 - Fault Prediction & Proactive Maintenance
and Fuel Consumption
9
10. Case 1 - Fault Prediction & Proactive Maintenance
and Fuel Consumption
10
• Damage and mechanical failures detection and predictive
maintenance of vessel equipment
• Investigation of the impact of the environmental conditions
and the operational decisions taken on the vessel's fuel
consumption
Goal
• Ship owners, Maritime EquipmentConstructors
Stakeholders
11. BigDataOcean Solution
Analytics and knowledge
base about fuel
consumption and repairs
Prediction models for
maintenance and fuel
consumption
Maritime Regulations, Flag
& Ship Classification
Provisions
Plant Maintenance
System for Main Engine
& spare parts
Vessel
routes
In-situ Observations
Cross-domain
Forecast Data
Port information
BigDataOcean
Solution
12.
13.
14.
15. Business Benefits
● Shipping companies and Maritime Equipment Constructors
Minimum repairs, maintenance cost and fuel consumption
Maximum vessels’ use and financial benefit
Minimize environment impact
Reliability and innovation
Advanced Data analytics related to maintenance and fuel
consumption
16. Case 2 – Mare protection – Oil Spill dispersion
forecast
16
17. Case 2 – Mare protection – Oil Spill dispersion
forecast
17
• Provide to the end users oil spill drift forecasting and
simulation services for the marine environment
• Enhance the efficiency in managing oil spill pollution risks.
Goal
• Emergency Response Companies, National Entities
(PublicAuthorities), NGOs, Marine Research Institutes,
Shipping companies and Oil drilling companies
Stakeholders
18. Natura /
Protecte
d areas
In-situ Observations
BigDataOcean Solution
Graphical output
Cross-domain
Forecast Data
POSEIDON
OSM
Oil spill scenario
submission
Location, Rate,
Nature & characteristics
Ocean Circulation
Forecast
Weather
Forecast
BigDataOcean
Solution
AIS
Data
BigDataOcean
Wave
Forecast
Oil Spill Dispersion Forecast
Acquisition
High Risk PollutionAreas
UnderwaterAccident
19.
20.
21.
22. Business Benefits
● Extended knowledge, models and enriched datasets
● New products addressed to environment protection
organizations and maritime authorities for rapid intervention
against oil spills in the sea
● Control and limit impact and damage on the coast and on
essential resources and structures.
● Efficiency in the protection of the marine environment and of
the marine life.
24. Case 3 – Wave Power as Clean Energy Source
24
• Evaluation of wave energy potential and contribution to
development of wave energy solutions.
Goal
• Offshore Renewables Service Providers, Offshore Pilot
Zone Concessionaires,WEC developers, Energy
Producers, Hydrographic Centres
Stakeholders
28. Business Benefits
● Wave resource characterization in your selected location or
area, based in historical data.
● Wave forecast.
● Assessment of Wave Energy Converters energy generation,
allowing to compare your device with others.
● Forecast of energy generation for your WEC device.
30. Case 4 – Security and anomaly path detection
• Identify vessel routes based on their motion patterns to act
proactively and minimise threats at sea.
Goal
• Port authorities, Ocean Observatories, Port/Cargo
Community systems,Transport and Logistics companies,
Harbour Pilots and Maritime Consultants
Stakeholders
34. Business Benefits
● Effectively handle the information volume from tracking
technologies and AIS data
● identify patterns of behavior and vessel risk profile
● proactively minimize the impact of possible threats
36. Timeplan
Analysis and Development
2017 6/2019
Start
MS4:
Prototype
MS2:
Requirements
and needs
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2
MS1:
Maritime data
value chain
definition
MS3:
Methodology
& Architecture
MS5: Final
Platform
MS6:
Business
Plan,
Lessons
Learnt
Evaluation
Market preparation
We are here
37. Lessons learned: exploitation
37
BDO Commercial Partnership for the platform
1. BDO Platform Services
2. BDO End-UsersApplication Services
3. BDO On-Demand Data Services and custom solutions
4. BDO Non-Technical Package
• Training
• Consulting
• Data science coaching
• Contract deals with maritime companies
• Focus on B2B applications
38. Semantic Challenges
● Plethora of file formats and metadata standards coming from diverse data sources
● Ability to perform advanced queries that combine those big datasets
● Enable ad hoc queries
Our Approach
● Automatic ingestion system to a harmonized, semantically aware schema (BDO
canonical Model)
● Based on DCAT, NETCDF convention standards (cfconventions.org)
39. Performance Challenges
● Performance issues
POSTGRESQL and relational databases typically used in legacy maritime applications –
Serious scalability issues for the size of the datasets
NoSQL databases -> Good performance for simple queries, but bad performance for
queries that combine data
Distributed, wide column stores (e.g. Cassandra) -> better optimised for known queries
Our Approach
● Presto with Apache Hive
40. Lessons learned: current solution
Solution for Query Designer
queries
Solution for further query
optimization
● Caching JOIN queries performed by
users
● Pre-join datasets for pilot
applications
● Adoption of Apache Parquet storage
and ORF file formats
HDFS
Query performance over million of rows: from 5 minutes to 5 seconds!
Using the capabilities of the big data ocean platform we have created 4 applications that deal with crucial challenges for the maritime industry – these applications are our flagship products to kickstart promote the platform
Its main goal is to support the first and most important step of the wave energy convertors. Where is the best place to place the WEC and and it’s the potential of the installation depending on the location at sea. Its stakeholders are….