Flink Forward 2019
StreamPipes is an open source self-service IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams
https://streampipes.apache.org/
Flink for Everyone: Self-Service Data Analytics with StreamPipes
1. Flink for Everyone: Self-Service Data
Analytics with StreamPipes
Patrick Wiener, Philipp Zehnder
Flink Forward Europe 2019, Berlin, 2019-10-08
2. www.streampipes.org | @streampipes | github.com/streampipes
2
"A self-service IoT toolbox to enable non-technical users
to connect, analyze and explore IoT data streams"
What's StreamPipes?
3. www.streampipes.org | @streampipes | github.com/streampipes
3
What's StreamPipes?
Big Data / Edge
InfrastructureExecute
Reusable
algorithm toolbox
Install
Model pipelines
4. www.streampipes.org | @streampipes | github.com/streampipes
About us
4
Dominik Riemer
Senior Research Scientist
Philipp Zehnder
Research Scientist
Patrick Wiener
Research Scientist
FZI Research Center for Information Technology, Karlsruhe, Germany
Stream Processing, Data Management, Machine Learning
Non-profit research center for applied ICT research (250 employees)
Started StreamPipes in 2014, first OSS release 2018
5. www.streampipes.org | @streampipes | github.com/streampipes
Agenda
The need for self-service IoT data analytics1
StreamPipes: Technical Overview
Demo
2
Lessons Learned w/ Flink & Getting Started3
7. www.streampipes.org | @streampipes | github.com/streampipes
Conveyor Belts
Pressure
Oil temperature
Dust particles
Production plans
Environmental Data
Gear box drive
Energy consumption
Telematics
Industrial Internet of Things
Data streams everywhere
8. Continuous Monitoring Situational Awareness
Continuous Data
Harmonization
Flexible data integration
from heterogeneous
sources and monitoring
of current system states
Detect time-critical
situations, e.g., by
means of rules or ML
approaches
Continuous pre-
processing and
transformation of input
streams for third party
systems
Industrial Internet of Things
Typical application scenarios
9. www.streampipes.org | @streampipes | github.com/streampipes
StreamPipes
Open Source framework to easily manage IoT data
Data Access
Data analytics &
harmonization
Data exploration &
exploitation
Generic adapters
Specific adapters
Metadata
Data streams & sets
Pre-processing
Filter/Aggregation
Pattern Detection
ML
Situation detection
Harmonized data sets
Visualizations
Third-party systems
9
29. Demo
Condition monitoring + StreamPipes
Rule-based monitoring of flow rate measurements in a multi tank system
30. Demo
Condition monitoring + StreamPipes
Rule-based monitoring of flow rate measurements in a multi tank system
Flow
Sensor
Aggregate
data
Detect
Leakage
Notify
MQTT
IoTDB
StreamPipes Connect
Calculate
Statistics
32. www.streampipes.org | @streampipes | github.com/streampipes
Potentially huge stream of sensor data needs scalability
Remote Environment eased the implementation of Flink Wrapper
Clean & intuitive Flink API enables fast processor development
Simple setup for development (mini cluster) and deployment
Easy to configure & monitor
Good integration with Apache Kafka
Flink + StreamPipes
Lessons learned
33. www.streampipes.org | @streampipes | github.com/streampipes
How to start
Setting up StreamPipes
Docker-based installation
streampipes.org/en/download
Download installer from Github1
./streampipes start2
Finish installation in browser3
33
34. www.streampipes.org | @streampipes | github.com/streampipes
34
What's next?
Data Access
Data analytics &
harmonization
Data exploration &
exploitation
Metadata recognition
PLC4X
Flink fault tolerance
Python wrapper
AutoML
Historical data
explorer
New features: Current work-in-progress
Infrastructure (Edge / Fog)
35. Let's connect!
…and if you like StreamPipes, star us on Github
streampipes.org
docs.streampipes.org
github.com/streampipes/streampipes
twitter.com/streampipes
feedback@streampipes.org