Build an end-to-end Kafka-based pipeline in minutes! Enhanced Data pipelines with no code. An Apache Kafka tutorial with Lenses by Stefan Bocutiu.

1. Stefan Bocutiu
@stepib
London Apache Kafka Meetup, Feb 2018
End to end topologies on Apache Kafka
An IoT use case CONNECT ANALYSE REACT

2. About Today
Intros
IoT
High Level Architecture
Build the Pipeline
Q&A

3. Hi! It’s Stefan!
@stepib
https://github.com/stheppi
CONNECT ANALYSE REACT
Streaming Data Processing & Integration
KAFKA - KUBERNETES

4. EVOLVESCALEBUILDIoT initiative challenges
★ 94% face challenges collecting and analysing IoT data
★ 26+ Billion devices by 2020, Gartner
More Data, More Complexity
★ 2.500.000 trillion bytes produced daily
REAL TIME DATA INGESTION + IMMEDIATE QUERIES =
ACTIONABLE / TIMELY INSIGHTS
IoT Data Flows

5. IoT data flow
store datastream & process datasensor data

6. IoT data flow
store datastream & process datasensor data
E T L

7. InfluxDBsensor data
MQTT
InfluxDB
Connector
LENSES SQL
PROCESSOR
KAFKA
CONNECT
KAFKA
CONNECT
MQTT
Connector
KAFKA STREAMS

8. InfluxDBsensor data
MQTT
InfluxDB
Connector
LENSES SQL
PROCESSOR
KAFKA
CONNECT
KAFKA
CONNECT
MQTT
Connector
KAFKA STREAMS
Synthetic Data
With Moquette
1 Broker,
data generate
sensor data
InfluxDB Docker

9. ®
IoT &
High Volumes, N devices
& irregular intervals
Real Time Analytics
& Microservices
Unlimited streams of data, async transfers
Producers & Consumers decoupled operation
Kafka Streams API
Processing and analysing data in motion
Multiple sources of data
& long term storeage
Kafka Connect API,
move data with pluggable reusable & scalable connectors
An open source streaming framework
with messaging semantics where records are key-value pairs.

10. LENSES BOX
For Kafka Development
Single docker image with all services
Growing list of examples & synthetic data generators.
Stream Reactor 25+ connectors
Kafka CLI autocomplete.
Rest and WebSocket endpoints
Redux JS Client
Testing Libraries
FREE!
All you need to start working with Apache Kafka ecosystem

11. MQTT Source connector
• Message Queueing Telemetry Transport messaging protocol
• Optimised for connected devices and M2M protocol
• Low bandwidth, high-latency, unreliable networks
• Plain or SSL connections
• Kafka Connect Query Language
INSERT INTO kafka_topic
SELECT * FROM /mqtt_source_topic
[WITHCONVERTER=`myclass`]
MQTT
Distributed, Fault tolerant, scalable
No code required
CLI / UI / Monitor
KCQL Support
Multiple Distributions

12. • Process data in motion with Lenses SQL
• Create & Deploy Lenses SQL Processors
• Native Kafka & Kubernetes Support Scalability
• Visual Topology
• Monitor and manage processors via LENSES
Process & Analyse
Data in motion

13. InfluxDB Sink connector
• AVRO & JSON Support
• Error handling ( Noop, Retry, Throw)
• Field selection & field alias
• Multiple inserts supported
• Kafka Connect Query Language
• MEASUREMENTS, KCQL: INSERTS
• TAGS, KCQL: WITHTAG
• TIMESTAMPS, KCQL: WITHTIMESTAMP
• DURATION
Distributed, Fault tolerant, scalable
No code required
CLI / UI / Monitor
KCQL Support
Multiple Distributions
INSERT INTO sensorMeasure
SELECT * FROM sensor_data_avg
WITHTAG (id, poweredBy=Lenses)

14. Distributed, Fault tolerant, scalable
No code required
CLI / UI / Monitor
KCQL Support
Multiple Distributions
InfluxDB Sink connector
• Timestamp nanosecond
• KCQL
• No filter support
• No function support
• Throughput involves domain knowlege
• Kafka Connect rebalances

15. visual data flow
Visual and Interactive Topology
for a 360 view of your data
End-To-End Topology Management
Visualise your Kafka-based data flows
in real time

16. Wrap it up!
CONNECT
ANALYSE
REACT
Real Time, IoT pipelines in minutes!
MQTT

17. Contact us
www.landoop.com
info@landoop.com
Twitter @LandoopLtd
Github github.com/landoop
Gitter https://gitter.im/Landoop
Slack https://launchpass.com/landoop-community
Thank you!