In this session, we are going to create a Lenses DataOps hub for IoT data with Apache Kafka and InfluxDB flows over Kubernetes. We will demonstrate how to create streaming flows and securely explore and monitor real-time data. We will use Kubernetes to spin up scalable flows and go through how we can simply provision such flows with secret management and monitoring end to end out capabilities.

1. DataOps
on streaming data
Kafka to InfluxDb via Kubernetes
Native Flows - IoT demo
Chris Kotsis
lenses.io

2. #About me
Reliability Engineer at Lenses.io
Data Flows on Kubernetes
Kafka
Cloud Native techs
DataOps Expert

3. #About lenses.io
Enterprise DataOps over Streaming Data

4. #About lenses.io
Enterprise DataOps over Streaming Data

5. #About lenses.io
Enterprise DataOps over Streaming Data
all Distributions - on Cloud - on Premises

6. #DataOps
Embrace Collaboration
Eliminate Friction,
turn data to Value Faster

7. #DataOps Personas
people who
● Collect and Prepare the data
● Analyse the data
● Use the findings for business value

8. #DataOps IoT Challenges
Enormous Volumes of Data
“500 billion connected devices by 2025”

9. #DataOps IoT Challenges
Enormous Volumes of Data
“500 billion connected devices by 2025”
Its Management and Analysis will become
harder and continue to break traditional tools

10. The IoT Data Flow

11. How to implement a Streaming
Architecture for IoT Time-Series
data and Real-Time insights?

12. Infrastructure Layer
Where the Data lives
Self-service Data Access,
Multi-tenancy,
Security, Governance to
Accessibility & Visibility
for ALL

14. IoT & kafka
High Volumes, N devices &
irregular intervals
Real Time Analytics &
Microservices
Multiple sources of data &
long term storage
An open source streaming framework
with messaging semantics where
records are key-value pairs
➔ Unlimited streams of data,
➔ Producers & Consumers
(pub/sub)
➔ Processing and analysing data
in motion
➔ Connect API to move data with
pluggable reusable components

15. The flow implementation
➔ Sensor produces data to Kafka
➔ Create my Data Flows:
◆ Kafka - to - Kafka
processing
◆ Route to InfluxDB

16. Application Layer
Multiple Distributed
Microservices for
complex Data Flows

17. Application Layer
Provisioning,
Monitoring, Alerting,
Security,
Governance,
Accessibility,
Self-service
deployments,....

18. How Lenses integrates
with the Application Layer?

19. Lenses SQL Processors
● Simply Filter, Enrich, Split & Bind your data
● Manipulate Live Streams of Data
● Scalable: Cloud / Kubernetes Native
PROCESS

20. Connect Kafka to InfluxDB
Real Time Ingestion
Distributed
Fault tolerance
Scalability
Error Handling
Monitoring & Alerting
Governance & Security
Easy Data Manipulation
...
INGEST

21. Lenses Connectors: Binding external
sources and sinks is a few clicks away

22. InfluxDB sink connector by Lenses.io
Kafka to InfluxDB
No code required
CLI/UI/API/Monitor
SQL Support
Multiple inserts supported
AVRO & JSON Support
Supported Features:
MEASUREMENTS (SQL: INSERT TO)
TIMESTAMPS (SQL: WITHTIMESTAMP)
TAGS (SQL: WITHTAG)
DURATION

24. The uncontrolled “Chaos”
of modern data streams

25. Lenses Topology, one view to bind them all

28. Demo

29. What it takes to demo this architecture

30. Lenses Box
● FREE for Developers
● Single Broker setup
● All ecosystem services
● 25+ Connectors
● Synthetic Data generators
● Live examples
● Lenses intuitive UI
● The powerful Lenses SQL
● Works on your Laptop
● Works on Cloudhttps://lenses.io/lenses-box

31. Thank you and <3

32. https://lenses.io
Follow us @lensesio