Organizations today are looking to stream IoT data to Apache Kafka. However, connecting tens of thousands or even millions of devices over unreliable networks can create some architecture challenges. In this session, we will identify and demo some best practices for implementing a large scale IoT system that can stream MQTT messages to Apache Kafka. We use HiveMQ as open source MQTT broker to ingest data from IoT devices, ingest the data in real time into an Apache Kafka cluster for preprocessing (using Kafka Streams / KSQL), and model training + inference (using TensorFlow 2.0 and its TensorFlow I/O Kafka plugin). We leverage additional enterprise components from HiveMQ and Confluent to allow easy operations, scalability and monitoring.

1. 1
Best Practices for Streaming IoT Data
with MQTT and Apache Kafka
Kai Waehner
Technology Evangelist
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Real Time Data Processing and Analytics
with 100000 Connected Cars
Dominik Obermaier
CTO HiveMQ
dominik.obermaier@hivemq.com
www.linkedin.com/in/dobermai
@dobermai
www.hivemq.com

2. 2
Agenda
• Use Case
• Architecture
• Live Demo
• Best Practices
• Next steps

3. 3
Agenda
• Use Case
• Architecture
• Live Demo
• Best Practices
• Next steps

4. 4
Global Automotive Company
Builds Connected Car Infrastructure
Use Cases:
• Connected Car Infrastructure (Cars, Traffic Lights, Cloud Services,
etc.)
• Real Time Analytics (Predictive Maintenance, etc.)
• Continuous Services / Sales
• Partner Integration (Car workshop, gas station, food market, etc.)
• …

5. 55
Other Components
Kafka Ecosystem
(3) Read Data
TensorFlow I/O
(5) Deploy
Model
(2) Preprocess
Data
(8a)
Alert Car
Real Time
Kafka App
TensorFlow
Serving
gRPC
Car Sensor
HiveMQ
MQTT Broker
MQTT
Connector
Kafka Connect
or
Confluent Proxy
or
HiveMQ Plugin
Kafka
Cluster
Kafka
Connect
Grafana
Elastic
Search
KSQL
Tensor
Flow
(04)
Train Model
Real Time
Kafka Streams
Application
(Java / Scala)
Tensor Flow
Real Time
Edge
Computing
(C / librdkafka)
Tensor Flow
Lite
(1)
Ingest Data
(8b)
Alert Driver
(e.g. mobile App)
(6a) Consume
Car Data
(6b)
All Data
(7) Potential
Defect

6. 6
Agenda
• Use Case
• Architecture
• Live Demo
• Best Practices
• Next steps

7. 7
Cloud Native
Infrastructure
Benefits
• Scalable
• Flexible
• Agile
• Elastic
• Automated
• Etc.

8. 8
• IoT-specific features for bad network / connectivity
• Widely used (mostly IoT, but also web and mobile
apps via MQTT over WebSocket)
• Built on top of TCP/IP for constrained devices and
unreliable networks
• Many (open source) broker implementations
• Many (open source) client libraries
MQTT - Publish / subscribe messaging protocol

9. 9
MQTT Trade-Offs
Pros
• Lightweight
• All programming languages supported
• Built for poor connectivity / high latency scenarios (e.g. mobile networks!)
• High scalability and availability *
• ISO Standard
• Most popular IoT protocol
Cons
• Only pub/sub, not stream processing
• Asynchronous processing (clients can be offline for long time)
• No reprocessing of events
* Depending on Broker

10. 10
A Streaming Platform is the Underpinning of an Event-driven Architecture
Ubiquitous
connectivity
Globally scalable platform for all
event producers and
Immediate data
access
Data accessible to all
consumers in real time
Single system of
record
Persistent storage to enable
reprocessing of past events
Continuous queries
Stream processing capabilities
for in-line data transformation
Microservices
DBs
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservice
s events
SaaS
data
Customer
experience
s
Streams of real time events
Stream processing apps

11. 11
Kafka Trade-Offs (from IoT perspective)
Pros
• Stream processing, not just pub/sub
• High throughput
• Large scale
• High availability
• Long term storage and buffering
• Reprocessing of events
• Good integration to rest of the enterprise
Cons
• Not built for tens of thousands connections
• Requires stable network and good infrastructure
• No IoT-specific features like keep alive, last will or testament

12. 12
(De facto) Standards for Processing IoT Data
A Match Made In Heaven
+ =

13. 13
Agenda
• Use Case
• Architecture
• Live Demo
• Best Practices
• Next steps

14. 14
Advanced Demo

15. 15
MVP

16. 16
Live Demo
End-to-End Integration and Data Processing for
100000 Connected Cars

17. 17
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
or
http://bit.ly/kafka-mqtt-ml-demo
=> Try it out in 30 minutes!
Demo 100.000 Connected Cars
(Kafka + MQTT + TensorFlow)

18. 18
Agenda
• Use Case
• Architecture
• Live Demo
• Best Practices
• Next steps

19. 19
Typical Journey
Value
Maturity (Investment &
time)
2
Enterprise
Streaming Pilot /
Early Production
Pub + Sub Store Process
5
Central
Nervous
System
1
Developer
Interest
Pre-Streaming
4
Global
Streaming
3
SLA
Ready,
Integrated
Streaming
Projects
Platform

20. 20
Start Small, but prepare for Scalability from Beginning
1. Use cloud native and scalable
components
• Confluent Platform is cloud native and built for scale
• HiveMQ is cloud native and built for scale
2. Don’t deep dive too much in the
beginning – but understand options
• HiveMQ Kafka Extension?
• Confluent MQTT connectors?
• Customer Integration?
3. Plan for Enterprise-readiness
• Security
• Monitoring
• Operations tooling
• Bi-directional communication

21. 21
Choose the right tool stack and infrastructure
Understand Trade-Offs and choose the right options for
deployments
• Edge
• On Premise
• Cloud
Use the best tools for the job
• Confluent Platform for Event Streaming
• HiveMQ for MQTT messaging and connectivity

22. 22
Separation of concerns
1. Devices
2. Gateway
3. Integration
4. Data Streaming
5. Consumer Apps
Decouple tasks
• Source integration
• Data processing
• Business logic
• Sink integration
• Analytics
• …

23. 23
Different data for different use cases
• Database, Data Lake
• Search
• Real time, Near Real Time, Batch
• Streaming, Request-Response
• CQRS, Event Sourcing
• Machine Learning
There is no single
MASTER DATA
EVENT…

24. 24
Agenda
• Use Case
• Architecture
• Live Demo
• Best Practices
• Next steps

25. 25
The HiveMQ Platform

26. 26
The HiveMQ Platform – Open Source and Enterprise-grade

27. 27
Confluent Platform
Operations and Security
Development & Stream Processing
Support,Services,Training&
Partners
Apache Kafka
Security plugins | Role-Based Access Control
Control Center | Replicator | Auto Data Balancer | Operator
Connectors
Clients | REST Proxy
MQTT Proxy | Schema
Registry
KSQL
Connect Continuous Commit Log Streams
Complete Event
Streaming Platform
Mission-critical
Reliability
Freedom of
Choice
Datacenter Public Cloud Confluent Cloud
Self-Managed Software Fully-Managed Service

28. 28
Confluent Cloud
Cloud-Native Confluent Platform Fully-Managed Service
Available on the leading public clouds with mission-critical SLAs and consumption-based pricing.
Serverless Kafka characteristics:
Pay-as-you-go, elastic auto-scaling, abstracting infrastructure (topics not brokers)
Spend your time on your applications!

29. 29
Next steps…
Try out the demo in 30 minutes:
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-
inference
http://bit.ly/kafka-mqtt-ml-demo
Check out the documentation and blog posts
• HiveMQ and Apache Kafka - Streaming IoT Data and MQTT Messages:
https://www.hivemq.com/blog/streaming-iot-data-and-mqtt-messages-to-apache-kafka/
• Internet of Things (IoT) and Event Streaming at Scale with Apache Kafka and MQTT:
https://www.confluent.io/blog/iot-with-kafka-connect-mqtt-and-rest-proxy
Contact us for questions or any other feedback:
• Website, Email, Slack, Phone, …
• Dominik: dominik@hivemq.com , Kai: kai.waehner@confluent.io

30. 30
Questions? Feedback?
Kai Waehner
Technology Evangelist
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Please contact us!
Dominik Obermaier
CTO HiveMQ
dominik.obermaier@hivemq.com
www.linkedin.com/in/dobermai
www.hivemq.com
www.twitter.com/dobermai