A fast and efficient integration of end device data into data processing systems is becoming increasingly important in the Internet of Things. Factors such as secure and reliable data transmission, real-time data processing and the analysis of huge amounts of data afterwards play a major role. In order to implement an architecture that enables smooth communication between end devices and systems, tools are needed that are designed for a type of communication that takes these factors into account. This presentation shows the strengths and application areas of MQTT, the de-facto standard communication protocol for the Internet of Things, and Apache Kafka, which is often used for data streaming in the Internet of Things, and explains how and why the technologies complement each other ideally.

1. IoT Data
Streaming
Why MQTT and Kafka are a match made in
heaven
Margaretha Erber & Dominik Obermaier

2. Insert
Picture
Copyright © by HiveMQ. All Rights Reserved.
Insert
Picture
Dominik
Obermaier
• Strong background in
distributed and large scale
systems architecture
• OASIS MQTT TC Member
• Author of „The Technical
Foundations of IoT“
• Conference Speaker and
Author
• Program committee member
for German and international
IoT conferences
Hive
MQ CTO & Co-Founder
Magi Erber
WELCOME
Product Manager
@HiveMQ
@dobermai
linkedin.com/in/doberm
ai/
@ErberMagi
linkedin.com/in/margaretha-
erber/
• Conference Speaker
• Author
• Expert for cloud native
technologies and Apache
Kafka

3. Copyright © by HiveMQ. All Rights Reserved.
People on the Internet
The Internet of Things is huge

4. Copyright © by HiveMQ. All Rights Reserved.

5. Technical IoT
Challenges
Confidential and Proprietary. Copyright © by HiveMQ. All Rights
Reserved.

6. Copyright © by HiveMQ. All Rights Reserved.
• Enterprise IT infrastructure is not
suitable for IoT
Copyright © by HiveMQ. All Rights Reserved.
• Massive scalability required for
millions of devices
Challenge 1 - Scalability

7. Copyright © by HiveMQ. All Rights Reserved.
• End customers are used to instant user
experiences like instant messaging with
WhatsApp
Copyright © by HiveMQ. All Rights Reserved.
• Critical systems need reliable and
instant data transfer like manufacturing
systems
• Bidirectional communication required
Challenge 2 - Instant Data Delivery required

8. Copyright © by HiveMQ. All Rights Reserved.
• Customer experience for IoT apps and
devices must be excellent even when
internet connectivity is flaky
Especially for moving “devices” like cars
Copyright © by HiveMQ. All Rights Reserved.
• Devices and apps must be easy to program and
maintain, complexity should be in the cloud not
on the device
Cloud is easier to update than physical devices
Challenge 3 - Unreliable Networks

9. Copyright © by HiveMQ. All Rights Reserved.
Technologies built for
the Internet of Humans
are not suitable for the
Internet of Things
9

10. Copyright © by HiveMQ. All Rights Reserved.

11. Copyright © by HiveMQ. All Rights Reserved.
• (I)IoT Messaging Protocol
• Created for extreme scale and instant data
exchange
• Publish/Subscribe based architecture
• Easy on the device side, pushes all
implementation complexity to the server
• Built for machines and constrained devices
(binary, data agnostic)
• Designed for reliable communication over
unreliable channels
What Is MQTT?

12. Copyright © by HiveMQ. All Rights Reserved.
Publish / Subscribe Pattern

13. Copyright © by HiveMQ. All Rights Reserved.
Use Cases
13
Connected Car
IIoT /
Industry 4.0
Logistics Telecommunicatio
n
IoT Messaging
Middleware

14. Push Communication
MQTT
Use
Cases
Reliable Communication over unreliable
networks
Constrained Devices
Low Bandwidth and High Latency
Industrial Message Bus

15. Copyright © by HiveMQ. All Rights Reserved.

16. Copyright © by HiveMQ. All Rights Reserved.

17. Copyright © by HiveMQ. All Rights Reserved.

18. Copyright © by HiveMQ. All Rights Reserved.
• Optimized to stream data between systems and
applications in a scalable manner
• Scale-out with multiple partitions for a topic and
multiple nodes
• Perfect for inter-system communication
• inside trusted network
• with stable IP addresses and connections
and
• limited number of producers and
consumers
Apache Kafka Strengths

19. Apache Kafka
for IoT
How does it fit in?

20. Copyright © by HiveMQ. All Rights Reserved.
Kafka brokers need to be addressed directly by the clients
Kafka does not support large amounts of topics
Kafka clients are reasonably complex and resource intensive compared to client
libraries for IoT protocols
Kafka clients require a stable TCP connection for best results
It’s unusual (and very often not even possible at all) to have tens of thousands or
even millions of clients connected to a single Kafka cluster
Kafka is missing some key IoT features
Challenges for Apache Kafka in IoT

21. How to use the best of
both worlds?

22. Copyright © by HiveMQ. All Rights Reserved.
💛
A Love Story Made in Heaven

23. Copyright © by HiveMQ. All Rights Reserved.
Fast, reliable
communication
Ideal for communication
over unreliable networks
Connects millions of
IoT Devices
Optimized to stream data
between systems and
applications
Communication inside
trusted networks
Connects limited number of
communication partners
MQTT and Kafka

24. Copyright © by HiveMQ. All Rights Reserved.
How to combine both worlds?
Kafka Connect MQTT Proxy Custom Bridge MQTT Broker
Extension

25. Copyright © by HiveMQ. All Rights Reserved.
Kafka Connect
sensor-data
alert
Kafka Connect
MQTT Broker
MQTT
devices
Mobile Application

26. Copyright © by HiveMQ. All Rights Reserved.
- Doesn’t scale well as Kafka Connect
has to act as MQTT Client
- MQTT Clients are not designed for
extremely large amounts of MQTT
messages
- Centralizes business and message
transformation logic
Kafka Connect
+ Ideal if you don’t control the
MQTT Broker or use a third party
MQTT Broker

27. Copyright © by HiveMQ. All Rights Reserved.
MQTT Proxy
MQTT
devices
MQTT Proxy
Broker
Kafka Consumer

28. Copyright © by HiveMQ. All Rights Reserved.
- Does not implement full MQTT ISO Standard
- Does not support features unique to MQTT
which were designed for IoT use cases
(LWT, retained messages,...)
- Client must ensure that no MQTT features
beside plain publishing are used
- Tightly coupled with Kafka, does not allow
the Pub/ Sub features of MQTT
MQTT Proxy
+ Does not require an MQTT broker
+ Usually stateless, which makes scaling
easier

29. Copyright © by HiveMQ. All Rights Reserved.
MQTT Custom Bridge
Mobile Application
MQTT
devices
Custom Bridge
MQTT Broker
Broker

30. Copyright © by HiveMQ. All Rights Reserved.
- Extremely hard to avoid message loss
- Developers must take care of fault tolerance
themselves
- Another system which adds complexity and
adds no value
MQTT Custom Bridge
+ Application used for transposing MQTT
to Kafka and vice versa built by
developers themselves, so no external
component

31. Copyright © by HiveMQ. All Rights Reserved.
Mobile
Applicatio
n
sensor-
data
Stream processing
using KSQL
Other
Backend
Systems
MQTT to Kafka
Kafka to MQTT
MQTT
devices
Confluent Schema Registry
Get Schema
alert
MQTT Broker Extension

32. Copyright © by HiveMQ. All Rights Reserved.
- Needs an MQTT broker that can be
extended
- Out of the Box MQTT Broker Extension to
connect with Kafka only available for
HiveMQ
MQTT Broker Extension
+ MQTT Broker uses native Kafka protocol
+ All MQTT Features can be used
+ Bi-directional producing and consumption
possible
+ High Scalability and resilience.
+ Extreme throughput. Can write hundreds of
thousands of MQTT messages per second
to Kafka
+ Ideal for aggregating MQTT topics to Kafka
Firehose Topics
+ Can write to multiple Kafka Deployments at
the same time

33. Copyright © by HiveMQ. All Rights Reserved.
What are the next steps?
+

34. THANK YOU