This document discusses streaming ETL using Apache Kafka and MongoDB as a modern data platform. It provides an overview of streaming data and how it can help with speed and agility compared to traditional batch ETL processes. It then discusses how Apache Kafka acts as a streaming platform and messaging system that can be used to build streaming data applications and integrate data from various sources using Kafka Connect. The document announces the availability of the MongoDB connector for Kafka Connect, which allows streaming data between Kafka and MongoDB collections. It concludes with a demo scenario showing how this could work in practice.

1. Streaming ETL on the Shoulders of Giants
Scott L’Hommedieu, MongoDB Hans-Peter Grahsl, NETCONOMY
llamadew hpgrahsl

2. Speed &
Agility
For businesses to stay relevant
they must deliver value at a
breakneck pace and be
constantly seeking new sources
of value.
A Top 5 Tech Risk

3. Managing, Process and Analyzing Data
We all use Data
To unlock insights
And drive value

4. But, historic ETL is so painful
Antipattern for Speed and Agility
§ Batch Data
§ ETL
§ Slow
§ Error Prone
§ Brittle

5. Solving the Pain of ETL through Streaming Data
Speed and Agility
§ Data as a Stream
§ ETL – to - Fabric
§ Loosely Coupled Data Services

6. Streams Processors
Connected Apps
Architecture of a Modern Data Platform?
Streaming Data Fabric
Data Sources
Connected Apps
Data Sources

7. On the shoulders of Giants
KafkaMongoDB

8. Modern Data Platform

9. Modern Data Platform
Doc Model
Run Anywhere
Distributed and Scalable
Resilient and Performant

10. Apache Kafka 101

11. Streaming Platform

12. Streaming Platform
• distributed
• horizontally scalable
• highly fault-tolerant

13. What is Streaming?
“a type of data processing that
is designed with infinite data
sets in mind”
–Tyler Akidau

14. "…everything that happens in a
company – every customer interaction, every
API request, every database change – can be
represented as real-time stream that
anything else can tap into, process or react to."

15. "…Kafka and the whole category of stream
processing represents a fundamental
paradigm shift in how the digital part of a
company is built, how data is used, and how
applications are built. This is actually a
pretty rare thing…"
– Jay Kreps

16. KStreams
App
Data
Sources
Data
Sinks
KSQL
App
Streams API
KSQL
Consumer API
Connect API
App
Apps
App
Apps
Connect API
Producer API

17. Kafka APIs in a Nutshell…
§ Producer & Consumer API
à publish-subscribe scenarios
§ Connect API
à streaming data integration scenarios
§ Streams API & KSQL
à code or SQL-based streaming scenarios

18. A bit more about Kafka Connect …

19. Kafka Connect Basics
ANY
Sink
Connect Connect
ANY
Source
ANY à e.g. file systems, data stores, REST endpoints, …

20. Kafka Connect Basics
often about data stores
Connect ConnectSOURCE SINK

21. Kafka Connect Basics
or more concretely
Connect Connect
https://hub.confluent.io à many many more

22. Kafka Connect Basics
or more concretely
Connect Connect
https://hub.confluent.io à many many more

23. How do connectors operate?

24. Kafka Source Connectors
Source
Connector
Converter
Serialize
S
M
T
1 … N
Single Message Transforms
for basic in-flight manipulations
…
S
M
T

25. Kafka Sink Connectors
Converter
Deserialize
Sink
Connector
S
M
T
1 … N
Single Message Transforms
for basic in-flight manipulations
…
S
M
T

26. Announcing …

27. Available on the Confluent Hub: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb
MongoDB Connector for Apache Kafka
Supported by MongoDB Verified Gold by

28. MongoDB Connector for Apache Kafka
Available on the Confluent Hub: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb

29. Demo Scenario - Action!

30. Producer APIdata
generation
Stream
Processor
KSQL
MongoDB
Sink Connector
MongoDB
Source Connector
data serving
REST
Change
Streams
device
management
SSE
Demo Scenario
!

31. That’s all folks!
THANK YOU
J Ready for Q & A J