This document discusses Apache Kafka and deploying it in cloud environments. It provides an overview of Kafka's history and how it has evolved from monolithic applications to microservices and event-driven architectures. Various cloud options for deploying Kafka are presented, including fully managed services from Confluent and AWS, as well as using Kubernetes operators. The advantages and disadvantages of different approaches are compared. Best practices for operating Kafka in production environments are also covered.

1. © Materna 2019 www.materna.com
We are digitizing your world!
Apache Kafka
Fast Data in the Cloud
Frank Pientka

2. © Materna 2019 www.materna.de
From App over service to events
90s
15s
05s
10s
20s
00s
EXE/BIN JAR
JVM
WAR/EAR
Web/Appserver
Container
VMware
Pod
Helm Chart
Opertor
Multi-Cloud
JAR
VMware
MicroservicePackages Components Functions EventsModules

3. © Materna 2019 www.materna.de
Containers
Cloud
Native
A B r i e f H i s t o r y o f t h e C l o u d
F r o m V i r t u a l i z a t i o n t o C l o u d N a t i v e
Open
Source
IaaS
PaaS
Open
Source
PaaS
Virtualiza-
tion
2000 2001 2006 2009 2010 2011
Non-
Virtualized
Hardware
2013 2015
IaaS
3

4. © Materna 2019 www.materna.de 4
5 Years of K8s
Cloud Native Computing Foundation (CNCF)

5. © Materna 2019 www.materna.de
Big data - fast data
6

6. © Materna 2019 www.materna.de
Which use cases for real-time or stream processing 2017 vs. 2019?
AI/ML, IoT and ETL adoption rose the most over
7
STREAMING DATA AND THE FUTURE TECH STACK
Lightbend, 2019

7. © Materna 2019 www.materna.de
What’s hard about Fast Data?
8
Lightbend
Enterprise development trends report 2018

8. © Materna 2019 www.materna.com 9
How do you deploy Kafka?
2018 Apache Kafka Report

9. © Materna 2019 www.materna.com
Cloud options: self vs managed
10

10. © Materna 2019 www.materna.com
Confluent Cloudsupport plans
11

11. © Materna 2019 www.materna.com
Confluent Cloud Consumption-Based price
12

12. © Materna 2019 www.materna.com
Confluent Cloud on Google Cloud Platform (GCP)
13

13. © Materna 2019 www.materna.com
Connection local Kafka cluster with Confluent Cloud by Replicator
14

14. © Materna 2019 www.materna.com
Disaster Recovery in Multi-Datacenter Apache Kafka Deployments
15

15. © Materna 2019 www.materna.com 16

16. © Materna 2019 www.materna.com
Amazon Managed Streaming for Kafka (MSK)
17
Montl. Gesamtpreis =
468,72 USD (Broker-Instance-Gebühr kafka.m5.large) +
151,61 USD (Speichergebühr) = 620,33 USD

17. © Materna 2019 www.materna.com 18

18. © Materna 2019 www.materna.com
Bitnami Kubernetes Production Runtime Multi-Tier Architecture
a collection of services to run production workloads in Kubernetes
19
available for
Google Kubernetes Engine (GKE)
Azure Kubernetes Service (AKS)
Amazon Elastic Container Service for Kubernetes (Amazon EKS)

19. © Materna 2019 www.materna.com
Bitnami Kubernetes Production Runtime is a collection of services
20
Applications
• Kafka
• Debezium
BKPR
• Monitoring
• Logging
• Ingress
Kubernetes

20. © Materna 2019 www.materna.com
Pro & cons cloud offerings – Confluent & AWS
21
Offering Pro Cons
Confluent From Inventor on AWS
Professional support, components
Smal delay with Kafka Versions
AWS MSK Fast start, scale
Only Kafka & Zookeeper
good integration CLI, CloudWath, CloudTrail
Easy calculation broker-instance-EC2 classes/h
(xlarge – vCPU, RAM)+storage/month
EKS/ECS for other kafka
component
Limited configuration options
Broker, Storage limits
More versions delay
No Cluster-Versions-Upgrades
AWS Fargate Leading Cloud provider
Good integration
vCPU, RAM price, datatransfer
Manage kafka components
Integration with other AWS
services
AWS EKS Control pane, good integration Worker nodes, data pane
management, extra costs
Bitnami privat
K8s
Recent K8s, Kafka
Full configuration options
Lot‘s of own work
more
less
Opinionated

21. © Materna 2019 www.materna.com
Normal Kubernetes Cluster Deployment
22
 Write some Deployment, Services, Configmaps etc.
 Deploy to K8s
 Create a Helm Chart
 YAML YAML YAML – kubectl kubectl kubectl etc.

22. © Materna 2019 www.materna.com
Overview of the Cluster Operator component
23
Further operators
Topic, User Operator

23. © Materna 2019 www.materna.com
Apache Kafka running on Kubernetes ôr OpenShift
24

24. © Materna 2019 www.materna.com
Kubernetes operator pattern
25
Kubernetes resources, e.g.:
•Pod
•StatefulSet
•Deployment
•NetworkPolicy
•Service
•ConfigMap

25. © Materna 2019 www.materna.com
Kubernetes Cluster tasks full lifecycle
26

26. © Materna 2019 www.materna.com 27

27. © Materna 2019 www.materna.com
Rebalance Kafka Cluster downsize
28

28. © Materna 2019 www.materna.com
Strimzi operators overview
29

29. © Materna 2019 www.materna.com
Helm Charts: Where to find more?
30

30. © Materna 2019 www.materna.com
Where to find more operators?
31

31. © Materna 2019 www.materna.com 32

32. © Materna 2019 www.materna.com
HandelXYZXYZ
Lessons Learned
Start small, grow fast – fail fast, learn fast, automate
Best-of-both (BKPR/Kubernetes Charts on AWS)
Lenses against Kafka blindness
Shift of complexity – from inside out
New and challenging topic – very exiting!!!
Don’t trust your settings – monitoring, health checks, logging are mandatory
Don’t build your own – (re-) use products, best-practices (Helm, Operators, …)

33. © Materna 2019 www.materna.com© Materna 2019 www.materna.de
Kontakt
Materna SE
Frank Pientka
Tel. +49 1570 1128854
E-Mail: frank.pientka@materna.de
www.materna.de
34

34. © Materna 2019 www.materna.de 35
More questions?