(Shrinand Javadekar, Intuit Inc.) Kafka Summit SF 2018 Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. It has great features for availability, scalability, monitoring, ease of deployment, a rich set of tools and an extremely fast-growing ecosystem that is making it ever more useful. However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data-driven approach for evaluating Kafka on Kubernetes in AWS. In this talk, we will provide details of our functional and non-functional requirements, the experimental configuration and the details of the evaluation. The evaluation process included functional tests for producing/consuming messages, network isolation tests, cross-region tests as well as performance and stress tests. We will focus on the problems we ran into and how we addressed them. This talk will demonstrate a Kubernetes cluster running Kafka along with the details of how each component is configured. Specifically, we will cover the Kafka and ZooKeeper StatefulSets, the ConfigMaps used for storing the server.properties used by all brokers, the service objects for enabling access to the brokers, securing the data and, last but not least, integration with Splunk and Wavefront for logging and monitoring respectively.

1. Shrinand Javadekar
Intuit, Inc.
Kafka On Kubernetes: From
Evaluation to Production at Intuit

2. 2Intuit Confidential and Proprietary
Agenda
● Why Kafka?
● Why Kubernetes?
● Kafka on Kubernetes Architecture
● Problems and solutions!

3. 3Intuit Confidential and Proprietary
Why Kafka?
● Intuit QuickBooks is a monolithic application accessed by million of
customers
● Complex in nature, tightly coupled internal modules
● Continuous deployment is harder
● Harder to scale
● Post decomposition, Kafka would serve as the mechanism for sharing
data between microservices.

4. 4Intuit Confidential and Proprietary
Requirements
- 9000 msgs / second
- 5KB message (assuming 10x compression)
- 7 day retention
- 3 replicas
- ~200 topics

5. 5Intuit Confidential and Proprietary
Why Kubernetes for Kafka?
● De-facto standard for running containerized applications
● Integrates nicely with AWS components
● Great set of features for Kafka:
○ StatefulSets, ConfigMaps, Secrets, Pod/node affinity
● Extensibility offered by containerized environments

6. 6Intuit Confidential and Proprietary
Kafka on Kubernetes: Getting off the ground
https://github.com/Yolean/kubernetes-kafka

7. 7Intuit Confidential and Proprietary
Performance: Experiments and Results (Baseline)
Setup:
● 5 instances of type r4.2xlarge (8 vcpu, 61GB
memory)
● Spread across 3 AZs
● Broker config had 8 vcpu, 6GB memory
● Replication factor of 3
● Acks = all
● min.insync.replicas = 2
● 10 producers
● 2000 messages
● Message size: 10KB
Note:
● No SSL
● No “background stress”
● No compression

8. 8Intuit Confidential and Proprietary
Performance: Experiments and Results (Baseline)

9. 9Intuit Confidential and ProprietaryIntuit Confidential and ProprietaryIntuit Confidential and Proprietary
Kafka on Kubernetes Architecture

10. 10Intuit Confidential and ProprietaryIntuit Confidential and ProprietaryIntuit Confidential and Proprietary
Kafka on Kubernetes Architecture

11. 11Intuit Confidential and Proprietary
Kafka on Kubernetes Architecture
● Kafka and ZK run in independent Autoscaling Groups
● One Kafka broker per EC2-instance
● One Zookeeper per EC2-instance

12. 12Intuit Confidential and ProprietaryIntuit Confidential and ProprietaryIntuit Confidential and Proprietary
Kafka on Kubernetes In Action!

13. 13Intuit Confidential and Proprietary
NLB Configuration
● How does one NLB need to be configured?
○ Each broker listens on a unique port
○ Create on Target Group for each broker:port
○ An extra Target Group for bootstrap

14. 14Intuit Confidential and Proprietary
Security
● Authentication
○ Mutual TLS
● Security
○ Over the wire: TLS
○ Encryption of data at rest (AWS EBS volume feature)

15. 15Intuit Confidential and Proprietary
Logging and Monitoring
● Logging
○ Fluentd Daemonset runs a pod on all instances.
○ Kafka and Zookeeper logs from the instances are written by Fluentd to Splunk.
● Monitoring
○ Kubernetes metrics are captured by Heapster and sent to Wavefront
○ Kafka and Zookeeper pods use the “telegraf” sidecar container to send specific metrics to
Wavefront

16. 16Intuit Confidential and Proprietary
Tools and utilities
● Yahoo Kafka-manager :
○ https://github.com/yahoo/kafka-manager
● ZK web ui:
○ https://github.com/qiuxiafei/zk-web
○ https://github.com/tobilg/docker-zookeeper-webui
● Argo:
○ https://github.com/argoproj/argo

17. 17Intuit Confidential and Proprietary
Performance: Experiments and Results
Setup:
● 9 instances of type r4.2xlarge (8 vcpu, 61GB
memory)
● Spread across 3 AZs
● Broker config had 8 vcpu, 6GB memory
● Replication factor of 3
● Acks = all
● min.insync.replicas = 2
● 10 producers
● 2000 messages
● Message size: ~60KB
Note:
● With SSL
● With and without background stress
● Snappy compression
● Background stress
● 3 producers trying to generate 4500
messages per second, message size is 25K.

18. 18Intuit Confidential and Proprietary
Performance: Experiments and Results

19. 19Intuit Confidential and Proprietary
Watch out for ...
● JMX metrics may not be supported directly
● NAT Gateways on AWS don’t support IP fragmentation for TCP
● Set TCP MTU correctly
● Log message rate can be very high

20. 20Intuit Confidential and Proprietary
Conclusions
● Kafka on Kubernetes ....

21. Thank You
Email: shrinand_javadekar@intuit.com
Twitter: @shrinandj