1.
Moving 150 TB of data resiliently
on Kafka With Quorum Controller
on Kubernetes
Murali Kaundinya
Troy Stow

2.
Outline
• Our current installed base of KAFKA...
• Operational metrics and growth trajectory…
• KAFKA in a Hybrid/Multi Cloud environment
• Replicator vs. Cluster Link
• Zookeeper vs. Quorum Controller
• Operators: Confluent for Kubernetes, KEDA and Strimzi
• Summary
2

3.
Jithendra Manne Keith Kroculick
Suresh Veda
Patrick White
Troy Stow Shyam Gadde Venky Ramakrishnan
Murali Kaundinya Prasanth Kuppa
Girish Tavag 3

4.
KEES Platform Overview
4
Vision Statement
Kafka Enterprise Event Streaming (KEES) will provide our Line of Business partners with an enterprise event
streaming capability that allows them to implement event driven architecture and integration patterns for
business and operational transaction, be able to dynamically provision with minimum hand holding and scale
with minimal friction.
Goals
• Enablement of Confluent Platform to the Core Data Centers
• Design and implementation of functional clusters based upon business requirements
• Provide engineer guidance and support for the infrastructure being deployed in Wells Fargo
• Document and operationalize management and support of the ecosystem
• Develop self-service capabilities, in partnership with Architecture

5.
KEES Platform Ecosystem
5

6.
Confluent Cluster Detail
6

7.
Confluent features offered to LOB partners
7

8.
Event Streaming Requirements/Hybrid Cloud
• Consistency across on-prem, one or more Cloud Service Providers
• Should Event Streaming be centralized, federated or localized?
• Ease of use, development and operations.
• Run Event Streaming on Kubernetes as the core substrate.
• Secure, Auto-Scaling, Portable, Managed Self-Service.
• Abstraction for several Functional & Non-Functional Requirements.
• Manage down middleware sprawl and be interoperable.
• Backing Services, Config., Processes, Disposability, Concurrency
8

9.
KAFKA in a Hybrid/Multi Cloud environment
9

10.
Zookeeper vs. Quorum Controller
• Metadata Management
• Duplication Functionality
• Additional Resources
10

11.
Confluent for Kubernetes
Quorum
Controller
11

12.
Confluent for Kubernetes
• Cloud-Native Declarative API:
• Declarative Kubernetes-native API
• Configure, deploy, and manage through IaC: GitOps
• Cloud-Native Security Best Practices
• RBAC, AuthN, TLS, Certs/Secrets
• Upgrades
• Automated rolling updates to configuration.
• Automated rolling upgrades w/o affecting downtime.
• Auto-Scaling
• API
• Reliability Checks.
12

13.
Confluent for Kubernetes
• Resiliency
• Restores Pod on failures.
• Automated rack awareness to spread replicas of a partition;
• Improves availability.
• Scheduling
• Supports Kubernetes labels & annotations
• Supports Kubernetes tolerations and pod/node affinity for pod placement.
• Monitoring
• Supports aggregated metrics export to Prometheus
13

14.
Summary
Features
• Cluster Link
• Support “Fail Forward”
• Confluent for Kubernetes
• Auto Deployment & Auto Scaling
• Configure once and run
everywhere.
• Kafka Stream
• Custom Utility for masking and
filtering
• ksqlDB
Lessons Learnt
• Quorum Controller has potential.
• Good for pilots.
• Test it extensively for enterprise use.
• Needs better documentation.
• Needs community engagement.
• Recommend Kafka 3.0 or 6.1.2
• Can use better developer guides
on new features.
14

15.
Thank you!