Modern Cloud-Native Streaming
Platforms: Event Streaming
Microservices with Kafka on
Kubernetes

Speakers
Kamala Dasika
Product Team
Cloud Platform and Ecosystem
Pivotal
@DasikaKN
Michael Ng
Product Manager
Confluent
@Micha8LNg

Changing Model for Application Delivery
Development Deployment
Sparingly at
designated times
Ready for prod at
any time
Architecture
Abstraction
Layer
“Day 2” Ops
App Server on Machine
App on “disposable”
infrastructure
App and Data
Monolith
Microservices /
Composite app
Linear / Sequential
Short cycles, test
driven, iterative
Many tools, ad hoc
automation
Manage services,
not servers

Driven by Modern Cloud Native Platform
Cont
inuous Del
ivery
An idea in the morning can ship by evening
Microservices
Release #1
Microservices
Release #2

Know your abstractions

Hardware
IaaS
CaaS
PaaS
FaaS
HPE, Dell,
IBM, Lenovo
Cloud providers:
AWS, GCP,
Microsoft Azure,
VMware
Kubernetes
Cloud Foundry,
Heroku
AWS Lambda, Azure
Functions,
OpenWhisk, kubeless,
PFS

Hardware
IaaS
CaaS
PaaS
FaaS
Strategic goal: Push as many workloads as
technically feasible to the top of the platform hierarchy
Higher flexibility and
less enforcement of
standards
Lower development
complexity and higher
operational efficiency

Hardware
IaaS
CaaS
PaaS
FaaS
• Data warehouses
• Storage with
regional implications
• Hypervisors
• Relational DBs
• Packaged software
• Virtual appliances
• Application platforms
(cloud abstraction)
• Containerized
Monoliths
• Stateful workloads
• Databases
• Messaging
• 12 -15 factor apps
• stateful and
stateless options
• APIs
• Streaming apps
• DB copies
• Change
synchronization
• Real time stream
analytics

Where to run what?

Goal: Operational consistency AND multiple
abstractions
FREEDOM: Run Your App on Any Cloud
AUTOMATION: Lowers Ops Cost and Increases Velocity
RESILIENCY: Auto-Scale, Blue-green Deploys, Health Mgmt.
GOVERNANCE: Common Developer-Self-Serviced Frameworks
VISIBILITY: Full Stack Logging, Monitoring & Auditability
SPEED & SCALE: Cut Cycle Time from Months to Days/Hours
Built for Cloud
Native
New Initiatives
Existing Workloads
Modernize to
Cloud-Native?

Outcomes trump technology bingo
Needs High Dev Efficiency
Needs Low Dev Efficiency
App Rarely changes App Frequently changes
• Traditional
apps
• Leave as-is
• Re-architect
• Replatform
• CI/CD
• Evaluate
• Evaluate

Evaluating the Enterprise App Portfolio
Business
Technical Economic
Score
• Business criticality
• Risk tolerance
• Change frequency
• Lifecycle stage
• Domain expert availability
• Licensing costs (HW / SW)
• Time-to-market implications
• Revenue impact-fullness
• Codebase
• Suitable framework / runtime
• Relatively lightweight footprint
• No “hard wired” file system
dependencies
• Usage / workload
For each app

Application Transformation Strategies
Re-Host
Containerize aka
“lift and shift” to
CaaS and gain
infrastructure +
platform ops benefits
Re-Platform
Minimal readiness
effort to run on PaaS
(4 -7 factor apps +
data service) that
can run on PaaS
Re-Factor
Convert to cloud-
native/microservice
(12-15 factor app +
data service) runs
well on PaaS/FaaS
Re-Build
Decompose and
rewrite from the
ground up
Application Portfolio
Funneling Process
Learn More: Tools and Recipes to Replatform Monolithic Apps to Modern Cloud Environments

Evolution of the enterprise application portfolio
Learn More: Microservices, Events, and Breaking the Data Monolith with Kafka
Microservices
or Dist. Monolith

Now what about Kafka?

“The world’s most valuable resource is no longer oil, but data” - The
Economist, May 6 2017

Apache Kafka is a Distributed Event
Streaming Platform
Process streams of events In real time, as they occur
110101
010111
001101
100010
Publish and subscribe to
streams of events
Similar to a message queue or
enterprise messaging system
110101
010111
001101
100010
Store streams of events In a fault tolerant way
110101
010111
001101
100010

Events
A Sale An Invoice A Trade A Customer
Experience

What is a company?
A business is a series of events and
reacting to those events.

Events enable Microservice Architectures -
Let’s refactor!
Submit
Order
Orders
Service
Shipping
Service
Customer
Service
Webserver
E-Commerce
Microservices (with RPC)
- Orders Service calls
Shipping Service to tell it
to ship item.
- Shipping service looks
up address to ship to
(from Customer Service)
- No Kafka 😢shipOrder() getCustomer()

Refactoring Orders and Shipping
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
RPC
Orders
Service
Shipping
Service
Customer
Service
Webserver
- Orders Service no longer
knows about the Shipping
service (or any other service).
- Events are fire and forget.

Refactoring Customers
Customer
Updated
Submit
Order
Order
Created
Webserver
KAFKA
Orders
Service
Shipping
Service
Customer
Service
- Call to Customer service is
gone.
- Data is replicated, as events,
into the shipping service, where
it is queried locally.

Events are the key to scalable microservices
Orders
Service

Kafka on Kubernetes

Why run Kafka on
Kubernetes?
33% of Confluent
Customers plan to
run Production
Kafka on Kubernetes

Microservices - driving container and
container orchestration adoption
Breaking down the monolith - enables dev teams to work and
scale independently
µ
Containers allow services to be independently deployable services
● Each service has its own purpose & implementation language
● Each service has its own dependencies
Orchestration - running containers at scale
● Microservices = container deployments at scale
● Service discovery
● Scale services independently
● High Availability, Self Healing of services
● Hardware Abstraction

The world seems to think Kafka on
Kubernetes is Difficult

There are some challenges
Translating an existing architecture to Kubernetes: K8s
pods, nodes, stateful sets, custom resources
External access: What do we do with advertised listeners
on Kubernetes?
Managing configuration of Kafka, Zookeeper and all
Confluent Components
Persistent Storage options on prem and clouds
Security Configuration and Upgrades

Confluent Operator enables you to:
Automate provisioning of Kafka
pods in minutes
Monitor SLAs through Confluent
Control Center or Prometheus
Scale Kafka elastically &
Automate rolling updates
Built on our first hand knowledge of
running Confluent at scale
Confluent Operator
is our Kubernetes
Operator and our
first step towards
becoming cloud
native
CONFIDENTIAL

Helm - The Confluent Operator Package
Manager ● Confluent Operator leverages Helm Charts to
deploy, upgrade and uninstall Confluent
Platform custom resources and pods
● Configuration front end for users to specify
how a Confluent Platform Cluster is deployed:
○ # of replicas for Kafka, Zookeeper
○ Security and Authentication configuration
○ Persistent Storage configuration
● Cluster configuration edits are also performed
using Helm
Operator Helm
Charts - yaml

Confluent Operator - Automated Provisioning

Confluent Operator - Scale Horizontally
Automate Scaling:
Spin up new brokers, connect workers
easily
Distribute partitions to new brokers:
Determine balancing plan
Execute balancing plan
Monitor Resources

Confluent Operator - Rolling Upgrade of all
components
Automated Rolling Upgrades of all
components - Kafka Brokers,
Zookeeper, Connect, Control Center
Kafka Broker Upgrades:
1. Stop the broker, upgrade Kafka
2. Wait for Partition Leader reassignment
3. Start the upgraded broker
4. Wait for zero under-replicated partitions
5. Upgrade the next broker

Confluent Operator - Automated Security
Configuration
SASL PLAIN and Mutual TLS
Authentication
Automate configuration of truststores
and keystores with secret objects
Automate configuration of Kafka and
all Confluent Platform Components

Confluent Operator GA Plans
We are in private Preview Release now
24 customers testing the Operator in Preview:
● Global customers
● Banks, Fin Tech, Retailers, Consumer Tech
We are in the final stages of Preview and about to launch soon.

Make this
easy!!!

Leverage multiple platforms/abstractions

Demo – Provision a Complex Cluster in K8s
with Confluent Operator
● Provision:
○ 3 Kafka Brokers, 3 Zookeeper Nodes
○ Confluent Control Center
○ Provisioned on GKE (7 node cluster)
● Mutual TLS Authentication, External Access via Service Endpoints, DNS
● Pub/Sub example (remote access to Kafka Cluster)