Modern Cloud-Native Streaming
Platforms:
Event Streaming Microservices with
Kafka on Kubernetes
Kamala Dasika Michael Ng Victor Gamov
@DasikaKN @Micha8LNg @gAmUssA

Changing Model for Application Delivery
Development Deployment
Sparingly at
designated times
Ready for prod at
any time
Architecture
Abstraction
Layer
“Day 2” Ops
App Server on Machine
App on “disposable”
infrastructure
App and Data
Monolith
Microservices /
Composite app
Linear / Sequential
Short cycles, test
driven, iterative
Many tools, ad hoc
automation
Manage services,
not servers

Driven by Modern Cloud Native Platform
Cont
inuous Del
ivery
DEV OPS
Customer
Feedback
Customer
Delivery
An idea in the morning can ship by evening
Microservices
Release #1
Microservices
Release #2

Know your abstractions

Hardware
IaaS
CaaS
PaaS
FaaS
HPE, Dell,
IBM, Lenovo
Cloud providers:
AWS, GCP,
Microsoft Azure,
VMware
Kubernetes
Cloud Foundry,
Heroku
AWS Lambda, Azure
Functions, OpenWhisk,
kubeless, PFS

Hardware
IaaS
CaaS
PaaS
FaaS
• Data warehouses
• Storage with
regional implications
• Hypervisors
• Relational DBs
• Packaged software
• Virtual appliances
• Application platforms
(cloud abstraction)
• Containerized
Monoliths
• Stateful workloads
• Databases
• Messaging
• 12 -15 factor apps
• stateful and
stateless options
• APIs
• Streaming apps
• DB copies
• Change
synchronization
• Real time stream
analytics

Hardware
IaaS
CaaS
PaaS
FaaS
Strategic goal: Push as many workloads as technically
feasible to the top of the platform hierarchy
Higher flexibility and
less enforcement of
standards
Lower development
complexity and higher
operational efficiency

What’s best for my app?

Goal: Operational consistency AND multiple abstractions
FREEDOM: Run Your App on Any Cloud
AUTOMATION: Lowers Ops Cost and Increases Velocity
RESILIENCY: Auto-Scale, Blue-green Deploys, Health Mgmt.
GOVERNANCE: Common Developer-Self-Serviced Frameworks
VISIBILITY: Full Stack Logging, Monitoring & Auditability
SPEED & SCALE: Cut Cycle Time from Months to Days/Hours
Built for Cloud
Native
New Initiatives
Existing Workloads
Modernize to
Cloud-Native?

Outcomes trump technology bingo
Needs High Dev Efficiency
Needs Low Dev Efficiency
App Rarely changes App Frequently changes
• Traditional
apps
• Leave as-is
• Re-architect
• Replatform
• CI/CD
• Evaluate
• Evaluate

Evaluating the Enterprise App Portfolio
Business
Technical Economic
Score
• Business criticality
• Risk tolerance
• Change frequency
• Lifecycle stage
• Domain expert availability
• Licensing costs (HW / SW)
• Time-to-market implications
• Revenue impact-fullness
• Codebase
• Suitable framework / runtime
• Relatively lightweight footprint
• No “hard wired” file system
dependencies
• Usage / workload
For each app

Application Transformation Strategies
Re-Host
Containerize aka
“lift and shift” to
CaaS and gain
infrastructure +
platform ops benefits
Re-Platform
Minimal readiness
effort to run on PaaS
(4 -7 factor apps +
data service) that
can run on PaaS
Re-Factor
Convert to cloud-
native/microservice
(12-15 factor app +
data service) runs
well on PaaS/FaaS
Re-Build
Decompose and
rewrite from the
ground up
Application Portfolio
Funneling Process
Learn More: Tools and Recipes to Replatform Monolithic Apps to Modern Cloud Environments

Evolution of the enterprise application portfolio
Learn More: Microservices, Events, and Breaking the Data Monolith with Kafka

Now what about Data?

“The world’s most valuable resource is no longer oil, but data” - The
Economist, May 6 2017

Apache Kafka is a Distributed Event Streaming
Platform
Process streams of events In real time, as they occur
110101
010111
001101
100010
Publish and subscribe to
streams of events
Similar to a message queue or
enterprise messaging system
110101
010111
001101
100010
Store streams of events In a fault tolerant way
110101
010111
001101
100010

Events
A Sale An Invoice A Trade A Customer
Experience

What is a company?
A business is a series of events and reacting to
those events.

All Your Data is a Stream of Events

Events enable Microservice Architectures - Let’s
refactor!
Submit
Order
Orders
Service
Shipping
Service
Customer
Service
Webserver
E-Commerce
Microservices (with RPC)
- Orders Service calls
Shipping Service to tell it
to ship item.
- Shipping service looks
up address to ship to
(from Customer Service)
- No Kafka !shipOrder() getCustomer()

Refactoring Orders and Shipping
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
RPC
Orders
Service
Shipping
Service
Customer
Service
Webserver
- Orders Service no longer
knows about the Shipping
service (or any other service).
- Events are fire and forget.

Refactoring Customers
Customer
Updated
Submit
Order
Order
Created
Webserver
KAFKA
Orders
Service
Shipping
Service
Customer
Service
- Call to Customer service is
gone.
- Data is replicated, as events,
into the shipping service, where
it is queried locally.

Events are the key to scalable microservices
Orders
Service

Kafka on Kubernetes

Why run Kafka on
Kubernetes?
33% of Confluent
Customers plan to run
Production Kafka on
Kubernetes

Microservices - driving container and container
orchestration adoption
Breaking down the monolith - enables dev teams to work and scale
independently
µ
Containers allow services to be independently deployable services
● Each service has its own purpose & implementation language
● Each service has its own dependencies
Orchestration - running containers at scale
● Microservices = container deployments at scale
● Service discovery
● Scale services independently
● High Availability, Self Healing of services
● Hardware Abstraction

The world seems to think Kafka on Kubernetes is
Difficult

There are some challenges
Translating an existing architecture to Kubernetes: K8s pods,
nodes, stateful sets, custom resources
External access: What do we do with advertised listeners on
Kubernetes?
Managing configuration of Kafka, Zookeeper and all Confluent
Components
Persistent Storage options on prem and clouds
Security Configuration and Upgrades

Confluent Operator enables you to:
Automate provisioning of Kafka pods
in minutes
Monitor SLAs through Confluent
Control Center or Prometheus
Scale Kafka elastically & Automate
rolling updates
Built on our first hand knowledge of
running Confluent at scale
Confluent Operator is
our Kubernetes
Operator and our first
step towards becoming
cloud native
CONFIDENTIAL

Confluent Operator - Automated Provisioning

Confluent Operator - Scale Horizontally
Automate Scaling:
Spin up new brokers, connect workers
easily
Distribute partitions to new brokers:
Determine balancing plan
Execute balancing plan
Monitor Resources

Confluent Operator - Rolling Upgrade of all
components
Automated Rolling Upgrades of all
components - Kafka Brokers, Zookeeper,
Connect, Control Center
Kafka Broker Upgrades:
1. Stop the broker, upgrade Kafka
2. Wait for Partition Leader reassignment
3. Start the upgraded broker
4. Wait for zero under-replicated partitions
5. Upgrade the next broker

Confluent Operator - Automated Security
Configuration
SASL PLAIN and Mutual TLS
Authentication
Automate configuration of truststores
and keystores with secret objects
Automate configuration of Kafka and all
Confluent Platform Components

Demo – Ride Sharing Location Tracking
2 Web Apps powered by a
microservice each
Apps are a ride sharing driver and
rider location tracking
Rider hails a ride and the driver
drives towards the rider
Real time location events powered
by Kafka on Kubernetes
Rider Driver

Confluent Operator GA Plans
We are in private Preview Release now
24 customers testing the Operator in Preview:
● Global customers
● Banks, Fin Tech, Retailers, Consumer Tech
We are in the final stages of Preview and about to launch soon.

!!!

Leverage multiple platforms/abstractions

Kamala Dasika
Product team,
Platform and
Ecosystem, Pivotal
Michael Ng
Product Manager,
Confluent
Thank You!
Viktor Gamov
Developer Advocate,
Confluent

Modern Cloud-Native Streaming
Platforms:
Event Streaming Microservices with
Kafka on Kubernetes
Kamala Dasika Michael Ng Victor Gamov
@DasikaKN @Micha8LNg @gAmUssA