Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Modern Cloud-Native Streaming
Platforms: Event Streaming
Microservices with Kafka on
Kubernetes
Speakers
Kamala Dasika
Product Team
Cloud Platform and Ecosystem
Pivotal
@DasikaKN
Michael Ng
Product Manager
Confluent
@M...
Changing Model for Application Delivery
Development Deployment
Sparingly at
designated times
Ready for prod at
any time
Ar...
Driven by Modern Cloud Native Platform
Cont
inuous Del
ivery
An idea in the morning can ship by evening
Microservices
Rele...
Know your abstractions
Hardware
IaaS
CaaS
PaaS
FaaS
HPE, Dell,
IBM, Lenovo
Cloud providers:
AWS, GCP,
Microsoft Azure,
VMware
Kubernetes
Cloud Fo...
Hardware
IaaS
CaaS
PaaS
FaaS
Strategic goal: Push as many workloads as
technically feasible to the top of the platform hie...
Hardware
IaaS
CaaS
PaaS
FaaS
• Data warehouses
• Storage with
regional implications
• Hypervisors
• Relational DBs
• Packa...
Where to run what?
Goal: Operational consistency AND multiple
abstractions
FREEDOM: Run Your App on Any Cloud
AUTOMATION: Lowers Ops Cost and...
Outcomes trump technology bingo
Needs High Dev Efficiency
Needs Low Dev Efficiency
App Rarely changes App Frequently chang...
Evaluating the Enterprise App Portfolio
Business
Technical Economic
Score
• Business criticality
• Risk tolerance
• Change...
Application Transformation Strategies
Re-Host
Containerize aka
“lift and shift” to
CaaS and gain
infrastructure +
platform...
Evolution of the enterprise application portfolio
Learn More: Microservices, Events, and Breaking the Data Monolith with K...
Now what about Kafka?
“The world’s most valuable resource is no longer oil, but data” - The
Economist, May 6 2017
Apache Kafka is a Distributed Event
Streaming Platform
Process streams of events In real time, as they occur
110101
010111...
Events
A Sale An Invoice A Trade A Customer
Experience
What is a company?
A business is a series of events and
reacting to those events.
Events enable Microservice Architectures -
Let’s refactor!
Submit
Order
Orders
Service
Shipping
Service
Customer
Service
W...
Refactoring Orders and Shipping
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
RPC
Orders
Service
Shippin...
Refactoring Customers
Customer
Updated
Submit
Order
Order
Created
Webserver
KAFKA
Orders
Service
Shipping
Service
Customer...
Events are the key to scalable microservices
Orders
Service
Kafka on Kubernetes
Why run Kafka on
Kubernetes?
33% of Confluent
Customers plan to
run Production
Kafka on Kubernetes
Microservices - driving container and
container orchestration adoption
Breaking down the monolith - enables dev teams to w...
The world seems to think Kafka on
Kubernetes is Difficult
There are some challenges
Translating an existing architecture to Kubernetes: K8s
pods, nodes, stateful sets, custom resou...
Confluent Operator enables you to:
Automate provisioning of Kafka
pods in minutes
Monitor SLAs through Confluent
Control C...
Helm - The Confluent Operator Package
Manager ● Confluent Operator leverages Helm Charts to
deploy, upgrade and uninstall ...
Confluent Operator - Automated Provisioning
Confluent Operator - Scale Horizontally
Automate Scaling:
Spin up new brokers, connect workers
easily
Distribute partition...
Confluent Operator - Rolling Upgrade of all
components
Automated Rolling Upgrades of all
components - Kafka Brokers,
Zooke...
Confluent Operator - Automated Security
Configuration
SASL PLAIN and Mutual TLS
Authentication
Automate configuration of t...
Confluent Operator GA Plans
We are in private Preview Release now
24 customers testing the Operator in Preview:
● Global c...
Make this
easy!!!
Leverage multiple platforms/abstractions
Demo – Provision a Complex Cluster in K8s
with Confluent Operator
● Provision:
○ 3 Kafka Brokers, 3 Zookeeper Nodes
○ Conf...
Nächste SlideShare
Wird geladen in …5
×

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Apache Kafka on Kubernetes

1.144 Aufrufe

Veröffentlicht am

Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.

Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.

This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Apache Kafka on Kubernetes

  1. 1. Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes
  2. 2. Speakers Kamala Dasika Product Team Cloud Platform and Ecosystem Pivotal @DasikaKN Michael Ng Product Manager Confluent @Micha8LNg
  3. 3. Changing Model for Application Delivery Development Deployment Sparingly at designated times Ready for prod at any time Architecture Abstraction Layer “Day 2” Ops App Server on Machine App on “disposable” infrastructure App and Data Monolith Microservices / Composite app Linear / Sequential Short cycles, test driven, iterative Many tools, ad hoc automation Manage services, not servers
  4. 4. Driven by Modern Cloud Native Platform Cont inuous Del ivery An idea in the morning can ship by evening Microservices Release #1 Microservices Release #2
  5. 5. Know your abstractions
  6. 6. Hardware IaaS CaaS PaaS FaaS HPE, Dell, IBM, Lenovo Cloud providers: AWS, GCP, Microsoft Azure, VMware Kubernetes Cloud Foundry, Heroku AWS Lambda, Azure Functions, OpenWhisk, kubeless, PFS
  7. 7. Hardware IaaS CaaS PaaS FaaS Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Higher flexibility and less enforcement of standards Lower development complexity and higher operational efficiency
  8. 8. Hardware IaaS CaaS PaaS FaaS • Data warehouses • Storage with regional implications • Hypervisors • Relational DBs • Packaged software • Virtual appliances • Application platforms (cloud abstraction) • Containerized Monoliths • Stateful workloads • Databases • Messaging • 12 -15 factor apps • stateful and stateless options • APIs • Streaming apps • DB copies • Change synchronization • Real time stream analytics
  9. 9. Where to run what?
  10. 10. Goal: Operational consistency AND multiple abstractions FREEDOM: Run Your App on Any Cloud AUTOMATION: Lowers Ops Cost and Increases Velocity RESILIENCY: Auto-Scale, Blue-green Deploys, Health Mgmt. GOVERNANCE: Common Developer-Self-Serviced Frameworks VISIBILITY: Full Stack Logging, Monitoring & Auditability SPEED & SCALE: Cut Cycle Time from Months to Days/Hours Built for Cloud Native New Initiatives Existing Workloads Modernize to Cloud-Native?
  11. 11. Outcomes trump technology bingo Needs High Dev Efficiency Needs Low Dev Efficiency App Rarely changes App Frequently changes • Traditional apps • Leave as-is • Re-architect • Replatform • CI/CD • Evaluate • Evaluate
  12. 12. Evaluating the Enterprise App Portfolio Business Technical Economic Score • Business criticality • Risk tolerance • Change frequency • Lifecycle stage • Domain expert availability • Licensing costs (HW / SW) • Time-to-market implications • Revenue impact-fullness • Codebase • Suitable framework / runtime • Relatively lightweight footprint • No “hard wired” file system dependencies • Usage / workload For each app
  13. 13. Application Transformation Strategies Re-Host Containerize aka “lift and shift” to CaaS and gain infrastructure + platform ops benefits Re-Platform Minimal readiness effort to run on PaaS (4 -7 factor apps + data service) that can run on PaaS Re-Factor Convert to cloud- native/microservice (12-15 factor app + data service) runs well on PaaS/FaaS Re-Build Decompose and rewrite from the ground up Application Portfolio Funneling Process Learn More: Tools and Recipes to Replatform Monolithic Apps to Modern Cloud Environments
  14. 14. Evolution of the enterprise application portfolio Learn More: Microservices, Events, and Breaking the Data Monolith with Kafka Microservices or Dist. Monolith
  15. 15. Now what about Kafka?
  16. 16. “The world’s most valuable resource is no longer oil, but data” - The Economist, May 6 2017
  17. 17. Apache Kafka is a Distributed Event Streaming Platform Process streams of events In real time, as they occur 110101 010111 001101 100010 Publish and subscribe to streams of events Similar to a message queue or enterprise messaging system 110101 010111 001101 100010 Store streams of events In a fault tolerant way 110101 010111 001101 100010
  18. 18. Events A Sale An Invoice A Trade A Customer Experience
  19. 19. What is a company? A business is a series of events and reacting to those events.
  20. 20. Events enable Microservice Architectures - Let’s refactor! Submit Order Orders Service Shipping Service Customer Service Webserver E-Commerce Microservices (with RPC) - Orders Service calls Shipping Service to tell it to ship item. - Shipping service looks up address to ship to (from Customer Service) - No Kafka 😢shipOrder() getCustomer()
  21. 21. Refactoring Orders and Shipping Message Broker (Kafka) Submit Order Order Created getCustomer() RPC Orders Service Shipping Service Customer Service Webserver - Orders Service no longer knows about the Shipping service (or any other service). - Events are fire and forget.
  22. 22. Refactoring Customers Customer Updated Submit Order Order Created Webserver KAFKA Orders Service Shipping Service Customer Service - Call to Customer service is gone. - Data is replicated, as events, into the shipping service, where it is queried locally.
  23. 23. Events are the key to scalable microservices Orders Service
  24. 24. Kafka on Kubernetes
  25. 25. Why run Kafka on Kubernetes? 33% of Confluent Customers plan to run Production Kafka on Kubernetes
  26. 26. Microservices - driving container and container orchestration adoption Breaking down the monolith - enables dev teams to work and scale independently µ Containers allow services to be independently deployable services ● Each service has its own purpose & implementation language ● Each service has its own dependencies Orchestration - running containers at scale ● Microservices = container deployments at scale ● Service discovery ● Scale services independently ● High Availability, Self Healing of services ● Hardware Abstraction
  27. 27. The world seems to think Kafka on Kubernetes is Difficult
  28. 28. There are some challenges Translating an existing architecture to Kubernetes: K8s pods, nodes, stateful sets, custom resources External access: What do we do with advertised listeners on Kubernetes? Managing configuration of Kafka, Zookeeper and all Confluent Components Persistent Storage options on prem and clouds Security Configuration and Upgrades
  29. 29. Confluent Operator enables you to: Automate provisioning of Kafka pods in minutes Monitor SLAs through Confluent Control Center or Prometheus Scale Kafka elastically & Automate rolling updates Built on our first hand knowledge of running Confluent at scale Confluent Operator is our Kubernetes Operator and our first step towards becoming cloud native CONFIDENTIAL
  30. 30. Helm - The Confluent Operator Package Manager ● Confluent Operator leverages Helm Charts to deploy, upgrade and uninstall Confluent Platform custom resources and pods ● Configuration front end for users to specify how a Confluent Platform Cluster is deployed: ○ # of replicas for Kafka, Zookeeper ○ Security and Authentication configuration ○ Persistent Storage configuration ● Cluster configuration edits are also performed using Helm Operator Helm Charts - yaml
  31. 31. Confluent Operator - Automated Provisioning
  32. 32. Confluent Operator - Scale Horizontally Automate Scaling: Spin up new brokers, connect workers easily Distribute partitions to new brokers: Determine balancing plan Execute balancing plan Monitor Resources
  33. 33. Confluent Operator - Rolling Upgrade of all components Automated Rolling Upgrades of all components - Kafka Brokers, Zookeeper, Connect, Control Center Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under-replicated partitions 5. Upgrade the next broker
  34. 34. Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
  35. 35. Confluent Operator GA Plans We are in private Preview Release now 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech We are in the final stages of Preview and about to launch soon.
  36. 36. Make this easy!!!
  37. 37. Leverage multiple platforms/abstractions
  38. 38. Demo – Provision a Complex Cluster in K8s with Confluent Operator ● Provision: ○ 3 Kafka Brokers, 3 Zookeeper Nodes ○ Confluent Control Center ○ Provisioned on GKE (7 node cluster) ● Mutual TLS Authentication, External Access via Service Endpoints, DNS ● Pub/Sub example (remote access to Kafka Cluster)

×