Watch this talk here: https://www.confluent.io/online-talks/bridge-to-cloud-apache-kafka-migrate-aws
Speakers: Priya Shivakumar, Director of Product, Confluent + Konstantine Karantasis, Software Engineer, Confluent + Rohit Pujari, Partner Solutions Architect, AWS
Most companies start their cloud journey with a new use case, or a new application. Sometimes these applications can run independently in the cloud, but often times they need data from the on premises datacenter. Existing applications will slowly migrate, but will need a strategy and the technology to enable a multi-year migration.
In this session, we will share how companies around the world are using Confluent Cloud, a fully managed Apache Kafka service, to migrate to AWS. By implementing a central-pipeline architecture using Apache Kafka to sync on-prem and cloud deployments, companies can accelerate migration times and reduce costs.
In this online talk we will cover:
•How to take the first step in migrating to AWS
•How to reliably sync your on premises applications using a persistent bridge to cloud
•Learn how Confluent Cloud can make this daunting task simple, reliable and performant
•See a demo of the hybrid-cloud and multi-region deployment of Apache Kafka
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
1. 1
Bridge to Cloud: Using Apache
Kafka to Migrate to AWS
Priya Shivakumar, Confluent Cloud
Konstantine Karantasis, Confluent Cloud
Rohit Pujari, Amazon Web Services
2. 2
Speakers
Priya is the Director of Product at Confluent, where she focuses on product and go-to-market strategy for Confluent Cloud, a fully managed Apache
Kafka service. She brings more than a decade of experience in the tech industry as an engineering and product leader. Prior to Confluent, she led
product marketing and GTM strategy for VSAN at VMware. As a growth strategy consultant at LEK before that, Priya advised clients on a range of
growth issues that include new product introduction, commercialization, geo expansion, and acquisition.
Priya Shivakumar
Director of Product, Confluent
Konstantine Karantasis is a Software Engineer at Confluent, Inc. He’s a main contributor to Apache Kafka and its Connect API and he’s the author of
widely used software, such as Confluent’s S3 and Replicator Connectors, class loading isolation in Apache Kafka Connect, Confluent CLI and more.
Previously, he built scalable open source web-services at Yahoo and did research on high-performance computing at the University of Illinois at
Urbana-Champaign. Konstantine holds a Ph.D. from the University of Patras, Greece.
Konstantine Karantasis
Software Engineer, Confluent
Rohit is a Partner Solutions Architect with AWS. He focuses on growing cloud business of their top technology partners by helping them build,
innovate, and go-to-market with customer-centric solutions on AWS. Rohit brings a wealth of experience in data engineering and analytics from having
worked with customers of all sizes in various stages of their data journey.
Rohit Pujari
Partner Solutions Architect, Amazon Web Services
3. 3
Agenda
The Great “Cloud Shift”
What is a Bridge to Cloud?
Key Considerations for Cloud Migration
A Customer Story
Live Demo
Q&A
4. 4
Poll
How is your infrastructure setup today?
• Mostly on-premises
• Mostly in the cloud
• Combination of on-premises and cloud (hybrid model)
5. 5
The Great “Cloud Shift”
A cloud product is not complete
without a cloud migration story.
11. 11C O N F I D E N T I A L
Apache Kafka, the de-facto standard for real-
time streaming
Real-time | Uses disk structure for constant performance at Petabyte
scale
Scalable | Distributed, scales quickly and easily without downtime
Persistent | Persists messages on disks, enables intra-cluster replication
Reliable | Replicates data, auto balances consumers upon failure
In production at more
than a third of the
Fortune 500
2 trillion messages a
day at LinkedIn
500 billion events a
day (1.3 PB) at Netflix
12. 12
Poll
How are you using Apache Kafka today?
• Apache Kafka on-premises
• Apache Kafka in cloud
• Apache Kafka on-premises and cloud (hybrid)
• Not using Apache Kafka currently
14. 14C O N F I D E N T I A L
Replicator | Reliable, Scalable, Simple
Feature List Replicator Mirror-maker
Reliable Auto creation of topics ✔ Partial
New partition addition
Configuration replication
✔ X
Single message transformations ✔ X
Active-active replication ✔ X
Scalable Aggregate cluster - single management point for multiple
clusters
✔ X
Auto scale - scale replication processes as Kafka traffic
increases with a single configuration
✔ X
Simple Control Center Integration - manage and monitor replication
via Control Center UI
✔ X
Disaster Recovery
support
Active-active replication - redirect events to avoid infinite
replication loops in active-active configurations ✔ X
15. 15
Disaster Recovery and Bridge to Cloud
Enables multi-DC deployments of
Apache Kafka
● On-premises: DC to DC
● Hybrid: DC to Region
● Cloud: Region to Region
Confluent Replicator
21. 21
Confluent
Cloud
manages
Kafka for you
Mission-Critical Reliability
Complete Streaming Service
Freedom of Choice
from the original creators of Apache Kafka
that evolves with your needs
for the best of private and public clouds
22. 22C O N F I D E N T I A L
Confluent Cloud Battle tested for massive
scale, mission-critical pipelines
● Designed by creators of Kafka
● Operated by Kafka committers
● 100+ years of combined Kafka
experience
● 80% of Kafka code commits
● More than a third of Fortune
100 companies trust Confluent
● Sub-25ms^ latencies at multi-
GBps throughput
● Highly secure with ACLs,
RBAC and encryption*
● 99.95% uptime SLA guarantee
● < 1 hr response time SLA for
P1 issues
Delivered by Experts Trusted by Enterprises Backed by Guarantees
^P95 latency
*ACLs, RBAC coming soon to Confluent Cloud
23. 23C O N F I D E N T I A L
DATA COMPATIBILITY | Schema registry
DEVELOPMENT & CONNECTIVITY | Clients | Connectors | REST proxy | KSQL
APACHE KAFKA | Connect | Pub-Sub | Streams
®
Database changes Log events IoT events Web events
Transformations
Custom apps
Analytics
Monitoring
Hadoop
Database
Data warehouse
CRM
Confluent Cloud
DATA INTEGRATION REAL-TIME APPS
Confluent Cloud Kafka re-engineered for cloud
MANAGEMENT & MONITORING | Multi-tenancy | Quotas | Usage tracking and billing
24. 24C O N F I D E N T I A L
Private Cloud
Deploy on-premises in your
datacenter with Confluent
Enterprise
Public Cloud
Migrate to or adopt cloud
at your own pace with fully-
managed Confluent Cloud
Hybrid Cloud
Build a persistent bridge b/n
datacenter and cloud with
Confluent Replicator
Confluent Cloud Industry’s only hybrid Kafka service
25. 25C O N F I D E N T I A L
Confluent | Singular Kafka focus and innovation
Apache Kafka re-engineered for the cloud
Confluent Vision for Kafka
Global
● Automated disaster recovery
● Global applications with geo-awareness
Infinite
● Efficient and infinite data with tiered storage
● Unlimited horizontal scalability for single clusters
● Faster elastic scaling for brokers and partitions
Elastic
● Easy Kubernetes- based orchestration and management with Confluent operator
● Faster elastic scaling when adding brokers and partitions
26. 26C O N F I D E N T I A L
Confluent Cloud on AWS AWS EcosystemOn-premises
Confluent Cloud + AWS Ecosystem
App
Oracle
App
Confluent
Enterprise
Confluent
Replicator
Confluent S3
Connector
Confluent
Cloud
Run SQL queries Visualize data
Run interactive queries
Run ad-hoc and big data analysis
32. 32C O N F I D E N T I A L
Security | Confluent Cloud
● Confluent Cloud offers dedicated clusters
for greater isolation and security
● Confluent Cloud supports VPC peering to
private data transfer
● With VPC peering, your cloud applications
can securely transfer data to your dedicated
Confluent Cloud cluster
● Data does not need to travel over internet
Your AWS VPC
Dedicated
Kafka
(Single-AZ)
CCloud VPC
AWS Region
N1 N2 N3
VPC
Peering
ELB
(Private IP)
Your Datacenter
Confluent
Replicator
Cloud
Applications
33. 33C O N F I D E N T I A L
Availability | Multi-AZ with Confluent Cloud
● Confluent Cloud offers zone-level failure
protection with Multi-AZ clusters
● Confluent Cloud guarantees 99.95%
uptime with SLA
● Maximize availability with 3x replication
across 3 zones
● Use Confluent replicator to stream data
across regions for region-level failure
protection
AZ3AZ2AZ1
Customer VPC
Dedicated
Kafka (Multi-
AZ)
CCloud VPC
AWS Region
N1 N2 N3
VPC
Peering
ELB
(Private IP)
34. 34C O N F I D E N T I A L
Compliance | Confluent Cloud
SOC-2 Type II
compliant
GDPR ready
PCI Phase 1
complete
HIPAA
(coming soon)
35. 35C O N F I D E N T I A L
Cost | Right-size, architect, and outsource as necessary
● Customers often incorrectly size / oversize clusters in the cloud
● Cut costs by right sizing, using reserved instances, and signing up for AWS EDP
Right sizing
Architecting for
cost
Optimizing for time-
to-market
De-risk downtime
and lag
● Getting up and running quickly is critical for costs and competitiveness
● Managed services and consumption-based models help with speed to market
● Distributed systems are complex with lots of APIs, metrics, systems, and configs
● Stateful systems also require careful capacity planning
● Plan carefully, ensure in-house expertise or get expert help to avoid business risk
● Data transfer costs can be significant, especially for data pipelines and streaming systems
● Eliminate point to point data transfer (often duplicate data transfer) with replicator
● Avoid data egress costs for VPC to VPC transfer with Confluent Cloud VPC peering
36. 36C O N F I D E N T I A L
Confluent | Self-managed or fully-managed on AWS
Self-Managed Software
Confluent Platform
The Leading Distribution of Apache Kafka
Deploy on any platform on-premises or on AWS
Fully-Managed Service
Confluent Cloud
Apache Kafka Re-engineered for the Cloud
Available as a fully managed service on AWS
VM
37. 37C O N F I D E N T I A L
Confluent | Complete portfolio of products and
services
built around Kafka
Complete support across the entire adoption lifecycle
Kafka Training Confluent Platform Professional Services Fully Managed Kafka
38. 38
A Customer
Story
A Way to Simplify Diverse
Infrastructure
for a Leading Financial Services
Company
39. 39
A Way to Simplify
Diverse Infrastructure
vs.
● Reduces total traffic between on-
premises systems and cloud
● Uses Confluent Replicator to sync to
the cloud
● Can be used for migrating
applications to the cloud