2. Building streaming data pipelines
with Confluent’s latest innovations
Engin Cukuroglu, PhD
Manager - Solutions Engineering
3. Intro
Understand Confluent’s product strategy
01
Recent Product Updates
12 month summary of Confluent launches
02
Demo
Confluent Cloud data streaming in action
03
Session Agenda
Confluent Roadmap
See the future of cloud-native data streaming
04
Getting Started
Start your free trial of Confluent Cloud today
05
5. Finding your solution for Apache Kafka
Weighing the expenses and risks of today’s different deployment options
Self Managing
Networking spend for
increased throughput
Excess, underutilized
storage/compute
Resources to manage
infrastructure failures
Managed Services
Snowflake
…comes with unnecessary
cost and complexity.
…don’t serve all your data
streaming needs.
…continue to deliver success
across every category.
Hosted Services
Networking considered
“out of scope”, and can
be 90% of your TCO
Kafka reliability,
patching, upgrades,
etc. on the user
Incomplete solutions
that don’t address all
data streaming needs
6. Confluent Cloud
Cloud Native
Apache Kafka®
, fully
managed and
re-architected to harness
the power of the cloud
Complete
Go above & beyond Kafka
with all the essential tools
for a complete data
streaming platform
Everywhere
Connect your data in real
time with a platform that
spans from on-prem to
cloud and across clouds
Cloud-native data streaming platform that is 10x better than Kafka
Confluent Cloud customers lower their overall TCO for Apache Kafka by up to 60%
7. Simple Hybrid/Multicloud Streaming
Confluent powers…
Enterprise-grade Security & Governance Modern, cloud-based data systems
Cloud-native / SaaS apps
Azure Synapse
Analytics
Legacy data systems
Mainframes
Applications
ksqlDB
in-flight stream processing
Infinite storage clusters supporting up
to 20 GBps overall throughput with a
99.99% Uptime SLA
120+ Pre-Built Source & Sink Connectors (70+ provided fully managed)
Hybrid and Multicloud
Cluster Linking | Replicator
Self-managed software
Kubernetes | VMs | Bare Metal
Fully managed cloud service
AWS | Azure | GCP
Flexible DevOps Automation
Terraform Provider | Ansible Playbooks
Source
Connectors
Sink
Connectors
Custom
Throughput
Limits by
App
Custom
Throughput
Limits by
App
8. Quarterly Bundled Launches
Everywhere
Connect your data in real
time with a platform that
spans from on-prem to
cloud and across clouds
Complete
Go above & beyond Kafka
with all the essential tools
for a complete data
streaming platform
Cloud Native
Apache Kafka®
, fully
managed and
re-architected to harness
the power of the cloud
Expanding upon Cloud Native, Complete, and Everywhere product differentiation
Net-new capabilities added to each pillar every quarter
10. Cloud Native
Apache Kafka®, fully managed and re-architected to harness the power of the cloud
Highlighted “Cloud Native” updates from the past 12 months
Dedicated Cluster
Expand & Shrink
Increased Uptime SLA
for Apache Kafka
99.99%
Cloud Client Quotas
Higher Throughput for
Serverless Clusters
Independent Network
Lifecycle Management
Confluent Terraform
Provider
11. Confluent Terraform Provider
Built with and officially verified by Hashicorp; Available open source on GitHub.
Confluent Cloud
Infrastructure as Code
CI/CD Automation
Multicloud data streaming
scaled with GitOps integration
+ =
Ongoing automation and
continuous monitoring throughout
the real-time application lifecycle
Resources & data sources supported include:
Kafka Clusters Topics
Networks Connectors RBAC Service Accounts + more
CODE
P
L
A
N
BUILD
TEST
R
E
L
E
A
S
E
DEPLOY
OPERATE
MONITOR
CI CD
Automate and standardize version-controlled infrastructure management across AWS, Azure, and Google Cloud
12. Complete
Go above & beyond Kafka with all the essential tools for a complete data streaming platform
Highlighted “Complete” updates from the past 12 months
70+ Fully Managed
Connectors
Enhanced RBAC
Centralized Identity
Management (OAuth)
Monitoring
Integrations
Stream Processing
Recipes
Stream Designer
Stream Governance
Advanced
Spotlight features within today’s product demo
13. Stream Designer, the visual builder for
streaming data pipelines
Easy. Extensible. Extra-fast.
● Visual UI: Visual canvas provides
point-and-click interface to build
streaming data pipelines in minutes
● Performant: Experience the high
performance of Confluent Cloud
● ksqlDB: Visualize filter, mapping,
aggregate, and joins as pre-built
pipeline building blocks instead of
writing boilerplate code
● Connectors: Leverage 70+ fully
managed connectors and SMTs to
connect your data sources / sinks
14. Stream Governance Advanced
Enterprise-grade governance and data visibility for production workloads
Stream Governance is provided fully managed on Confluent Cloud
Stream Quality
Deliver trusted, high-quality
data streams to the business
Stream Catalog
Increase collaboration and productivity
with self-service data discovery
Stream Lineage
Understand complex data relationships
and uncover more insights
(Schema Registry & Validation)
15. Stream Governance Advanced
Enterprise-grade governance and data visibility for production workloads
Stream Governance is provided fully managed on Confluent Cloud
Stream Quality
Stream Catalog
Stream Lineage
(Schema Registry & Validation)
Point-in-time Lineage Lineage Search
Business Metadata GraphQL API
99.95% SLA 20K Schemas 30 Regions
16. Everywhere
Recent platform updates
Highlighted “Everywhere” updates from the past 12 months
Private Service Connect
for Google Cloud
Supported Regions
Expansion
Confluent Platform
Updates
Global
Schema Linking
One Confluent CLI
Zone Selection for
Private Networking
Spotlight features within today’s product demo
Networking expansion
for Cluster Linking
17. Cluster Linking: Simple geo-replication
New networking options now supported including Transit Gateway and VPC Peered destination clusters.
An easier approach with perfectly mirrored and globally consistent topic replication
Global Data
Sharing
Regional Disaster
Recovery
Cluster
Migrations
Cost effective, secure, and performant
data transport layer across public
clouds and regions.
Near real-time DR with failover to
perfectly mirrored Kafka topics on a
separate region or different cloud
provider altogether.
Prescriptive, low-downtime migrations
between cloud providers, across
regions, or from another Kafka service
to Confluent Cloud.
19. Demo Scenario
Everything shown during today’s demo will be made available for download within a GitHub repository
Data Engineer
Develop a new streaming pipeline which
combines in-store and online activity to better
personalize every customer experience
● Stream Designer will be used to build the
pipeline incorporating user clickstreams
with in-store shopping activity
● Prepared data should be setup with
real-time replication out to global clusters
● Timing is tight, so any data existing on the
platform for reuse should be leveraged
AcmeCorp
Tenure: ~3 months
20. Start building your own pipeline today
Download today’s demo
repo from GitHub
bit.ly/41oIgCc
22. All topics discussed here are confidential and protected by
NDA. Confidential information may include, but shall not be
limited to, sales or marketing data or plans, product or service
information or roadmaps and/or our requirements or
preferences.
The non-disclosure agreement is not designed for and will not
cover the exchange of confidential information in support of
joint development or invention disclosure activities.
No photos please.
NON-DISCLOSURE REMINDER
23. As our roadmap may change in the future, the features
referred to herein may change, may not be delivered on time or
may not be delivered at all.
This information is not a commitment to deliver any
functionality and customers should make their purchasing
decisions based upon features that are currently available.
24. Increasing the value of real-time data streams
Roadmap highlights
Expanded
In-flight
Processing
Apache Flink®
Strict Data
Contracts
Data Policies
3rd Party
Integrations
Simple Data
Sharing
Stream Sharing
Stream Designer
Updates
More
Flexible
Integrations
Bring Your Own
Connector
More Cost-effective Infrastructure Enterprise SKU
25. Expanding in-flight processing capabilities
Flink Kafka Streams ksqlDB
Fully managed on Confluent Cloud
Self-managed on Confluent Cloud
and Confluent Platform
Fully managed on Confluent Cloud,
Self-managed on Confluent Platform
SQL at GA, Java and Python to follow Java SQL
Future Available Now
26. Our fully managed Flink service will be simple,
secure, and scalable
Simple Secure Scalable
• Fully integrated with
Confluent’s enterprise-
grade identity and access
management tooling
• Fully integrated with
Stream Governance to
ensure high data quality
and policies are enforced
• Audit logging always
enabled for compliance
• Immediate, automatic,
online scaling for elastic
scalability
• Cloud-native separation of
compute, metadata, state,
and storage
• Seamless management at
scale via Terraform
provider
• Serverless endpoint
offering a unified API for
batch and streaming
• Comprehensive standard
SQL interface with JDBC
driver for data querying
• Tiered Kafka-backed tables
to provide enhanced
performance, data
durability, and re-use
27. Confluent Cloud offers the largest portfolio of fully
managed connectors in the market
70+
fully
managed
connectors
Amazon S3
Amazon Redshift
Amazon
DynamoDB
Google Cloud
Spanner
AWS Lambda
Amazon SQS
Amazon Kinesis
Azure Service Bus
Azure Event Hubs
Azure Synapse
Analytics
Azure Blob
Storage
Azure Functions Azure Data Lake
Google
BigTable
Access the Confluent Cloud connectors pricing page to see the full list of available fully managed connectors 27
What’s Next
Bring Your Own
Connector
BYOC enables customers
to leverage their own
connectors with Confluent
managing the underlying
Connect workers
Single Message Transforms Data Output Preview Connect Log Events
Roadmap
29. Metadata Integrations
Expand data visibility and productivity with 1st class catalog integrations
Data Streams Catalog
Kafka Cluster
ksqlDB
Stream & Table
Connector
Schema Topic
…
Technical metadata (automated)
User-defined metadata
Management, search, & discovery
Confluent
Cloud UI
APIs: REST
and GraphQL
Tags Descriptions Business
Metadata
Roadmap
Real-time
metadata syncing
Microsoft Purview
GA Functionality
30. Start and scale security with ease
Client-side field
encryption and access
control Schema
Registry
Producer
Kafka
serializer
Kafka
deserializer
Consumer
Kafka
5. Produce
message with
encrypted fields
8. Consume
message with
encrypted fields
6. Get
schema &
policies
7. Return
schema &
policies
Invalid
message
2. Get
schema
& policies
3. Return
schema &
policies
KMS
4. Use master key(s) to
encrypt fields
9. Use master key(s) to
decrypt fields
1. Schema
& Policy
defined
Protect sensitive data from any other
third parties (incl. Confluent)
Selectively grant access to sensitive
data by specific individual(s)
Meet Infosec / compliance
requirements quickly
Tag-based encryption allows you to simply
tag a field, define the policy, and begin
streaming encrypted data securely
31. ● Easy collaboration on live data with
partners, customers, and vendors
✓ One-click sharing
✓ Trusted and governed
✓ Secure, granular, auditable
● Single, org-wide portal to discover data
streams from trusted sources
Stream Sharing
Simplify inter-enterprise and B2B data
sharing
Available Now:
In Early Access
What’s Next:
GA
Roadmap
32. Stream Designer: Go beyond expertise silos with a
low-code solution for building data pipelines
● Round trip: Changes made in the UI
are reflected in SQL and vice versa
● Future-proof: Easily add or replace
data systems or edit stream
processing logic without needing to
rebuild pipelines
● Troubleshoot: View built-in alerts
and pipeline validations for
end-to-end testing
● Security: RBAC at the pipeline
(Stream Designer) and component
(e.g., Kafka, Connect, ksqlDB) levels
<SQL>
Stream Catalog Integration
Dedicated Cluster Support
Roadmap
Available Now
33. More cost-effective Infrastructure
Re-invest in net-new projects when saving with the Enterprise cluster type
Basic
Get started with scale
from $0 pricing
For experimentation,
early development,
and basic use cases.
Standard
Production-ready for
most applications
For production-
ready use cases
leveraging an
extended feature set
and elastic scaling
(up to 250/750MBps)
Dedicated
Customizable for any
application
For massive-
throughput use
cases at GBps+ scale
that require private
networking
Enterprise
For use cases with
stringent security and
networking requirements
● Scale from
0-250/750MBps
(ingress/egress) in place
automatically
● Private network
connectivity: Private Link
/ Private Services
Connect available
● 99.99% Uptime SLA
● Additional premium
security features
35. Deploy how and where you want
35
Confluent
AWS
MARKETPLACE
Azure
MARKETPLACE
Google Cloud
MARKETPLACE
Start your multi-cloud
journey
Simplify procurement with easy click-though through annual commitments and draw
down against existing Enterprise Agreements in your cloud marketplace of choice
Options for getting started with Confluent Cloud
36. Scalable pricing fit for any budget
Start streaming on Confluent Cloud for free and gain industry-leading cost savings
Get started quickly with
on-demand cluster
provisioning and
consumption-based billing
New signups receive $400
to spend during their first 30
days—no credit card
required
Access scaled discounts for
production or high-volume
workloads with committed
usage
37. Why Confluent is the world’s most trusted
data streaming platform
Focus & Expertise
Only company focused on data in motion:
● Founded in 2014 by the Original Creators of
Apache Kafka
● Over 80% of Kafka commits are by
Confluent employees
● Advised on thousands of real-world Kafka
deployments across a wide range of
patterns & industries
Building and supporting a world class product:
● >9 million engineering hours spent
building building our product
● We internally manage >30,000 clusters
(and counting) in Confluent Cloud
● Over 1 million cumulative hours of Kafka
expertise within Confluent support &
services
Execution at Scale
38. Start your free trial of
Confluent Cloud & get $400 in
credits
Get started with Confluent Cloud is easy!
confluent.io/confluent-cloud/tryfree/
Request a personalized TCO
assessment for your current
Kafka needs or future Data
Streaming Platform
39. Our partners include global
and regional industry-
leading consulting, services
and technology companies.
Regardless of your size or
industry, our partners can
help you buy, build,
implement, service, support
and run Confluent Platform.
Confluent
Partners
Ensure Your
Success
Featured partners: