SlideShare ist ein Scribd-Unternehmen logo
1 von 29
Downloaden Sie, um offline zu lesen
Hybrid Streaming Analytics
for Apache Kafka Users
Cody Irwin
Solution Manager, Smart
Analytics
Google Cloud
Josh Treichel
Sr. Manager, Partner
Solutions
Confluent
Jeff Ferguson
Global Google Cloud
Alliance Lead
Confluent
Agenda:
1) Reference Architecture (5 min - Jeff Ferguson, Confluent)
2) Stream Analytics on GCP (10 min - Cody Irwin, Google Cloud)
3) Bridging your Kafka Deployment to GCP (10 min - Josh Treichel, Confluent)
➢ Customer Architectures and Stories
➢ Demo, Quickstart Guide, & Free Trial
Apache Kafka is an Event Streaming Platform
Hadoop ...
Device
Logs ... App ...MicroserviceMainframes
GCP Data
Services Splunk ...
Data Stores Logs 3rd Party Apps Custom Apps / Microservices
Same Day
Transactioning
(Account Open)
Fees Charges
& Billing
Real-time
Customer 360
Machine
Learning
Models
Real-time Data
Transformation
Real-time Fraud
Detection
(1) App modernization
(2) Data modernization
(3) Bidirectional hybrid
On-premises or Other Cloud
Hybrid Kafka Reference Architecture
Dataflow
BigQuery
Cloud
Storage
Data
Studio
Cloud
Functions
AI
Platform
Bigtable
Confluent Replicator
KSQL
MySQL HDFS Teradata,
Netezza
Mainframe
Business is transforming
Businesses have to anticipate and
act on risks and opportunities
faster than ever before
The data and events needed for
analysis are increasing in velocity,
volume, and type
Companies that are able to quickly identify and capitalize on insights within this
changing landscape have a strategic advantage.
Why Enterprises
choose Google Cloud
for Streaming Analytics
Serverless Architecture
Robust ingestion services
Unified batch and stream processing
Comprehensive set of analysis tools
Flexibility for users
Serverless data analytics
From infrastructure to platform for insights
Performance tuning
Monitoring
Reliability
Deployment &
configuration
Utilization
improvements
The traditional data analytics platform
Analysis and insights
Resource provisioning
Handling growing scale
Analysis and
insights
The serverless data
analytics model
Right-time Action
Dashboard
Visualize and share anomalous events in
your data.
Alerts
Manage by exception through
condition-based notifications.
Actions
Automatically trigger workflows in other
systems using conditions.
1
2
3
Looker
Blocks
Comprehensive set of analysis tools
BigQuery
Cloud Data
Warehouse
Easy setup
Directly integrated with
streaming Dataflow and
Confluent Cloud
Real time
Fast insights and action
powered by BigQuery’s
Streaming API
Intelligent
Built-in ML for
out-of-the-box predictive
insights
Cloud AI
Platform
AI & ML Tools
Plug-and-play
Easily experiment and
collaborate with Google’s
AI Hub
Building blocks
Tools for sight, language,
conversation, and
structured data
Fast deployment
Code-based AI platform
quickly moves ML ideas
to deployment
Tensorflow
Extended (TFX)
Improve the customer experience with Real-time AI
TFX uses Dataflow and Apache Beam as the distributed data processing engine to enable several
aspects of the ML life cycle, all supported with CI/CD for ML through Kubeflow pipelines.
Predictive
Analytics
Fraud
Detection
Real-time
Personalization
More!
AI for every level of expertise
APIs
Pre-trained Models
Infrastructure
AI Foundation
Building
Blocks
Platform
AI Platform
Development
Environment
AutoML
Custom
Models
Services & Solutions
Ease of Implementation
Structured DataSight Language Conversation
ServicesSolutions Collaboration
Structured DataSight Language
Integrated withBuilt-in Tools On-prem
FrameworksAccelerators
Document
Understanding AI
Talent
Solution
Contact
Center AI ASL
Professional
Services
Cloud AI
PartnersAI Hub
Video
Intelligence
Vision
Natural
Language
Translation Inference
Recommendations
AI
Speech-
to-Text
Text-to-
Speech
Dialogflow
Enterprise
Vision
Natural
Language Translation TablesVideo
Datasets Training DataprocDataflow Dataprep
Data
StudioBigQueryKubeflowPredictions
Data
Labeling
New
Pre-built
Algorithms
New
Notebook
New
VM
Images
New
GPUTPU CPU
Flexibility for users
Apache Beam
Open-source,
unified model and
set of SDKs for
defining and
executing data
processing
Open source programming
model
Serves as the SDK for
creating Cloud Dataflow jobs;
community development
increases flexibility
Choose your language
Java, Python, Scala, and GO are available;
join DA Spotlight
for news on languages
Portability
Program in Beam, and gain the ability to
move between
Spark, Flink, Dataflow, and more
Dataflow
Simplified stream and
batch data processing
Batch and Stream
Reduce complexity and reuse code by
driving batch and stream workloads
from the same tool
Reliable and consistent processing
Exactly once processing with built-in
support for fault-tolerant execution
Simplified operations & management
Performance, scaling, availability,
security, and compliance
handled automatically
Integrated
Integration with Kafka/Confluent Cloud,
the Google Data Analytics suite,
and GCP broadly
Unified stream and batch
processing
Ingest Transform Analyze
Ingest and distribute
data reliably
Fast, correct computations
quickly and simply
Machine learning &
data warehouse
Cloud Dataflow
Cloud MLPub/Sub BigQueryDataflow
Flexible stream analytics with OSS
KSQL
Title Safe >
< Action Safe
The Business Case
Architectural Approach Business Solutions
FSI | Fraud Analytics, Trade Data Capture1
Retail I Recommendations, Inventory
Management, POS Processing2
Manufacturing | Anomaly Detection,
Edge-to-Cloud ML3
General | Real-time Clickstream. CDC4
Many more to come!∞
On-premises or Other Cloud
Kafka as the Real-Time Bridge Simplifies Cloud Migration
Dataflow
BigQuery
Cloud
Storage
Data
Studio
Cloud
Functions
AI
Platform
Bigtable
Confluent Replicator
KSQL
MySQL HDFS Teradata,
Netezza
Mainframe
Confluent Replicator Architecture
Kafka Broker
test-topic
Kafka Broker
test-topic
Replicator
consumer producer
eventsevents
Origin Destination
Make clusters globally available
Replicate clusters or a subset of
topics across any distance
Aggregate or migrate clusters anywhere
Aggregate many clusters together or
migrate entire clusters to a preferred
environment
Bridge self-managed clusters to
a fully managed Kafka service
Enable hybrid-cloud deployments
with Confluent Cloud
Schema Registry
Make data backwards
compatible and future-proof
KSQL
Develop real-time stream
processing apps writing only
SQL
Connectors
Easily send data to cloud
storage with BigQuery, GCS +
more
Confluent Cloud - Fully Managed Kafka and Much more!
! Schema
Registry
Kafk
a
topic
!
Serial
izer
Serial
izer
Kafka enables Unity’s massive GCP migration
Unity Monetization Platform &
Gaming Dev Platform
Confluent
Connector
Dataflow
Cloud
Storage
Confluent Replicator
BigQuery
Other Cloud
Unlock advanced AI/ML on GCP using data on prem
On-premise or Other
Cloud
Web
IoT
Mobile
Data
Store
Dataflow BigQueryEvents
Confluent Replicator
KSQL
Train Fraud Models
Deploy
Models
Tenso low Fraud
ApplicationCurated Data
Streams
Full Data
Stream
Fraud
ApplicationFraud
ApplicationsFraud
Applications
Fraud App Consumption and Production
Mainframe, Hadoop,
Oracle
On-premises
FinServ Fraud Analytics - On-prem to GCP
Confluent
Connector
Cloud
Dataproc
Cloud
Dataflow
BigQuery
Cloud
Storage
Cloud
Bigtable
Cloud Machine
Learning Engine
Confluent Replicator
Give it a try
https://docs.confluent.io/current/tutorials
/examples/kubernetes/replicator-gke-cc/
docs/index.html
Launch it form the GCP console.
Confluent Cloud $200/month for free for
3 months
Thank You!
Title Safe >
< Action Safe
On-premises or Other
Cloud
What does a hybrid Kafka architecture look like on GCP?
Web
IoT
Mobile
Data
Store
Dataflow
BigQuery
Cloud
Storage
1
Trigger &
Send
2
Ingest &
Prepare
3
Transform &
Enrich
4
Store &
Analyze
5
Share &
Activate
Data
Studio
Cloud
Functions
AI
Platform
Bigtable
Events
Confluent Replicator
Unlock the value of
event streaming
Title Safe >
< Action Safe
Unity leveraged Confluent Hybrid Kafka platform to build a massive data
infrastructure and migrated from AWS to GCP. This infrastructure is powering
Unity Gaming Dev Platform and Monetization Network, scaling to process
Million events per second with zero outages.Gaming & Media
Solution
Confluent was chosen for better
control, enterprise scale, Kafka
innovation, and guidance on Kafka
architecture and best practices.
Challenge
Bring together, unify, and modernize
all the different data pipelines and
technology stacks running in each
department of the company as well
as migrate from AWS to GCP.
“As a small team we have large responsibilities
that include managing the data infrastructure
that underpins the Unity platform and helping
make Unity a data-driven company. That’s
one of the reasons we built our data
infrastructure on Confluent Platform and
Apache Kafka. Today, this infrastructure
handles on average about a half million events
per second, with peaks of about a million
events per second. It also reliably handles
millions of dollars of monetary transactions.
In fact, since we went live with Confluent
Platform and Kafka a year ago we have had
zero outages that resulted in money loss.”
Oguz Kayral, Engineering Manager, Data
Platform, Unity
Results
● Completed a massive migration with
PetaBytes of data from AWS to GCP
● Scaled to handle Million events per
second and reliably handle Millions
of dollars of monetary transactions
with zero outages
● Well-proven data infrastructure
based on Confluent Platform & GCP
Dataflow, BigQuery analytics has
opened a lot of new possibilities for
product teams across Unity.
Unity-Confluent blog / Unity-Google blog
Title Safe >
< Action Safe
Other Solution Opportunities
1) App modernization with Event-driven Microservices
(with Anthos option)
2) Data Lake & Data Warehouse modernization (Mainframe,
Oracle, Hadoop, Teradata, )
3) IoT (Manufacturing, Utilities, Smart Cars, etc)
Other Use Cases
Confluent Cloud provides other opportunities to help customers in unique
ways like...
Reference Architecture
Ingest Pipelines
Storage
Analytics
Application &
Presentation
App
Engine
Kubernetes
Engine
Cloud
Storage
Cloud
Dataflow
Cloud
Dataflow
Cloud
Datastore
Cloud
Bigtable
BigQuery
Cloud
Dataproc
Cloud
Datalab
Compute
Engine
colo / dc / on-premises / other cloud

Weitere ähnliche Inhalte

Was ist angesagt?

Elastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using ConfluentElastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using Confluentconfluent
 
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...HostedbyConfluent
 
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent
 
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDBData Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDBconfluent
 
Real time data processing and model inferncing platform with Kafka streams (N...
Real time data processing and model inferncing platform with Kafka streams (N...Real time data processing and model inferncing platform with Kafka streams (N...
Real time data processing and model inferncing platform with Kafka streams (N...KafkaZone
 
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...confluent
 
A guide through the Azure Messaging services - Update Conference
A guide through the Azure Messaging services - Update ConferenceA guide through the Azure Messaging services - Update Conference
A guide through the Azure Messaging services - Update ConferenceEldert Grootenboer
 
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...HostedbyConfluent
 
Operational Analytics on Event Streams in Kafka
Operational Analytics on Event Streams in KafkaOperational Analytics on Event Streams in Kafka
Operational Analytics on Event Streams in Kafkaconfluent
 
MongoDB .local London 2019: Streaming Data on the Shoulders of Giants
MongoDB .local London 2019: Streaming Data on the Shoulders of GiantsMongoDB .local London 2019: Streaming Data on the Shoulders of Giants
MongoDB .local London 2019: Streaming Data on the Shoulders of GiantsLisa Roth, PMP
 
KSQL: Open Source Streaming for Apache Kafka
KSQL: Open Source Streaming for Apache KafkaKSQL: Open Source Streaming for Apache Kafka
KSQL: Open Source Streaming for Apache Kafkaconfluent
 
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...HostedbyConfluent
 
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud ServicesBuild a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Servicesconfluent
 
GCP for Apache Kafka® Users: Stream Ingestion and Processing
GCP for Apache Kafka® Users: Stream Ingestion and ProcessingGCP for Apache Kafka® Users: Stream Ingestion and Processing
GCP for Apache Kafka® Users: Stream Ingestion and Processingconfluent
 
Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...
Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...
Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...confluent
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Kai Wähner
 
Enterprise Metadata Integration
Enterprise Metadata IntegrationEnterprise Metadata Integration
Enterprise Metadata IntegrationDr. Mirko Kämpf
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaKai Wähner
 
Bridge to Cloud: Using Apache Kafka to Migrate to GCP
Bridge to Cloud: Using Apache Kafka to Migrate to GCPBridge to Cloud: Using Apache Kafka to Migrate to GCP
Bridge to Cloud: Using Apache Kafka to Migrate to GCPconfluent
 
Real time analytics in Azure IoT
Real time analytics in Azure IoT Real time analytics in Azure IoT
Real time analytics in Azure IoT Sam Vanhoutte
 

Was ist angesagt? (20)

Elastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using ConfluentElastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using Confluent
 
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
 
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
 
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDBData Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDB
 
Real time data processing and model inferncing platform with Kafka streams (N...
Real time data processing and model inferncing platform with Kafka streams (N...Real time data processing and model inferncing platform with Kafka streams (N...
Real time data processing and model inferncing platform with Kafka streams (N...
 
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
 
A guide through the Azure Messaging services - Update Conference
A guide through the Azure Messaging services - Update ConferenceA guide through the Azure Messaging services - Update Conference
A guide through the Azure Messaging services - Update Conference
 
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
 
Operational Analytics on Event Streams in Kafka
Operational Analytics on Event Streams in KafkaOperational Analytics on Event Streams in Kafka
Operational Analytics on Event Streams in Kafka
 
MongoDB .local London 2019: Streaming Data on the Shoulders of Giants
MongoDB .local London 2019: Streaming Data on the Shoulders of GiantsMongoDB .local London 2019: Streaming Data on the Shoulders of Giants
MongoDB .local London 2019: Streaming Data on the Shoulders of Giants
 
KSQL: Open Source Streaming for Apache Kafka
KSQL: Open Source Streaming for Apache KafkaKSQL: Open Source Streaming for Apache Kafka
KSQL: Open Source Streaming for Apache Kafka
 
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
 
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud ServicesBuild a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
 
GCP for Apache Kafka® Users: Stream Ingestion and Processing
GCP for Apache Kafka® Users: Stream Ingestion and ProcessingGCP for Apache Kafka® Users: Stream Ingestion and Processing
GCP for Apache Kafka® Users: Stream Ingestion and Processing
 
Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...
Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...
Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
 
Enterprise Metadata Integration
Enterprise Metadata IntegrationEnterprise Metadata Integration
Enterprise Metadata Integration
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache Kafka
 
Bridge to Cloud: Using Apache Kafka to Migrate to GCP
Bridge to Cloud: Using Apache Kafka to Migrate to GCPBridge to Cloud: Using Apache Kafka to Migrate to GCP
Bridge to Cloud: Using Apache Kafka to Migrate to GCP
 
Real time analytics in Azure IoT
Real time analytics in Azure IoT Real time analytics in Azure IoT
Real time analytics in Azure IoT
 

Ähnlich wie Hybrid Streaming Analytics for Apache Kafka Users on Google Cloud Platform

Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, GoogleHybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, GoogleHostedbyConfluent
 
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, GoogleHybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, GoogleHostedbyConfluent
 
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...HostedbyConfluent
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flinkconfluent
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
 
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdfDIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdfconfluent
 
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it YourselfWhy Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it YourselfDATAVERSITY
 
Apache Kafka® + Machine Learning for Supply Chain 
Apache Kafka® + Machine Learning for Supply Chain Apache Kafka® + Machine Learning for Supply Chain 
Apache Kafka® + Machine Learning for Supply Chain confluent
 
IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...
IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...
IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...Kai Wähner
 
Unlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptxUnlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptxAhmed791434
 
Data Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEAData Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEAAndrew Morgan
 
Webinar: Data Streaming with Apache Kafka & MongoDB
Webinar: Data Streaming with Apache Kafka & MongoDBWebinar: Data Streaming with Apache Kafka & MongoDB
Webinar: Data Streaming with Apache Kafka & MongoDBMongoDB
 
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans JespersenBest Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersenconfluent
 
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...confluent
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...confluent
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023confluent
 
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...HostedbyConfluent
 
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...HostedbyConfluent
 
Reinventing Kafka in the Data Streaming Era - Jun Rao
Reinventing Kafka in the Data Streaming Era - Jun RaoReinventing Kafka in the Data Streaming Era - Jun Rao
Reinventing Kafka in the Data Streaming Era - Jun Raoconfluent
 
Unleashing Apache Kafka and TensorFlow in the Cloud

Unleashing Apache Kafka and TensorFlow in the Cloud
Unleashing Apache Kafka and TensorFlow in the Cloud

Unleashing Apache Kafka and TensorFlow in the Cloud
Kai Wähner
 

Ähnlich wie Hybrid Streaming Analytics for Apache Kafka Users on Google Cloud Platform (20)

Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, GoogleHybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
 
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, GoogleHybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
Hybrid Streaming Analytics for Apache Kafka Users | Firat Tekiner, Google
 
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...
Navigating Your Data Landscape With Siddharth Desai and Elena Cuevas | Curren...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice Architectures
 
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdfDIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
 
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it YourselfWhy Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself
 
Apache Kafka® + Machine Learning for Supply Chain 
Apache Kafka® + Machine Learning for Supply Chain Apache Kafka® + Machine Learning for Supply Chain 
Apache Kafka® + Machine Learning for Supply Chain 
 
IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...
IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...
IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Ti...
 
Unlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptxUnlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptx
 
Data Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEAData Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEA
 
Webinar: Data Streaming with Apache Kafka & MongoDB
Webinar: Data Streaming with Apache Kafka & MongoDBWebinar: Data Streaming with Apache Kafka & MongoDB
Webinar: Data Streaming with Apache Kafka & MongoDB
 
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans JespersenBest Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
 
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
 
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
Qlik and Confluent Success Stories with Kafka - How Generali and Skechers Kee...
 
Reinventing Kafka in the Data Streaming Era - Jun Rao
Reinventing Kafka in the Data Streaming Era - Jun RaoReinventing Kafka in the Data Streaming Era - Jun Rao
Reinventing Kafka in the Data Streaming Era - Jun Rao
 
Unleashing Apache Kafka and TensorFlow in the Cloud

Unleashing Apache Kafka and TensorFlow in the Cloud
Unleashing Apache Kafka and TensorFlow in the Cloud

Unleashing Apache Kafka and TensorFlow in the Cloud

 

Mehr von HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonRenaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolEvolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesEnsuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent
 
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonFish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent
 
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonTiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent
 
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondHostedbyConfluent
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent
 

Mehr von HostedbyConfluent (20)

Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonRenaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit London
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolEvolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesEnsuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
 
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonFish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit London
 
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonTiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit London
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
 
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit London
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSL
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
 

Kürzlich hochgeladen

SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxBkGupta21
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demoHarshalMandlekar2
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 

Kürzlich hochgeladen (20)

SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptx
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Sample pptx for embedding into website for demo
Sample pptx for embedding into website for demoSample pptx for embedding into website for demo
Sample pptx for embedding into website for demo
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 

Hybrid Streaming Analytics for Apache Kafka Users on Google Cloud Platform

  • 1. Hybrid Streaming Analytics for Apache Kafka Users
  • 2. Cody Irwin Solution Manager, Smart Analytics Google Cloud Josh Treichel Sr. Manager, Partner Solutions Confluent Jeff Ferguson Global Google Cloud Alliance Lead Confluent
  • 3. Agenda: 1) Reference Architecture (5 min - Jeff Ferguson, Confluent) 2) Stream Analytics on GCP (10 min - Cody Irwin, Google Cloud) 3) Bridging your Kafka Deployment to GCP (10 min - Josh Treichel, Confluent) ➢ Customer Architectures and Stories ➢ Demo, Quickstart Guide, & Free Trial
  • 4. Apache Kafka is an Event Streaming Platform Hadoop ... Device Logs ... App ...MicroserviceMainframes GCP Data Services Splunk ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Same Day Transactioning (Account Open) Fees Charges & Billing Real-time Customer 360 Machine Learning Models Real-time Data Transformation Real-time Fraud Detection (1) App modernization (2) Data modernization (3) Bidirectional hybrid
  • 5. On-premises or Other Cloud Hybrid Kafka Reference Architecture Dataflow BigQuery Cloud Storage Data Studio Cloud Functions AI Platform Bigtable Confluent Replicator KSQL MySQL HDFS Teradata, Netezza Mainframe
  • 6. Business is transforming Businesses have to anticipate and act on risks and opportunities faster than ever before The data and events needed for analysis are increasing in velocity, volume, and type Companies that are able to quickly identify and capitalize on insights within this changing landscape have a strategic advantage.
  • 7. Why Enterprises choose Google Cloud for Streaming Analytics Serverless Architecture Robust ingestion services Unified batch and stream processing Comprehensive set of analysis tools Flexibility for users
  • 8. Serverless data analytics From infrastructure to platform for insights Performance tuning Monitoring Reliability Deployment & configuration Utilization improvements The traditional data analytics platform Analysis and insights Resource provisioning Handling growing scale Analysis and insights The serverless data analytics model
  • 9. Right-time Action Dashboard Visualize and share anomalous events in your data. Alerts Manage by exception through condition-based notifications. Actions Automatically trigger workflows in other systems using conditions. 1 2 3 Looker Blocks
  • 10. Comprehensive set of analysis tools BigQuery Cloud Data Warehouse Easy setup Directly integrated with streaming Dataflow and Confluent Cloud Real time Fast insights and action powered by BigQuery’s Streaming API Intelligent Built-in ML for out-of-the-box predictive insights Cloud AI Platform AI & ML Tools Plug-and-play Easily experiment and collaborate with Google’s AI Hub Building blocks Tools for sight, language, conversation, and structured data Fast deployment Code-based AI platform quickly moves ML ideas to deployment Tensorflow Extended (TFX)
  • 11. Improve the customer experience with Real-time AI TFX uses Dataflow and Apache Beam as the distributed data processing engine to enable several aspects of the ML life cycle, all supported with CI/CD for ML through Kubeflow pipelines. Predictive Analytics Fraud Detection Real-time Personalization More!
  • 12. AI for every level of expertise APIs Pre-trained Models Infrastructure AI Foundation Building Blocks Platform AI Platform Development Environment AutoML Custom Models Services & Solutions Ease of Implementation Structured DataSight Language Conversation ServicesSolutions Collaboration Structured DataSight Language Integrated withBuilt-in Tools On-prem FrameworksAccelerators Document Understanding AI Talent Solution Contact Center AI ASL Professional Services Cloud AI PartnersAI Hub Video Intelligence Vision Natural Language Translation Inference Recommendations AI Speech- to-Text Text-to- Speech Dialogflow Enterprise Vision Natural Language Translation TablesVideo Datasets Training DataprocDataflow Dataprep Data StudioBigQueryKubeflowPredictions Data Labeling New Pre-built Algorithms New Notebook New VM Images New GPUTPU CPU
  • 13. Flexibility for users Apache Beam Open-source, unified model and set of SDKs for defining and executing data processing Open source programming model Serves as the SDK for creating Cloud Dataflow jobs; community development increases flexibility Choose your language Java, Python, Scala, and GO are available; join DA Spotlight for news on languages Portability Program in Beam, and gain the ability to move between Spark, Flink, Dataflow, and more Dataflow Simplified stream and batch data processing Batch and Stream Reduce complexity and reuse code by driving batch and stream workloads from the same tool Reliable and consistent processing Exactly once processing with built-in support for fault-tolerant execution Simplified operations & management Performance, scaling, availability, security, and compliance handled automatically Integrated Integration with Kafka/Confluent Cloud, the Google Data Analytics suite, and GCP broadly Unified stream and batch processing
  • 14. Ingest Transform Analyze Ingest and distribute data reliably Fast, correct computations quickly and simply Machine learning & data warehouse Cloud Dataflow Cloud MLPub/Sub BigQueryDataflow Flexible stream analytics with OSS KSQL
  • 15. Title Safe > < Action Safe The Business Case Architectural Approach Business Solutions FSI | Fraud Analytics, Trade Data Capture1 Retail I Recommendations, Inventory Management, POS Processing2 Manufacturing | Anomaly Detection, Edge-to-Cloud ML3 General | Real-time Clickstream. CDC4 Many more to come!∞
  • 16. On-premises or Other Cloud Kafka as the Real-Time Bridge Simplifies Cloud Migration Dataflow BigQuery Cloud Storage Data Studio Cloud Functions AI Platform Bigtable Confluent Replicator KSQL MySQL HDFS Teradata, Netezza Mainframe
  • 17. Confluent Replicator Architecture Kafka Broker test-topic Kafka Broker test-topic Replicator consumer producer eventsevents Origin Destination Make clusters globally available Replicate clusters or a subset of topics across any distance Aggregate or migrate clusters anywhere Aggregate many clusters together or migrate entire clusters to a preferred environment Bridge self-managed clusters to a fully managed Kafka service Enable hybrid-cloud deployments with Confluent Cloud
  • 18. Schema Registry Make data backwards compatible and future-proof KSQL Develop real-time stream processing apps writing only SQL Connectors Easily send data to cloud storage with BigQuery, GCS + more Confluent Cloud - Fully Managed Kafka and Much more! ! Schema Registry Kafk a topic ! Serial izer Serial izer
  • 19. Kafka enables Unity’s massive GCP migration Unity Monetization Platform & Gaming Dev Platform Confluent Connector Dataflow Cloud Storage Confluent Replicator BigQuery Other Cloud
  • 20. Unlock advanced AI/ML on GCP using data on prem On-premise or Other Cloud Web IoT Mobile Data Store Dataflow BigQueryEvents Confluent Replicator KSQL Train Fraud Models Deploy Models Tenso low Fraud ApplicationCurated Data Streams Full Data Stream Fraud ApplicationFraud ApplicationsFraud Applications Fraud App Consumption and Production Mainframe, Hadoop, Oracle
  • 21. On-premises FinServ Fraud Analytics - On-prem to GCP Confluent Connector Cloud Dataproc Cloud Dataflow BigQuery Cloud Storage Cloud Bigtable Cloud Machine Learning Engine Confluent Replicator
  • 22. Give it a try https://docs.confluent.io/current/tutorials /examples/kubernetes/replicator-gke-cc/ docs/index.html Launch it form the GCP console. Confluent Cloud $200/month for free for 3 months
  • 24. Title Safe > < Action Safe On-premises or Other Cloud What does a hybrid Kafka architecture look like on GCP? Web IoT Mobile Data Store Dataflow BigQuery Cloud Storage 1 Trigger & Send 2 Ingest & Prepare 3 Transform & Enrich 4 Store & Analyze 5 Share & Activate Data Studio Cloud Functions AI Platform Bigtable Events Confluent Replicator
  • 25. Unlock the value of event streaming
  • 26. Title Safe > < Action Safe Unity leveraged Confluent Hybrid Kafka platform to build a massive data infrastructure and migrated from AWS to GCP. This infrastructure is powering Unity Gaming Dev Platform and Monetization Network, scaling to process Million events per second with zero outages.Gaming & Media Solution Confluent was chosen for better control, enterprise scale, Kafka innovation, and guidance on Kafka architecture and best practices. Challenge Bring together, unify, and modernize all the different data pipelines and technology stacks running in each department of the company as well as migrate from AWS to GCP. “As a small team we have large responsibilities that include managing the data infrastructure that underpins the Unity platform and helping make Unity a data-driven company. That’s one of the reasons we built our data infrastructure on Confluent Platform and Apache Kafka. Today, this infrastructure handles on average about a half million events per second, with peaks of about a million events per second. It also reliably handles millions of dollars of monetary transactions. In fact, since we went live with Confluent Platform and Kafka a year ago we have had zero outages that resulted in money loss.” Oguz Kayral, Engineering Manager, Data Platform, Unity Results ● Completed a massive migration with PetaBytes of data from AWS to GCP ● Scaled to handle Million events per second and reliably handle Millions of dollars of monetary transactions with zero outages ● Well-proven data infrastructure based on Confluent Platform & GCP Dataflow, BigQuery analytics has opened a lot of new possibilities for product teams across Unity. Unity-Confluent blog / Unity-Google blog
  • 27. Title Safe > < Action Safe Other Solution Opportunities 1) App modernization with Event-driven Microservices (with Anthos option) 2) Data Lake & Data Warehouse modernization (Mainframe, Oracle, Hadoop, Teradata, ) 3) IoT (Manufacturing, Utilities, Smart Cars, etc)
  • 28. Other Use Cases Confluent Cloud provides other opportunities to help customers in unique ways like...
  • 29. Reference Architecture Ingest Pipelines Storage Analytics Application & Presentation App Engine Kubernetes Engine Cloud Storage Cloud Dataflow Cloud Dataflow Cloud Datastore Cloud Bigtable BigQuery Cloud Dataproc Cloud Datalab Compute Engine colo / dc / on-premises / other cloud