More Related Content Similar to Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector | Denis Rosa, Couchbase (20) More from HostedbyConfluent (20) Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector | Denis Rosa, Couchbase1. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
Building Scalable & Real-Time Data
Pipelines with Couchbase Kafka Connector
Denis Rosa | Developer Advocate
3. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 3
Kafka
• Data Replication
• Sharding
• Fault Tolerant
• High Performance
• Highly Scalable
Operational
Data
4. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
Why Kafka + Couchbase?
There are some many connectors out there already ...
5. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 5
Why Kafka + Couchbase?
• Sharding
• Data Replication
• Highly Scalable
• High Performance
• Fault Tolerant
• Sharding
• Data Replication
• Highly Scalable
• High Performance
• Fault Tolerant
• Easy to Manage
6. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 6
Auto-Sharding / Replication
Replication
7. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 7
Cross Data Center Replication (XDCR)
8. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 8
Highly Scalable - Couchbase Cluster
9. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 9
Highly Scalable - Couchbase Cluster
16. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 16
Easy to Manage
• Cluster Manager is embedded
• Online Upgrades, Failovers, Scale Up/Down/Out
20. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 20
Couchbase Autonomous Operator Architecture
POD 10
Kubernetes Cluster
API Server Scheduler
Controller Manager Server
Etcd DNS
Couchbase Admission Controller Mutating Configuration Validating Configuration
Couchbase Autonomous Operator
Master
Services Config Maps
Resources
Persistent Volume Claim
Bucket Resources Backup Resources User Resources Replication Resources
POD 1 POD 2 POD 3 POD 4 POD 5 POD 6 POD 7 POD 8 POD 9
PERSISTENT
STORAGE
PERSISTENT STORAGE
PV01 PV02 PV03 PV06 PV07
DATA SERVICE
QUERY SERVICE
-
INDEX SERVICE
PERSISTENT
STORAGE
PV08 PV09
SEARCH SERVICE
PERSISTENT
STORAGE
PV04 PV05
ANALYTICS
SERVICE
POD 10
Pods
EVENTING SERVICE
Controller
Webhook
Cluster Resources Secret
namespace – kube-system
Couchbase Cluster
namespace – user-defined
PERSISTENT
STORAGE – BACKUP
PV
PVPV
namespace – monitoring
Prometheus
Alert Manager
21. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 21
Cloud-Native Database Releases
• Automated Provisioning: Deploy a
self-driving Couchbase cluster with
just one kubectl command
• Auto Recovery: Automatically
recover from node failure and bring
cluster back to capacity
• On-demand Scale: Deliver and
elastic, on-demand Couchbase data
platform that can scale-up or down
dynamically
• Configuration Management: Ability
handle all the Couchbase data
platform configuration through
centralized repository
• Production Certification: Open
source Kubernetes & RedHat
OpenShift
• Automated Upgrade: Push button
online rolling zero downtime upgrade
• Public Connectivity for
Couchbase Clients: Connecting
clients to the Couchbase cluster over
the internet via public IP-based
addressing and dynamic DNS.
• Helm Chart: Official Couchbase
charts make installation simple and
repeatable
• Production Certification Public
Cloud Kubernetes Service: AWS
EKS, Azure AKS and Google GKE
CAO 1.0 2018 CAO 1.x 2019
• Automated Security: Provide Role-
Based Access Control
Authentication and Authorization
• Automated Backup: Backup
scheduling through Operator
• Centralized Monitoring
Prometheus: Monitoring entire
Couchbase deployment with
centralized dashboard
• Automated Cross Datacenter
Replication Management:
Automatically configure and
establish XDCR replication from
source to target Couchbase cluster
• Open Service Broker: Simplified
Service Broker architecture
• Certificate Authentication using
Mutual TLS Support
• Auto-Scaling Stateless: Scale-up
or down automatically based-on
predefined Thresholds
• Usage Metering: Usage tracking for
self-service provisioning and usage
reporting
• Sync Gateway on Kubernetes:
Support provisioning, configuration,
on-demand scaling, and recovery for
Couchbase Sync Gateway
• Support Istio/Envoy: Allows
Operator to work with a Istio Service
Mesh for creating a network of
deployed services
• Cluster Hibernation: Ability to
PAUSE and RESUME Couchbase
Instance without losing the data
CAO 2.0 Q1 2020 CAO 2.1 Q4 2020
Azure AKS
Google GKE
27. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
THANK YOU
denis.rosa@couchbase.com
@deniswsrosa