DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020

•

0 gefällt mir•1,279 views

DataOps challenges us to build data experiences in a repeatable way. For those with Kafka, this means finding a means of deploying flows in an automated and consistent fashion. The challenge is to make the deployment of Kafka flows consistent across different technologies and systems: the topics, the schemas, the monitoring rules, the credentials, the connectors, the stream processing apps. And ideally not coupled to a particular infrastructure stack. In this talk we will discuss the different approaches and benefits/disadvantages to automating the deployment of Kafka flows including Git operators and Kubernetes operators. We will walk through and demo deploying a flow on AWS EKS with MSK and Kafka Connect using GitOps practices: including a stream processing application, S3 connector with credentials held in AWS Secrets Manager.

Technologie

for a Kafka streaming platform
DataOps automation

Presenters
Andrew Stevenson
CTO Lenses.io
Spiros Economakis
DevOps Cloud Lead Lenses.io

Delivering data intensity on Kafka
Lenses.io

What is DataOps & Lenses.io
How to automate your data landscape
GitOps demo
Agenda

Deploying Kafka isn’t interesting any more?

Tech Intensity
Use commodity infrastructure
Don’t reinvent
Focus on building data landscape

If not kafka, what are we automating?
The important bit...
Data Intensity
Tech Intensity

Pushing
Are sure you want to give access?
Who resolves the desired state?
Who applies the desired state?

GitOps Benefits
Focus build reusable apps - config driven - less ops
Reduced attack surface to sensitive environments
Immutable
Auditable
Rollback
Separation of concerns
Use any Git server - Don’t need k8s
Operator enforces multi-tenancy and governance

GitOps for Kafka
Infrastructure deployment is now a commodity - Terraform, Cloudformation, etc.
Tech intensity
Repeatable, rapid deployment of the data landscape
Topics
Schemas
ACLs
Alert rules
Data policies
Applications
Data intensity

LEARN MORE
All-in-one Lenses.io DataOps
workspace & Kafka docker
container
Lenses.io workspace with
Kafka in the cloud
lenses.io/start
launchpass.com/lensesio
https://launchpass.com/lensesio

for a Kafka streaming platform
DataOps automation
Tuesday, August 25 2020
07:30 PDT / 14h30 GMT
events.kafka-summit.org/2020-schedule

Empfohlen

Leveraging Mainframe Data for Modern Analyticsconfluent

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...HostedbyConfluent

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...HostedbyConfluent

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...HostedbyConfluent

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...HostedbyConfluent

Maximize the Business Value of Machine Learning and Data Science with Kafka (...confluent

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...HostedbyConfluent

Empfohlen

Leveraging Mainframe Data for Modern Analyticsconfluent

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...HostedbyConfluent

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...HostedbyConfluent

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...HostedbyConfluent

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...HostedbyConfluent

Maximize the Business Value of Machine Learning and Data Science with Kafka (...confluent

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...HostedbyConfluent

How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020HostedbyConfluent

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...HostedbyConfluent

Keep your Metadata Repository Current with Event-Driven Updates using CDC and...confluent

Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...HostedbyConfluent

Real-Time Analytics with Confluent and MemSQLSingleStore

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...HostedbyConfluent

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020confluent

PCAP Graphs for Cybersecurity and System TuningDr. Mirko Kämpf

user Behavior Analysis with Session Windows and Apache Kafka's Streams APIconfluent

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...HostedbyConfluent

Data Integration with Apache Kafka: What, Why, HowPat Patterson

INTRODUCING: CREATE PIPELINESingleStore

Enterprise Metadata IntegrationDr. Mirko Kämpf

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent

Disrupting Big Data with Apache Spark in the CloudJen Aman

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Implyconfluent

Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamS...Pat Patterson

Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...confluent

Operational Analytics on Event Streams in Kafkaconfluent

All the Ops: DataOps with GitOps for Streaming data on Kafka and KubernetesDevOps.com

[Capitole du Libre] #serverless -  mettez-le en oeuvre dans votre entreprise...Ludovic Piot

Weitere ähnliche Inhalte

Was ist angesagt?

How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020HostedbyConfluent

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...HostedbyConfluent

Keep your Metadata Repository Current with Event-Driven Updates using CDC and...confluent

Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...HostedbyConfluent

Real-Time Analytics with Confluent and MemSQLSingleStore

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...HostedbyConfluent

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020confluent

PCAP Graphs for Cybersecurity and System TuningDr. Mirko Kämpf

user Behavior Analysis with Session Windows and Apache Kafka's Streams APIconfluent

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...HostedbyConfluent

Data Integration with Apache Kafka: What, Why, HowPat Patterson

INTRODUCING: CREATE PIPELINESingleStore

Enterprise Metadata IntegrationDr. Mirko Kämpf

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent

Disrupting Big Data with Apache Spark in the CloudJen Aman

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Implyconfluent

Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamS...Pat Patterson

Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...confluent

Operational Analytics on Event Streams in Kafkaconfluent

Was ist angesagt? (20)

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

Keep your Metadata Repository Current with Event-Driven Updates using CDC and...

Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...

Real-Time Analytics with Confluent and MemSQL

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020

PCAP Graphs for Cybersecurity and System Tuning

user Behavior Analysis with Session Windows and Apache Kafka's Streams API

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Data Integration with Apache Kafka: What, Why, How

INTRODUCING: CREATE PIPELINE

Enterprise Metadata Integration

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...

Disrupting Big Data with Apache Spark in the Cloud

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Imply

Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamS...

Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...

Operational Analytics on Event Streams in Kafka

Ähnlich wie DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020

All the Ops: DataOps with GitOps for Streaming data on Kafka and KubernetesDevOps.com

[Capitole du Libre] #serverless -  mettez-le en oeuvre dans votre entreprise...Ludovic Piot

Building Data Intensity with AWS MSK & Lenses.ioLenses.io

2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructuredevopsdaysaustin

OpenStack Preso: DevOps on Hybrid Infrastructurerhirschfeld

Gluecon Preso: Hybrid Container Infrastructurerhirschfeld

Continuous Lifecycle London 2018 Event KeynoteWeaveworks

Wasp2 - IoT and Streaming PlatformPaolo Platter

Yannis Zarkadas. Enterprise data science workflows on kubeflowMarynaHoldaieva

Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...Lviv Startup Club

Enterprise Application MigrationVMware Tanzu

Episode 4: Operating Kubernetes at Scale with DC/OSMesosphere Inc.

Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersenconfluent

James Watters Kafka Summit NYC 2019 KeynoteJames Watters

Containers and KubernetesAltoros

Netflix Cloud Architecture and Open Sourceaspyker

The Never Landing Stream with HTAP and StreamingTimothy Spann

ATO 2018 - What is Serverless Useful For?BerndtJung

End to end Machine Learning using Kubeflow - Build, Train, Deploy and ManageAnimesh Singh

How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...Lightbend

Ähnlich wie DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020 (20)

All the Ops: DataOps with GitOps for Streaming data on Kafka and Kubernetes

[Capitole du Libre] #serverless -  mettez-le en oeuvre dans votre entreprise...

Building Data Intensity with AWS MSK & Lenses.io

2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructure

OpenStack Preso: DevOps on Hybrid Infrastructure

Gluecon Preso: Hybrid Container Infrastructure

Continuous Lifecycle London 2018 Event Keynote

Wasp2 - IoT and Streaming Platform

Yannis Zarkadas. Enterprise data science workflows on kubeflow

Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...

Enterprise Application Migration

Episode 4: Operating Kubernetes at Scale with DC/OS

Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen

James Watters Kafka Summit NYC 2019 Keynote

Containers and Kubernetes

Netflix Cloud Architecture and Open Source

The Never Landing Stream with HTAP and Streaming

ATO 2018 - What is Serverless Useful For?

End to end Machine Learning using Kubeflow - Build, Train, Deploy and Manage

How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...

Mehr von HostedbyConfluent

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent

TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

Mehr von HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Kürzlich hochgeladen

Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services

🐬 The future of MySQL is Postgres 🐘RTylerCroy

How to convert PDF to text with Nanonetsnaman860154

What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

GenCyber Cyber Security Day PresentationMichael W. Hawkins

A Domino Admins Adventures (Engage 2024)Gabriella Davis

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

Histor y of HAM Radio presentation slidevu2urc

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

Kürzlich hochgeladen (20)

Strategies for Landing an Oracle DBA Job as a Fresher

🐬 The future of MySQL is Postgres 🐘

How to convert PDF to text with Nanonets

What Are The Drone Anti-jamming Systems Technology?

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Handwritten Text Recognition for manuscripts and early printed texts

Scaling API-first – The story of a global engineering organization

Data Cloud, More than a CDP by Matt Robison

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

GenCyber Cyber Security Day Presentation

A Domino Admins Adventures (Engage 2024)

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Presentation on how to chat with PDF using ChatGPT code interpreter

[2024]Digital Global Overview Report 2024 Meltwater.pdf

Histor y of HAM Radio presentation slide

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

CNv6 Instructor Chapter 6 Quality of Service

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020

1. for a Kafka streaming platform DataOps automation

2. Presenters Andrew Stevenson CTO Lenses.io Spiros Economakis DevOps Cloud Lead Lenses.io

3. Delivering data intensity on Kafka Lenses.io

4. What is DataOps & Lenses.io How to automate your data landscape GitOps demo Agenda

5. What do we do?

6. Deploying Kafka isn’t interesting any more?

7. Well trodden path

8. Tech Intensity Use commodity infrastructure Don’t reinvent Focus on building data landscape

9. If not kafka, what are we automating? The important bit... Data Intensity Tech Intensity

10. How are you automating?

11. Pushing Are sure you want to give access? Who resolves the desired state? Who applies the desired state?

12. K8s operator for Kafka Better but...

13. Observe git apply desired state GitOps

14. GitOps for DataOps

15. GitOps Benefits Focus build reusable apps - config driven - less ops Reduced attack surface to sensitive environments Immutable Auditable Rollback Separation of concerns Use any Git server - Don’t need k8s Operator enforces multi-tenancy and governance

16. GitOps for Kafka Infrastructure deployment is now a commodity - Terraform, Cloudformation, etc. Tech intensity Repeatable, rapid deployment of the data landscape Topics Schemas ACLs Alert rules Data policies Applications Data intensity

17. GitOps for Kafka applications Demo

18. Demo architecture

19.

20. Building data intensity DataOps

21. LEARN MORE All-in-one Lenses.io DataOps workspace & Kafka docker container Lenses.io workspace with Kafka in the cloud lenses.io/start launchpass.com/lensesio https://launchpass.com/lensesio

22. for a Kafka streaming platform DataOps automation Tuesday, August 25 2020 07:30 PDT / 14h30 GMT events.kafka-summit.org/2020-schedule