Apache Kafka Fundamentals for Architects, Admins and Developers

•

1 gefällt mir•554 views

confluent

Speaker: Patrick Druley, Senior Systems Engineer, Confluent

Technologie

1
Mid-Ohio Data Users Group
Meetup
Organizer: Derek Kane
Host: Navigator Management Partners

2
Apache Kafka®
Fundamentals for
Architects, Admins and
Developers

3
About Me
● Patrick Druley
● Senior Systems Engineer @ Conﬂuent (1yr)
● Formerly at Oracle and Teradata
● Ohioan, currently in Medina Ohio
● Github: https://github.com/pdruley
● Twitter: @PatrickLovesAK

5
A Modern, Distributed Platform for
Data Streams

6
Apache Kafka® is made up of
distributed, immutable, append-only
commit logs

7
Intelligent Clients and Naive Pipes
Writers
Kafka
cluster
Readers

88
Scalability of a ﬁlesystem
• hundreds of MB/s throughput
• many TB per server
• commodity hardware

99
Guarantees of a Database
• Strict ordering
• Persistence

1010
Rewind & Replay
Reset to any point in the shared narrative

1111
Distributed by design
• Replication
• Fault Tolerance
• Partitioning
• Elastic Scaling

1212
Kafka Topics
my-topic
my-topic-partition-0
my-topic-partition-1
my-topic-partition-2
broker-1
broker-2
broker-3

1313
Partition Leadership and Replication
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

1414
Partition Leadership and Replication - node failure
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

1515
Demo Time!
https://github.com/pdruley/public_notebooks/blob/master/Glorious_Kafka_Command_Line_Utilites.ipynb

1616
Resources
Free E-Books from Conﬂuent!
I Heart Logs:
https://www.conﬂuent.io/ebook/i-heart-logs-event-data-stream-processing-and-data-integration/
Kafka: The Deﬁnitive Guide: https://www.conﬂuent.io/resources/kafka-the-deﬁnitive-guide/
Conﬂuent Blog: https://www.conﬂuent.io/blog
Conﬂuent Podcast:
https://podcasts.apple.com/us/podcast/streaming-audio-a-confluent-podcast-about-apache-kafka/id1
401509765?mt=2
Thank You!
patrick@conﬂuent.io
@PatrickLovesAK

Weitere ähnliche Inhalte

Was ist angesagt?

Stream processing using KafkaKnoldus Inc.

Apache kafkaNexThoughts Technologies

Apache Kafka Architecture & Fundamentals Explainedconfluent

Apache KafkaDiego Pacheco

Kafka presentationMohammed Fazuluddin

Apache Kafka at LinkedInDiscover Pinterest

Kafka 101Aparna Pillai

A visual introduction to Apache KafkaPaul Brebner

kafkaAmikam Snir

How Apache Kafka® Worksconfluent

Apache kafkaViswanath J

Apache kafkaLong Nguyen

ksqlDB: A Stream-Relational Database Systemconfluent

An Introduction to Apache KafkaAmir Sedighi

Apache kafkaSrikrishna k

Apache kafkaRamakrishna kapa

Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar

Stream Processing with Apache Kafka and .NETconfluent

Integrating Apache Kafka Into Your Environmentconfluent

Producer Performance Tuning for Apache KafkaJiangjie Qin

Was ist angesagt? (20)

Stream processing using Kafka

Apache kafka

Apache Kafka Architecture & Fundamentals Explained

Apache Kafka

Kafka presentation

Apache Kafka at LinkedIn

Kafka 101

A visual introduction to Apache Kafka

kafka

How Apache Kafka® Works

Apache kafka

ksqlDB: A Stream-Relational Database System

An Introduction to Apache Kafka

Apache kafka

Kafka Tutorial - Introduction to Apache Kafka (Part 1)

Stream Processing with Apache Kafka and .NET

Integrating Apache Kafka Into Your Environment

Producer Performance Tuning for Apache Kafka

Ähnlich wie Apache Kafka Fundamentals for Architects, Admins and Developers

NoSQL_NightClarence J M Tauro

Lessons learned from running Spark on DockerDataWorks Summit

How is Kafka so Fast?Ricardo Paiva

Hands-on Workshop: Apache PulsarSijie Guo

Capital One Delivers Risk Insights in Real Time with Stream Processingconfluent

Next-Gen DHCPAndreas Taudte

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

Streaming Analytics with Spark, Kafka, Cassandra and AkkaHelena Edelson

Fundamentals and Architecture of Apache KafkaAngelo Cesaro

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena EdelsonSpark Summit

MySQL Options in OpenStackTesora

Messaging, storage, or both? The real time story of Pulsar and Apache Distri...Streamlio

Storage Requirements and Options for Running Spark on KubernetesDataWorks Summit

Open Source SQL DatabasesEmanuel Calvo

Sa introduction to big data pipelining with cassandra & spark west mins...Simon Ambridge

OpenStack Days East -- MySQL Options in OpenStackMatt Lord

Spark volume requirements 2018Rachit Arora

Scala and Spark are Ideal for Big Data - Data Science Pop-up SeattleDomino Data Lab

Introduction to couchbaseDipti Borkar

A New Centralized Volume Storage Solution for Docker and Container Cloud by W...Docker, Inc.

Ähnlich wie Apache Kafka Fundamentals for Architects, Admins and Developers (20)

NoSQL_Night

Lessons learned from running Spark on Docker

How is Kafka so Fast?

Hands-on Workshop: Apache Pulsar

Capital One Delivers Risk Insights in Real Time with Stream Processing

Next-Gen DHCP

Apache Kafka - Scalable Message-Processing and more !

Streaming Analytics with Spark, Kafka, Cassandra and Akka

Fundamentals and Architecture of Apache Kafka

Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson

MySQL Options in OpenStack

Messaging, storage, or both? The real time story of Pulsar and Apache Distri...

Storage Requirements and Options for Running Spark on Kubernetes

Open Source SQL Databases

Sa introduction to big data pipelining with cassandra & spark west mins...

OpenStack Days East -- MySQL Options in OpenStack

Spark volume requirements 2018

Scala and Spark are Ideal for Big Data - Data Science Pop-up Seattle

Introduction to couchbase

A New Centralized Volume Storage Solution for Docker and Container Cloud by W...

Mehr von confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent

Santander Stream Processing with Apache Flinkconfluent

Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent

Workshop híbrido: Stream Processing con Flinkconfluent

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...confluent

AWS Immersion Day Mapfre - Confluentconfluent

Eventos y Microservicios - Santander TechTalkconfluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloudconfluent

Citi TechTalk Session 2: Kafka Deep Diveconfluent

Build real-time streaming data pipelines to AWS with Confluentconfluent

Q&A with Confluent Professional Services: Confluent Service Meshconfluent

Citi Tech Talk: Event Driven Kafka Microservicesconfluent

Confluent & GSI Webinars series - Session 3confluent

Citi Tech Talk: Messaging Modernizationconfluent

Citi Tech Talk: Data Governance for streaming and real time dataconfluent

Confluent & GSI Webinars series: Session 2confluent

Data In Motion Paris 2023confluent

Confluent Partner Tech Talk with Synthesisconfluent

The Future of Application Development - API Days - Melbourne 2023confluent

The Playful Bond Between REST And Data Streamsconfluent

Mehr von confluent (20)

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

The Future of Application Development - API Days - Melbourne 2023

The Playful Bond Between REST And Data Streams

Kürzlich hochgeladen

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los

Automating Google Workspace (GWS) & more with Apps Scriptwesley chun

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco

Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

How to convert PDF to text with Nanonetsnaman860154

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

A Domino Admins Adventures (Engage 2024)Gabriella Davis

Real Time Object Detection Using Open CVKhem

Kürzlich hochgeladen (20)

08448380779 Call Girls In Friends Colony Women Seeking Men

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Automating Google Workspace (GWS) & more with Apps Script

Data Cloud, More than a CDP by Matt Robison

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

What Are The Drone Anti-jamming Systems Technology?

Tata AIG General Insurance Company - Insurer Innovation Award 2024

CNv6 Instructor Chapter 6 Quality of Service

How to convert PDF to text with Nanonets

Scaling API-first – The story of a global engineering organization

Exploring the Future Potential of AI-Enabled Smartphone Processors

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Boost PC performance: How more available memory can improve productivity

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Driving Behavioral Change for Information Management through Data-Driven Gree...

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

A Domino Admins Adventures (Engage 2024)

Real Time Object Detection Using Open CV

Apache Kafka Fundamentals for Architects, Admins and Developers

1. 1 Mid-Ohio Data Users Group Meetup Organizer: Derek Kane Host: Navigator Management Partners

2. 2 Apache Kafka® Fundamentals for Architects, Admins and Developers

3. 3 About Me ● Patrick Druley ● Senior Systems Engineer @ Conﬂuent (1yr) ● Formerly at Oracle and Teradata ● Ohioan, currently in Medina Ohio ● Github: https://github.com/pdruley ● Twitter: @PatrickLovesAK

4. 4

5. 5 A Modern, Distributed Platform for Data Streams

6. 6 Apache Kafka® is made up of distributed, immutable, append-only commit logs

7. 7 Intelligent Clients and Naive Pipes Writers Kafka cluster Readers

8. 88 Scalability of a ﬁlesystem • hundreds of MB/s throughput • many TB per server • commodity hardware

9. 99 Guarantees of a Database • Strict ordering • Persistence

10. 1010 Rewind & Replay Reset to any point in the shared narrative

11. 1111 Distributed by design • Replication • Fault Tolerance • Partitioning • Elastic Scaling

12. 1212 Kafka Topics my-topic my-topic-partition-0 my-topic-partition-1 my-topic-partition-2 broker-1 broker-2 broker-3

13. 1313 Partition Leadership and Replication Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4

14. 1414 Partition Leadership and Replication - node failure Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4

15. 1515 Demo Time! https://github.com/pdruley/public_notebooks/blob/master/Glorious_Kafka_Command_Line_Utilites.ipynb

16. 1616 Resources Free E-Books from Confluent! I Heart Logs: https://www.confluent.io/ebook/i-heart-logs-event-data-stream-processing-and-data-integration/ Kafka: The Definitive Guide: https://www.confluent.io/resources/kafka-the-definitive-guide/ Confluent Blog: https://www.confluent.io/blog Confluent Podcast: https://podcasts.apple.com/us/podcast/streaming-audio-a-confluent-podcast-about-apache-kafka/id1 401509765?mt=2 Thank You! patrick@confluent.io @PatrickLovesAK

17. 1717 Thank You!

Apache Kafka Fundamentals for Architects, Admins and Developers

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Apache Kafka Fundamentals for Architects, Admins and Developers

Ähnlich wie Apache Kafka Fundamentals for Architects, Admins and Developers (20)

Mehr von confluent

Mehr von confluent (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Apache Kafka Fundamentals for Architects, Admins and Developers