Suche senden
Hochladen
Kafka Retry and DLQ
•
8 gefällt mir
•
6,325 views
G
George Teo
Folgen
Kafka Retry and DLQ work presented for Bay Area Apache Kafka Meetup.
Weniger lesen
Mehr lesen
Technologie
Melden
Teilen
Melden
Teilen
1 von 72
Empfohlen
Deploying Flink on Kubernetes - David Anderson
Deploying Flink on Kubernetes - David Anderson
Ververica
Apache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and Developers
confluent
Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka
Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka
Kai Wähner
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
confluent
Making Apache Spark Better with Delta Lake
Making Apache Spark Better with Delta Lake
Databricks
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?
Kai Wähner
Apache Flink internals
Apache Flink internals
Kostas Tzoumas
Kafka 101
Kafka 101
Clement Demonchy
Empfohlen
Deploying Flink on Kubernetes - David Anderson
Deploying Flink on Kubernetes - David Anderson
Ververica
Apache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and Developers
confluent
Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka
Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka
Kai Wähner
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
confluent
Making Apache Spark Better with Delta Lake
Making Apache Spark Better with Delta Lake
Databricks
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?
Kai Wähner
Apache Flink internals
Apache Flink internals
Kostas Tzoumas
Kafka 101
Kafka 101
Clement Demonchy
Stream processing using Kafka
Stream processing using Kafka
Knoldus Inc.
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®
confluent
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Jean-Paul Azar
Introduction to Apache Kafka
Introduction to Apache Kafka
Jeff Holoman
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Flink Forward
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache Flink
DataWorks Summit
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
Flink Forward
Introduction to Kafka Streams
Introduction to Kafka Streams
Guozhang Wang
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!
Guido Schmutz
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processing
Till Rohrmann
Introduction to Apache Kafka
Introduction to Apache Kafka
AIMDek Technologies
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
Araf Karsh Hamid
Introduction to Apache Flink
Introduction to Apache Flink
mxmxm
Introduction to Apache Flink
Introduction to Apache Flink
datamantra
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDB
confluent
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
confluent
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Legacy Typesafe (now Lightbend)
Apache Flink and what it is used for
Apache Flink and what it is used for
Aljoscha Krettek
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Kai Wähner
kafka
kafka
Amikam Snir
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA
Introduction to Apache Kafka
Introduction to Apache Kafka
Ricardo Bravo
Weitere ähnliche Inhalte
Was ist angesagt?
Stream processing using Kafka
Stream processing using Kafka
Knoldus Inc.
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®
confluent
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Jean-Paul Azar
Introduction to Apache Kafka
Introduction to Apache Kafka
Jeff Holoman
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Flink Forward
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache Flink
DataWorks Summit
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
Flink Forward
Introduction to Kafka Streams
Introduction to Kafka Streams
Guozhang Wang
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!
Guido Schmutz
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processing
Till Rohrmann
Introduction to Apache Kafka
Introduction to Apache Kafka
AIMDek Technologies
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
Araf Karsh Hamid
Introduction to Apache Flink
Introduction to Apache Flink
mxmxm
Introduction to Apache Flink
Introduction to Apache Flink
datamantra
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDB
confluent
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
confluent
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Legacy Typesafe (now Lightbend)
Apache Flink and what it is used for
Apache Flink and what it is used for
Aljoscha Krettek
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Kai Wähner
kafka
kafka
Amikam Snir
Was ist angesagt?
(20)
Stream processing using Kafka
Stream processing using Kafka
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Introduction to Apache Kafka
Introduction to Apache Kafka
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache Flink
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
Introduction to Kafka Streams
Introduction to Kafka Streams
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Kafka
Introduction to Apache Kafka
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
Introduction to Apache Flink
Introduction to Apache Flink
Introduction to Apache Flink
Introduction to Apache Flink
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDB
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Apache Flink and what it is used for
Apache Flink and what it is used for
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
kafka
kafka
Ähnlich wie Kafka Retry and DLQ
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA
Introduction to Apache Kafka
Introduction to Apache Kafka
Ricardo Bravo
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache Kafka
Hotstar
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Mingmin Chen
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
confluent
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
confluent
Akka gRPC quick-guide
Akka gRPC quick-guide
Knoldus Inc.
Akka gRPC quick-guide
Akka gRPC quick-guide
Knoldus Inc.
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
Ceph Community
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
HostedbyConfluent
Query and audit logging in cassandra
Query and audit logging in cassandra
Vinay Kumar Chella
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward
Storing State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your Analytics
Yaroslav Tkachenko
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
AWSKRUG - AWS한국사용자모임
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Steve Michael Fernandes
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
confluent
Building a Dynamic Rules Engine with Kafka Streams
Building a Dynamic Rules Engine with Kafka Streams
HostedbyConfluent
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
confluent
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
Lagopus SDN/OpenFlow switch
Ähnlich wie Kafka Retry and DLQ
(20)
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Introduction to Apache Kafka
Introduction to Apache Kafka
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache Kafka
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
Akka gRPC quick-guide
Akka gRPC quick-guide
Akka gRPC quick-guide
Akka gRPC quick-guide
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Query and audit logging in cassandra
Query and audit logging in cassandra
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Storing State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your Analytics
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Building a Dynamic Rules Engine with Kafka Streams
Building a Dynamic Rules Engine with Kafka Streams
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
Kürzlich hochgeladen
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
Michael W. Hawkins
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Martijn de Jong
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
UK Journal
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
Delhi Call girls
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Neo4j
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Malak Abu Hammad
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Safe Software
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
wesley chun
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Earley Information Science
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Katpro Technologies
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Anna Loughnan Colquhoun
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Enterprise Knowledge
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
sudhanshuwaghmare1
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
The Digital Insurer
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
The Digital Insurer
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
Results
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
The Digital Insurer
Kürzlich hochgeladen
(20)
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Kafka Retry and DLQ
1.
Reliable Messaging Systems
with Kafka at Uber George Teo Software Engineer @ Streaming Data June 19, 2018
2.
Agenda ● Kafka @
Uber ● What is Messaging? ● High-level design ● Future work
3.
Stream Processing - Driver-Rider Match -
ETA App Views Vehicle Information KAFKA Real-time Driver-Rider Matching
4.
A bunch more... ●
Fraud Detection ● Share My ETA ● Driver & Rider Signups ● Etc.
5.
Apache Kafka -
Use Cases ● Inter-service messaging ● Stream processing ○ AthenaX - self-serve platform (Samza, Flink) ● Database changelog transport ○ Schemaless, Cassandra, MySQL ● Data Lake Ingestion ○ HDFS, S3 ● Log Aggregation
6.
PRODUCERS CONSUMERS Real-time Analytics, Alerts, Dashboards Samza /
Flink Applications Data Science Analytics Reporting Kafka Vertica / Hive Rider App Driver App API / Services Etc. Ad-hoc Exploration ELK Kafka @ Uber Debugging Hadoop Surge Mobile App Cassandra Schemaless MySQL DATABASES AWS S3 (Internal) Services
7.
Apache Kafka -
Use Cases ● Inter-service messaging ● Stream processing ○ AthenaX - self-serve platform (Samza, Flink) ● Database changelog transport ○ Schemaless, Cassandra, MySQL ● Ingestion ○ HDFS, S3 ● Logging
8.
Trillions+ ~PBs Messages/Day Data Volume Scale excluding
replication Tens of Thousands Topics 10+ clusters 100+ broker per cluster
9.
Agenda ● Kafka @
Uber ● What is Messaging? ● High-level Design ● Future Work
10.
Messaging ● Streaming ● Queuing
11.
Messaging ● Streaming ○ Ordered
12.
Streaming - Ordered Partition 0
1 2 3 4 5 6 7 8 9 10 11 12 Commit Offset 7 Consumer Application 78
13.
Messaging ● Streaming ○ Ordered ○
Commit up to
14.
Streaming - Commit
Up To Partition 0 1 2 3 4 5 6 7 8 9 10 11 12 Commit Offset 9 Consumer Application 789
15.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned
16.
Streaming - Partitioned Kafka
Topic Partition 0 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 1 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 2 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Instance 2 Consumer Instance 1 Consumer Application
17.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered
18.
Queueing- Unordered 0 1
2 3 4 5 6 7 8 9 10 11 12 Consumer Application 6 Queue 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Application
19.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered ○ Ack
20.
Queueing- Ack 0 1
2 3 4 5 6 7 8 9 10 11 12 Consumer Application 6 Queue 0 1 2 3 4 5 6 7 8 9 10 11 12 Ack 6 Consumer Application
21.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered ○ Ack/Nack
22.
Queueing- Nack 0 1
2 3 4 5 6 7 8 9 10 11 12 Consumer Application Queue 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Application Nack 4 5 Ack 5 4
23.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered ○ Ack/Nack ○ Redelivery/DLQ
24.
Queueing- Redelivery/DLQ 0 1
2 3 4 5 6 7 8 9 10 11 12 Consumer Application Queue 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Application4 Nack 4
25.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered ○ Ack/Nack ○ Redelivery/DLQ ○ Competing Consumer
26.
Queuing - Competing Consumer Instance
2 Consumer Instance 1 Consumer Application 0 1 2 3 4 5 6 7 8 9 10 11 12 Queue 0 1 2 3 4 5 6 7 8 9 10 11 12
27.
Why Streaming? ● Data
Analytics ● Event-driven Architecture/Event Sourcing ● Log Aggregation ● Scalability*
28.
Why Queuing? ● More
friendly acknowledgement semantics ● Non-blocking retry ● Elastic Consumer scalability ● High consumer throughput
29.
Agenda ● Kafka @
Uber ● Motivation ● High-level design ● Future work
30.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter failover ● Golang, Java, Python client
31.
Why Kafka? ● Other
options: ○ ___MQ ○ Apache Pulsar ○ Build yet another messaging queue from ground up ● Why Kafka? ○ Reliability ○ Performance ○ Cost efficiency ■ Operational, engineering costs
32.
Native Kafka Consumer topic Partition
0 0 1 2 3 4 5 6 Consumer Process
33.
Consumer Process Competing Consumer topic Partition 0 0
1 2 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1
34.
Competing Consumer Kafka Topic Partition
0 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 1 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 2 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Instance 2 Consumer Instance 1 Consumer Application Worker 1 Worker 2 Worker 3 Worker 4
35.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
36.
Message Flow -
Incoming Message topic Partition 0 0 1 2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 Consumer Worker1Consumer Worker1Consumer Worker1 1 56 4 3
37.
Message Flow -
Inflight Message Tracker topic Partition 0 0 1 2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 2
38.
Message Flow -
Passed to worker topic Partition 0 0 1 2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 2 3 4 5
39.
Consumer Application In-Flight Offset
tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed
40.
Consumer Application In-flight Offset
Tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3
41.
Consumer Application In-flight Offset
Tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3 6 4 5
42.
Consumer Application In-flight Offset
Tracker - Ack 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3 6 4
43.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
44.
Consumer Application In-flight Offset
Tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3 6 4 7
45.
Consumer Application In-flight Offset
Tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3 6 8 7
46.
Consumer Application In-flight Offset
Tracker - Nack 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 6 8 7
47.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 6 7 8 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 (3) 3 Nack
48.
Consumer Application In-flight Offset
Tracker - Nack 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 6 8 7
49.
Ack topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 6 7 8 9 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 (3) 4 Commit up to 5
50.
Consumer Application In-flight Offset
Tracker - Nack 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 6 8 7
51.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
52.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 (3) 5 Redelivery 6 7 8 9 0 (3)
53.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
54.
Delay Delay Between Retries topic Partition
0 0 1 2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 1 5 6 7 8 9
55.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
56.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 1 2 3 4 5 6 topic__cg__dlq Partition 0 0 1 2 3 4 5 6 Dead Letter Queue
57.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
58.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 1 2 3 4 5 6 topic__cg__dlq Partition 0 0 1 2 3 4 5 6 Purge DLQ - Purge
59.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 1 2 3 4 5 6 topic__cg__dlq Partition 0 0 1 2 3 4 5 6 Merge DLQ - Merge
60.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
61.
DC2 DC1 Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync Service Aggregate Kafka uReplicator Kafka
Setup @ Uber
62.
DC2 DC1 Consumer Application Active Consumer Application Passive Multi-DC Consumer Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync
Service Aggregate Kafka uReplicator
63.
Multi-DC Consumer DC2 DC1 Consumer Application Active Consumer Application Passive Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync
Service Aggregate Kafka uReplicator
64.
Multi-DC Consumer DC2 DC1 Consumer Application Active Consumer Application Passive Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync
Service Aggregate Kafka uReplicator Offset sync
65.
Multi-DC Consumer DC2 DC1 Consumer Application Active Consumer Application Passive Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync
Service Aggregate Kafka uReplicator
66.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter failover ● Golang, Java, Python client
67.
API in Golang handlerConsumer,
_ := consumer.NewHandlerConsumer(consumer.Params{ Subscriptions: []Subscription{ { Handler: func(context.Context, []byte) error { // Do stuff return nil }, TopicConfig: NewTopicConfig("topic", AllActiveStrategyActivePassive) .WithConcurrency(100) .WithTimeout(time.Second), }, } }) handlerConsumer.Start() defer handlerConsumer.Stop()
68.
Agenda ● Use cases
& scale ● Motivation ● High-level design ● Future work
69.
Future Work ● Open
source ● Async RPC ● Runtime offset reset ● Active-Active MultiDC Consumer ● Federated multi-cluster Kafka
70.
Links ● https://github.com/uber-go/kafka-client ● https://eng.uber.com/reliable-reprocessing/ ●
https://github.com/uber/uReplicator
71.
Thank you Proprietary and
confidential © 2016 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber. More open-source projects at eng.uber.com
72.
Proprietary and confidential
© 2018 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.