Kafka overview and use cases

•Als PPTX, PDF herunterladen•

3 gefällt mir•957 views

Overview of kafka, how it works, components of kafka, use cases. Kafka at LinkedIn. Download the slides to see animations explaining how the components fit. This was presented on kafka meetup help on June 11, 2016 @LinkedinBangalore ofc

Technologie

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Kafka - Overview
Indrajeet Kumar
Site Reliability Engineer at LinkedIn

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved. 2
So what is it?
It is a high-throughput, low-latency messaging system

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
And who uses it?
3

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
What for?
•Messaging
•Website Activity Tracking
•Metrics
•Log Aggregation
•Stream Processing
•For fun ;)
4

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
So how does it work?
▪ Components
– Producer
– Broker
▪ Topic
▪ Partition
– Consumer
5

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Broker
producer producer producer
B2B1P1 P2P1
R P2
R
6
consumer consumer consumer

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
The consumer
7
consumer
B2B1P1 P2
B3P3
C1 C2
P1
R

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
The Producer
8
Producer
B2B1P1 P2
B3P3
P1 P2
P1
R

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Attributes of a Kafka Cluster
▪ Durable
▪ Scalable
▪ Low Latency
▪ Finite Retention
▪ No single point of failure
9

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Kafka At LinkedIn
▪ Multiple Datacenters, Multiple Clusters
▪ Mirroring between clusters
▪ Message Types
– Metrics
– Tracking
– Queuing
▪ Data transport from applications to Hadoop, and back
10

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Some numbers!
▪ 1800+ Broker machines
▪ 79K+ Topics
▪ 1.1M+ Partitions
▪ 1.3 Trillion messages per day
▪ 330 Terabytes in/day
▪ 1.2 Petabytes out/day
▪ Peak load for a single cluster
– 2 million messages/sec
– 4.7 Gigabits/sec inbound
– 15 Gigabits/sec outbound
11

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Questions
12

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.

Empfohlen

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...confluent

How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...HostedbyConfluent

Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...confluent

Building Event-Driven Services with Apache Kafkaconfluent

Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...HostedbyConfluent

Distributed Enterprise Monitoring and Management of Apache Kafka (William McL...HostedbyConfluent

Scaling a backend for a big data and blockchain environment by Rafael Ríos at...Big Data Spain

Building Microservices with Apache Kafkaconfluent

Empfohlen

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...confluent

How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...HostedbyConfluent

Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...confluent

Building Event-Driven Services with Apache Kafkaconfluent

Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...HostedbyConfluent

Distributed Enterprise Monitoring and Management of Apache Kafka (William McL...HostedbyConfluent

Scaling a backend for a big data and blockchain environment by Rafael Ríos at...Big Data Spain

Building Microservices with Apache Kafkaconfluent

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...HostedbyConfluent

Best Practices for Streaming IoT Data with MQTT and Apache Kafka®confluent

More Datacenters, More ProblemsTodd Palino

Help, My Kafka is Broken! (Emma Humber & Gantigmaa Selenge, IBM) Kafka Summit...HostedbyConfluent

Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, JupiterHostedbyConfluent

Deep Dive Series #3: Schema Validation + Structured Audit Logsconfluent

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...HostedbyConfluent

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...HostedbyConfluent

Creating an Elastic Platform Using Kafka and Microservices in OpenShift confluent

Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...HostedbyConfluent

Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...HostedbyConfluent

How Zillow Unlocked Kafka to 50 Teams in 8 months | Shahar Cizer Kobrinsky, Z...HostedbyConfluent

Elastically Scaling Kafka Using Confluentconfluent

What is Apache Kafka and What is an Event Streaming Platform?confluent

How Much Can You Connect? | Bhavesh Raheja, Disney + HotstarHostedbyConfluent

From legacy systems to microservices and back | Andera Gioia, QuantycaHostedbyConfluent

How Yelp Leapt to Microservices with More than a Message Queueconfluent

Multi tier, multi-tenant, multi-problem kafkaTodd Palino

AWS Api Gateway by Łukasz Marchewka ScalaccScalac

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...HostedbyConfluent

Event Driven Architectures with Apache Kafka on HerokuHeroku

Real time Analytics with Apache Kafka and Apache SparkRahul Jain

Weitere ähnliche Inhalte

Was ist angesagt?