The session discusses on how companies are using Apache Kafka & also covers under the hood details like partitions, brokers, replication. About apache kafka: Apache Kafka is a distributed a streaming platform, Apache Kafka provides low-latency, high-throughput, fault-tolerant publish and subscribe pipelines and is able to process streams of events. Kafka provides reliable, millisecond responses to support both customer-facing applications and connecting downstream systems with real-time data.

1. From : Madhusudhana
IG Group
Date :14-April-2018

2.  Need of Messaging system
 What , Why , How of Apache Kafka
Producers, Brokers , Consumers ,Topics and Partitions
 Developing Producers and Consumers
 Questions
Overview

3. Need of a messaging system

4. Front end UVData pipe line
It often starts with one data pipe line

5. Front end UV
Ledger systems
Instrument
management system
Reuse of existing providers for new consumers

6. Front end
Ledger systems
Instrument
management system
Security systems
Database server
Data ware house
Account Maintenance
system
Real time monitoring
Overnight chargesPricing system
Eventually the solution becomes the problem

7. Front end
Ledger systems
Instrument
management system
Security systems
Database server
Data ware house
Account Maintenance
system
Real time monitoring
Overnight chargesPricing system
Messaging system

8. How Kafka solves this problem

9. Kafka decouples data pipe lines
Data warehouse
Account Maintenance system
Ledger system
Kafka
Real time monitoring Overnight charges
Pricing system Instrument management system Open position system

10. What is Apache Kafka?
 Apache Kafka is a distributed publish-subscribe message system
 It was originally developed at LinkedIn, later open sourced in 2011 and
became a part of Apache project
 Kafka is fast , scalable ,durable , fault-tolerant and distributed by design

11. Kafka Features
High Throughput: Provides support for Hundreds
of thousand of messages
Scalability: Highly scalable distributed systems
with no downtime
Durability: Provides support to persisting
messages on disk
Replication: Messages can be replicated across
clusters , witch supports multiple subscribers
Stream Processing: Kafka can be used along with real time
streaming applications like Spark and storm

13. Kafka Cluster
Producer Consumer
Producer ConsumerKafka cluster
Broker 1
Broker 3
Broker 2
Zookeeper

14. Apache Kafka architecture

17. Apache Kafka architecture
0
Topic Partition1
Partition2
Partition3
Producer A
Consumer A
Consumer B
Consumer C
Broker1
0 1 2
10
1

18. Producer and consumer DEMO