Viktor Gamov, Confluent, Developer Advocate Apache Kafka is an open source distributed streaming platform that allows you to build applications and process events as they occur. Viktor Gamov (developer Advocate at Confluent) walks through how it works and important underlying concepts. As a real-time, scalable, and durable system, Kafka can be used for fault-tolerant storage as well as for other use cases, such as stream processing, centralized data management, metrics, log aggregation, event sourcing, and more. This talk will explain what a streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. https://www.meetup.com/Chennai-Kafka/events/269942117/

1. 1
Your Projector is Not Broken

2. What is Apache Kafka
Event-Streaming
Platform?January, 2020
@gamussa | #confluentvug | @ConfluentINc

3. @gamussa | #confluentvug | @ConfluentINc
3

4. @gamussa | #confluentvug | @ConfluentINc
4
I build highly scalable
Hello World
apps

5. @gamussa | #confluentvug | @ConfluentINc
5
A company is build on
DATA FLOWS
but
All we have is
DATA STORES

6. @gamussa | #confluentvug | @ConfluentINc
6
Storage

7. @gamussa | #confluentvug | @ConfluentINc
7
● DB - table
● Hadoop - file
● Kafka - ?
Core
Abstraction

9. @gamussa | #confluentvug | @ConfluentINc
9
LOG

10. @gamussa | #confluentvug | @ConfluentINc
10
The log is a simple idea
Messages are added at the end of the log
Old New

11. @gamussa | #confluentvug | @ConfluentINc
11
Messages are added at the end of the log
Old New
The log is a simple idea

12. @gamussa | #confluentvug | @ConfluentINc
12
Pub / Sub

13. @gamussa | #confluentvug | @ConfluentINc
13
Time

14. @gamussa | #confluentvug | @ConfluentINc
14
C2 C3C1
Time

15. @gamussa | #confluentvug | @ConfluentINc
15
Time
A
B
C
D
hash(key) %
numPartitions = N

16. @gamussa | #confluentvug | @ConfluentINc
16
Messages will be produced
in a round robin fashion
Time

17. @gamussa | #confluentvug | @ConfluentINc
17
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

18. @gamussa | #confluentvug | @ConfluentINc
18
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan
Consumers have a position all of their own

19. @gamussa | #confluentvug | @ConfluentINc
19
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan
Consumers have a position all of their own

20. @gamussa | #confluentvug | @ConfluentINc
20
Only Sequential Access
Old New
Read to offset & scan

21. CONSUMER GROUP
COORDINATOR
CONSUMERS
CONSUMER GROUP

22. @gamussa | #confluentvug | @ConfluentINc
22
C

23. @gamussa | #confluentvug | @ConfluentINc
23
CC
C1
CC
C2

24. @gamussa | #confluentvug | @ConfluentINc
24
C C
C C

25. @gamussa | #confluentvug | @ConfluentINc
25
0 1
2 3

26. @gamussa | #confluentvug | @ConfluentINc
26
0 1
2 3

27. @gamussa | #confluentvug | @ConfluentINc
27
0, 3 1
2 3

28. @gamussa | #confluentvug | @ConfluentINc
28
Linearly Scalable Architecture
Single topic:
- Many producers machines
- Many consumer machines
- Many Broker machines
No Bottleneck!!
Producers
Consumers

29. 29
From/to other systems: Kafka Connect
and more
Tip: Great option to gradually move workloads to Kafka while keeping production running!

30. 30
Kafka Connect
● Deployed standalone (development) or as a distributed cluster (production)
● Elastic service that works on bare-metal, VMs, containers, Kubernetes, ...
● The individual ‘Connector’ determines delivery guarantees, e.g., exactly-once
VM VM

31. 31
Single Message Transforms for real-time ETL
Ingress: modify an Event before storing
● Obfuscate sensitive information, e.g. PII
● Add origin of event for lineage tracking
● Remove unnecessary data fields
● … and more
Egress: modify an Event on its way out
● Route high-priority events to faster stores
● Direct events to different Elasticsearch indexes
● Cast data types to match destination
● … and more
{ user: ab123,
gender: female,
ip: 1.2.3.95 }
{ user: ab123,
ip: 1.2.3.XXX }

32. @gamussa | #confluentvug | @ConfluentINc
32
Replicate to get fault tolerance
replicate
msg
msg
leader
Machine A
Machine B

33. @gamussa | #confluentvug | @ConfluentINc
33
Partition Leadership and Replication
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

34. @gamussa | #confluentvug | @ConfluentINc
34
Replication provides resiliency
A replica takes over on machine failure

35. @gamussa | #confluentvug | @ConfluentINc
35
Partition Leadership and Replication - node failure
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

36. 36
Similar to a traditional messaging system
(ActiveMQ, Rabbit etc) but with:
(a) Far better scalability
(b) Built in fault tolerance / HA
(c) Storage
The log is a type of durable
messaging system

37. @gamussa | #confluentvug | @ConfluentINc
Stop!
Demo time!

38. @gamussa | #confluentvug | @ConfluentINc
38
Processing

39. @gamussa | #confluentvug | @ConfluentINc
39
Streaming
is the toolset
for dealing with
events
as they move!

40. @gamussa | #confluentvug | @ConfluentINc
40
authorization_attempts possible_fraud
What exactly is Stream Processing?

41. @gamussa | #confluentvug | @ConfluentINc
41
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?

42. @gamussa | #confluentvug | @ConfluentINc
42
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?

43. @gamussa | #confluentvug | @ConfluentINc
43
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?

44. @gamussa | #confluentvug | @ConfluentINc
44
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?

45. @gamussa | #confluentvug | @ConfluentINc
45
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?

46. @gamussa | #confluentvug | @ConfluentINc
46
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud
What exactly is Stream Processing?

47. @gamussa | #confluentvug | @ConfluentINc
47
Lower the bar to enter the world of streaming
User Population
CodingSophistication
Core developers who use Java/Scala
Core developers who don’t use Java/Scala, e.g. .NET, Go
Data engineers, architects, DevOps/SRE
BI analysts
streams

48. @gamussa | #confluentvug | @ConfluentINc
48
KSQL #FTW
4 Headless1 UI 2 CLI
ksql>
3 REST
POST /query

49. @gamussa | #confluentvug | @ConfluentINc
49
Interaction with Kafka
Kafka
(data)
KSQL
(processing)
Application
(processing) Java/KStreams, .NET
Does not run on
Kafka brokers
Does not run on
Kafka brokers

50. @gamussa | #confluentvug | @ConfluentINc
50
Find your local Meetup Group
https://cnfl.io/kafka-meetups
Join us in Slack
http://cnfl.io/slack
Grab Stream Processing books
https://cnfl.io/book-bundle

51. @
@gamussa | #confluentvug | @ConfluentINc
Thanks!
@gamussa
viktor@confluent.io