ndependent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target. This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Apache Kafka
Skalierbare Nachrichtenverarbeitungund mehr!
Guido Schmutz

2. Guido Schmutz
Working for Trivadis for more than 19 years
Oracle ACE Director for Fusion Middleware and SOA
Co-Author of different books
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Member of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 25 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Twitter: gschmutz

3. Unser Unternehmen.
Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution
Engineering und der Erbringung von IT-Services mit Fokussierung auf - und
-Technologien in der Schweiz, Deutschland, Österreich und Dänemark. Trivadis erbringt
ihre Leistungen aus den strategischen Geschäftsfeldern:
Trivadis Services übernimmt den korrespondierenden Betrieb Ihrer IT Systeme.
B E T R I E B

4. KOPENHAGEN
MÜNCHEN
LAUSANNE
BERN
ZÜRICH
BRUGG
GENF
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
WIEN
Mit über 600 IT- und Fachexperten bei Ihnen vor Ort.
14 Trivadis Niederlassungen mit
über 600 Mitarbeitenden.
Über 200 Service Level Agreements.
Mehr als 4'000 Trainingsteilnehmer.
Forschungs- und Entwicklungsbudget:
CHF 5.0 Mio.
Finanziell unabhängig und
nachhaltig profitabel.
Erfahrung aus mehr als 1'900 Projekten
pro Jahr bei über 800 Kunden.

5. Agenda
1. Introduction & Motivation
2. Kafka Core
3. Kafka Connect
4. Kafka Streams
5. Kafka and ”Big Data” / ”Fast Data” Ecosystem
6. Confluent Data Platform
7. Kafka in Architecture
8. Summary

6. Introduction & Motivation

7. A little story of a “real-life” customer situation
Traditional system interact with its clients
and does its work
Implemented using legacy technologies (i.e.
PL/SQL)
New requirement:
• Offer notification service to notify
customer when goods are shipped
• Subscription and inform over different
channels
• Existing technology doesn’t fit
delivery
Logistic
System
Oracle
Mobile Apps
Sensor ship
sort
8
Rich (Web)
Client Apps
DB
schedule
Logic
(PL/SQL)
delivery

8. A little story of a “real-life” customer situation
Events are “owned” by traditional
application (as well as the channels
they are transported over)
Implement notification as a new Java-based
application/system
But we need the events ! => so let’s
integrate
delivery
Logistic
System
Oracle
Mobile Apps
Sensor ship
sort
9
Rich (Web)
Client Apps
DB
schedule
Notification
Logic
(PL/SQL)
Logic
(Java)
delivery
SMS
Email
…

9. A little story of a “real-life” customer situation
integrate in order to get the information!
Oracle Service Bus was already there
Rule Engine implemented in Java and invoked from
OSB message flow
Notification system informed via queue
Higher Latency introduced (good enough in this
case)
delivery
Logistic
System
Oracle
Oracle
Service Bus
Mobile Apps
Sensor AQship
sort
10
Rich (Web)
Client Apps
DB
schedule
Filter
Notification
Logic
(PL/SQL)
JMS
Rule Engine
(Java)
Logic
(Java)
delivery
shipdelivery
delivery true SMS
Email
…

10. A little story of a “real-life” customer situation
Treat events as first-class citizens
Events belong to the “enterprise” and
not an individual system => Catalog of
Events similar to Catalog of
Services/APIs !!
Event (stream) processing can be
introduced and by that latency reduced!
delivery
Logistic
System
Oracle
Oracle
Service Bus
Mobile Apps
Sensor AQship
sort
11
Rich (Web)
Client Apps
DB
schedule
Filter
Notification
Logic
(PL/SQL)
JMS
Rule Engine
(Java)
Logic
(Java)
delivery
shipdelivery
delivery true SMS
Email
…

11. Treat Events as Events and share them!
delivery
Logistic
System
Oracle
Oracle
Service Bus
Mobile Apps
Sensor
ship
sort
12
Rich (Web)
Client Apps
DB
schedule
Filter
Notification
Logic
(PL/SQL)
JMS
Rule Engine
(Java)
Logic
(Java)
delivery
ship
delivery true SMS
Email
…
Event
Bus/Hub
Stream/Event
Processing

12. Treat Events as Events,share and make use of them!
delivery
Logistic
System
Oracle
Mobile Apps
Sensor
ship
sort
13
Rich (Web)
Client Apps
DB
schedule
Filter
Notification
Logic
(PL/SQL)
Rule Engine
(Java)
Logic
(Java)
delivery
SMS
Email
…
Event
Bus/Hub
Stream/Event
Processing
notifiableDelivery
notifiableDelivery
delivery

13. Kafka Stream Data Platform
Source: Confluent

14. Kafka Core

15. Apache Kafka - Overview
Distributed publish-subscribe messaging system
Designed for processing of real time activity stream data (logs, metrics
collections, social media streams, …)
Initially developed at LinkedIn, now part of Apache
Does not use JMS API and standards
Kafka maintains feeds of messages in topics

16. Apache Kafka - Motivation
LinkedIn’s motivation for Kafka was:
• “A unified platform for handling all the real-time data feeds a large company might
have.”
Must haves
• High throughput to support high volume event feeds.
• Support real-time processing of these feeds to create new, derived feeds.
• Support large data backlogs to handle periodic ingestion from offline systems.
• Support low-latency delivery to handle more traditional messaging use cases.
• Guarantee fault-tolerance in the presence of machine failures.

17. Kafka High Level Architecture
The who is who
• Producers write data to brokers.
• Consumers read data from
brokers.
• All this is distributed.
The data
• Data is stored in topics.
• Topics are split into partitions,
which are replicated.
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble

18. Apache Kafka - Architecture
Kafka Broker
Movement
Processor
Movement Topic
Engine-Metrics Topic
1 2 3 4 5 6
Engine
Processor1 2 3 4 5 6
Truck

19. Apache Kafka - Architecture
Kafka Broker
Movement
Processor
Movement Topic
Engine-Metrics Topic
1 2 3 4 5 6
Engine
Processor
Partition 0
1 2 3 4 5 6
Partition 0
1 2 3 4 5 6
Partition 1 Movement
Processor
Truck

20. Apache
Kafka
Kafka Broker 1
Movement
Processor
Truck
Movement Topic
P 0
Movement
Processor
1 2 3 4 5
P 2 1 2 3 4 5
Kafka Broker 2
Movement Topic
P 2 1 2 3 4 5
P 1 1 2 3 4 5
Kafka Broker 2
Movement Topic
P 0 1 2 3 4 5
P 1 1 2 3 4 5
Movement
Processor

21. Apache Kafka - Architecture
• Write Ahead Log / Commit Log
• Producers always append to tail
• think append to file
Kafka Broker
Movement Topic
1 2 3 4 5
Truck
6 6

22. Durability Guarantees
Producer can configure acknowledgements
Value Impact Durability
0 • Producer doesn’t wait for leader weak
1 (default) • Producer waits for leader
• Leader sends ack when message written to log
• No wait for followers
medium
all • Producer waits for leader
• Leader sends ack when all In-Sync Replica have
acknowledged
strong

23. Apache Kafka - Partition offsets
Offset: messages in the partitions are each assigned a unique (per partition) and
sequential id called the offset
• Consumers track their pointers via (offset, partition, topic) tuples
Consumer Group A Consumer Group B

24. Data Retention – 3 options
1. Never
2. Time based (TTL)
log.retention.{ms | minutes | hours}
3. Size based
log.retention.bytes
4. Log compaction based (entries with same key are removed)
kafka-topics.sh --zookeeper localhost:2181
--create --topic customers
--replication-factor 1 --partitions 1
--config cleanup.policy=compact

25. Apache Kafka – Some numbers
Kafka at LinkedIn => over 1800+ broker machines / 79K+ Topics
Kafka Performance at our own infrastructure => 6 brokers (VM) / 1 cluster
• 445’622 messages/second
• 31 MB / second
• 3.0405 ms average latency between producer / consumer
1.3 Trillion messages per
day
330 Terabytes in/day
1.2 Petabytes out/day
Peak load for a single cluster
2 million messages/sec
4.7 Gigabits/sec inbound
15 Gigabits/sec outbound
http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
https://engineering.linkedin.com/kafka/running-kafka-scale

26. Kafka Connect

27. Kafka Connect Architecture

28. Kafka Connector Hub – Certified Connectors
Source: http://www.confluent.io/product/connectors

29. Kafka Connector Hub – Additional Connectors
Source: http://www.confluent.io/product/connectors

30. Kafka Streams

31. Kafka Streams
• Designed as a simple and lightweight library in Apache Kafka
• no external dependencies on systems other than Apache Kafka
• Leverages Kafka as its internal messaging layer
• agnostic to resource management and configuration tools
• Supports fault-tolerant local state
• Event-at-a-time processing (not microbatch) with millisecond latency
• Windowing with out-of-order data using a Google DataFlow-like model

32. Kafka Streams Architecture

33. Kafka and ”Big Data” / ”Fast Data”
Ecosystem

34. Kafka and the Big Data / Fast Data ecosystem
Kafka integrates with many popular
products / frameworks
• Apache Spark Streaming
• Apache Flink
• Apache Storm
• Apache NiFi
• Streamsets
• Apache Flume
• Oracle Stream Analytics
• Oracle Service Bus
• Oracle GoldenGate
• Spring Integration Kafka Support
• …Storm built-in Kafka Spout to consume events from Kafka

35. Confluent Platform

36. Confluent Data Platform 3.0

37. Kafka in Architecture

38. Hadoop Clusterd
Hadoop Cluster
Hadoop Cluster
Customer Event Hub – taking Velocity into account
Location
Social
Click
stream
Sensor
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Call
Center
Mobile
Apps
Batch Analytics
Streaming Analytics
Event
Hub
Event
Hub
Event
Hub
NoSQL
Parallel
Processing
Distributed
Filesystem
Stream Analytics
NoSQL
Reference /
Models
SQL
Search
Dashboard
BI Tools
Enterprise Data
Warehouse
Search
Online & Mobile
Apps
File Import / SQL Import
Weather
Data

39. Weather
Data
SQL Import
Hadoop Clusterd
Hadoop Cluster
Hadoop Cluster
Location
Social
Click
stream
Sensor
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Call
Center
Mobile
Apps
Batch Analytics
Streaming Analytics
Event
Hub
Event
Hub
Event
Hub
NoSQL
Parallel
Processing
Distributed
Filesystem
Stream Analytics
NoSQL
Reference /
Models
SQL
Search
Dashboard
BI Tools
Enterprise Data
Warehouse
Search
Online & Mobile
Apps
Customer Event Hub – mapping of technologies

40. Summary

41. Summary
• Kafka can scale to millions of messages per second, and more
• Easy to start with for a PoC
• A bit more to invest to setup production environment
• Monitoring is key
• Vibrant community and ecosystem
• Fast pace technology
• Confluent provides Kafka Distribution

42. Guido Schmutz
Technology Manager
guido.schmutz@trivadis.com