Slides from XXVII Betabeers talk about removing dependencies between services using Apache Kafka.

2. How to spread information between services? (I)
Traditionally, a direct call from the producer to the consumer/s inside the
producer service logic is performed. This approach has several drawbacks:
• Caller lack of knowledge of the external services.
• Complex logic when trying to ensure data consistency.
• Caller needs to know if the consumer/s can process information as quick as it is
produced (max. throughput)
• Etc.

3. How to spread information between services? (II)
The usage of a message broker software is one of the better approaches to share
information between services. It has several benefits:
• Forget about nested web services, rollbacks and just focus in your own business.
• Producer services does not need to know about the consumer service throughput.
• Having a strong and reliable communication system between different systems in the
company which will allow us to have new business opportunities.
• Publish/Subscribe pattern.
• Etc.

4. Message broker solutions
• Apache ActiveMQ
• RabbitMQ
• Microsoft Azure Service Bus
• Apache Kafka
• Etc

5. What is Kafka?
• Kafka is a distributed streaming platform developed by Apache
Software Foundation and written in Scala and Java.
• Scalability is possible both logical and physical.
• It can be used as:
• Messaging System
• Event notification
• Website Activity Tracking
• Log Aggregation
• Storage System
• Stream Processing

6. Kafka Infrastructure
• Cluster of one or more nodes (brokers)
• Each topic partition has a specific node as leader.
• Apache ZooKeeper to keep in sync the brokers.
• Streams of records are stored in categories  Topics
• We can see a topic as a key/value dictionary. Offset and a timestamp are
stored in the record as metadata.

7. Writing to/Reading from Kafka
• Producers
• Consumers
• Consumer group (parallel processing)
• Connectors
• Source (producers)
• Sink (consumers)
• Stream processors

8. Writing to/Reading from Kafka (II)
• Each record has an unique position inside the partition (offset)
• Kafka stores the partition offset for a consumer group.

9. Real scenario using Kafka

10. What do we use it for?
P&P
WS
P&P Apps
WebShop
BS
PLM

11. What do we use it for? (II)
• Under the previous scenario we were asked to:
• Ensure “transactionality” on the different “update” web services.
• When something goes wrong, be sure that all systems are in a consistent
state.

12. What do we use it for? (III)

13. What do we use it for? (IV)
Producer
Webshop consumer group
C1 C2 C3
PLM consumer group
C1 C2

14. Useful Links
• Apache Kafka documentation page:
• https://kafka.apache.org/documentation/
• Confluent Blog:
• https://www.confluent.io/blog/