1) Event-driven microservices involve microservices communicating primarily through events published to an event backbone. This loosely couples microservices and allows for eventual data consistency. 2) Apache Kafka is an open-source streaming platform that can be used to build an event backbone, allowing microservices to reliably publish and subscribe to events. It supports streaming, storage, and processing of event data. 3) Common patterns for event-driven microservices include database per service for independent data ownership, sagas for coordinated multi-step processes, event sourcing to capture all state changes, and CQRS to separate reads from writes.

1. Apache Kafka
© 2019 IBM Corporation
Event-Driven
Microservices
Using IBM Apache Kafka
Andrew Schofield
Chief Architect, Event Streams
IBM Hursley Park
Event-Driven Architecture meetup

2. © 2019 IBM Corporation
What are microservices?
Microservices is a technique for structuring
an application as a collection of services
• Self-contained with clear interfaces and
a distinct purpose
• Loosely coupled – communicate over a
network
• Independently deployable, scalable,
maintainable and testable
Microservices ApplicationMonolithic Application

3. © 2019 IBM Corporation
Event-driven microservices
Microservices communicate primarily with
events, with APIs where required
• Microservices can produce and
consume events using publish/subscribe
messaging
• Events are handled by an event
backbone
• Data is eventually consistentMICROSERVICES APPLICATION
API
MICROSERVICE
MICROSERVICE
MICROSERVICE
EVENT BACKBONE
MICROSERVICE
MICROSERVICE
MICROSERVICE
MICROSERVICE
MICROSERVICE
API
SERVICE DISCOVERY
PUBLISH
PUBLISH
SUBSCRIBE
SUBSCRIBE

4. © 2019 IBM Corporation
Comparing messaging patterns
QUEUE 0 1 2 3 4 5 6 7
CONSUMER CONSUMER CONSUMER
PRODUCER PRODUCER
TOPIC 0 1 2 3 4 5 6 7
SUBSCRIPTION SUBSCRIPTION
PRODUCER PRODUCER
CONSUMER CONSUMER CONSUMER
POINT-TO-POINT PUBLISH/SUBSCRIBE

5. © 2019 IBM Corporation
Apache Kafka is an Open-Source Streaming Platform
PUBLISH/SUBSCRIBE
Read and write streams of events, like a
traditional messaging system
PROCESS
Support scalable stream processing
applications that react to events in real
time
STORE
Store streams of data safely in a
distributed, replicated, fault-tolerant
cluster
https://kafka.apache.org/
Kafka
Cluster
Producer ProducerProducer
Consumer ConsumerConsumer
Stream
Processor
Stream
Processor
Source
Connector
Sink Connector

6. © 2019 IBM Corporation
Introducing the event backbone
EVENT BACKBONE
Microservic
e
Microservic
e

7. © 2019 IBM Corporation
Introducing the event backbone
1
Building Blocks
1 :: Event sources
Microservic
e
Microservic
e
EVENT BACKBONE

8. © 2019 IBM Corporation
Introducing the event backbone
1
2
Building Blocks
1 :: Event sources
2 :: Stream processing
Microservic
e
Microservic
e
EVENT BACKBONE

9. © 2019 IBM Corporation
Introducing the event backbone
1
2 App
3
Building Blocks
1 :: Event sources
2 :: Stream processing
3 :: Event archive
Microservic
e
Microservic
e
EVENT BACKBONE

10. © 2019 IBM Corporation
Introducing the event backbone
App
1
2
3
4
Building Blocks
1 :: Event sources
2 :: Stream processing
3 :: Event archive
4 :: Notifications
Microservic
e
Microservic
e
EVENT BACKBONE

11. © 2019 IBM Corporation
A simple stream processing example
EVENT BACKBONE
Transactions

12. © 2019 IBM Corporation
A simple stream processing example
EVENT BACKBONE
Transactions
Alerts

13. © 2019 IBM Corporation
A simple stream processing example
EVENT BACKBONE
Transactions
Alerts
SMS alerts
Email alerts
Push alerts

14. © 2019 IBM Corporation
A simple stream processing example
EVENT BACKBONE
Send SMS
Transactions
Alerts
Send emails
Send push
notifications
SMS alerts
Email alerts
Push alerts

15. © 2019 IBM Corporation
How to get started
You could run an event storming workshop
https://www.ibm.com/cloud/garage/architectures/eventDrivenArchitecture/event-storming-
methodology
Identify:
Events
Actors
Data
Commands

16. © 2019 IBM Corporation
A more complex event-driven example
https://ibm-cloud-architecture.github.io/refarch-kc/design/readme/

17. © 2019 IBM Corporation
A more complex example – component view
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
API API

18. © 2019 IBM Corporation
A more complex example – component view
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
API API

19. © 2019 IBM Corporation
A more complex example – component view
EVENT BACKBONE
ships containers orders voyages
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
API API

20. © 2019 IBM Corporation
A more complex example – component view
EVENT BACKBONE
ships containers problems orders voyages
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
Streaming
analytics
API API

21. © 2019 IBM Corporation
Patterns for event-driven microservices
You want:
Loose coupling
Data consistency
Efficient queries
You need PATTERNS, such as:
Database per service
Saga
Event sourcing
CQRS

22. © 2019 IBM Corporation
Pattern – Database per service
Each microservice persists its own data
Protects independence of the microservice against external change
Introduces complexity for data consistency across microservices
EVENT BACKBONE
ships containers problems orders voyages
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice

23. © 2019 IBM Corporation
Kafka log compaction for data replication
0
key:a
val:A1
1
key:b
val:B1
2
key:a
val:A2
3
key:c
val:C1
TOPIC
PARTITION
4
key:c
val:C2
PRODUCER
Consumer
takes latest
values and
builds own
data store
KEY VALUE
a A2
b <deleted>
c C2
SOURCE CHANGE
LOG
Read

24. © 2019 IBM Corporation
Kafka log compaction for data replication
0
key:a
val:A1
1
key:b
val:B1
2
key:a
val:A2
3
key:c
val:C1
TOPIC
PARTITION
4
key:c
val:C2
KEY VALUE
a A2
b <deleted>
c C2
2
key:a
val:A2
4
key:b
val:C2
TOPIC
PARTITION
(rewritten)
PERIODIC COMPACTION ELIMINATES
DUPLICATE KEYS TO MINIMIZE STORAGE

25. © 2019 IBM Corporation
Pattern – Sagas
Orchestration of multi-step operations across microservices
Data consistency across microservices without distributed transactions
Programming can be complex, particularly for failure compensation
EVENT BACKBONE
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
1 2 3 4 5

26. © 2019 IBM Corporation
EVENT BACKBONE
Pattern – Event sourcing
Every state change to an object is captured as an event stored in sequence
Replaying the events in order rebuilds the current state of the objects
By subscribing to the events, you get an evolving copy of the objects’ state
Query
handler
Command
handler
USER
Order1
Created
Order1
Updated
Order2
Created
Order1
Complete
Order3
Created
Order2
Canceled
Order3
Updated
Order3
Complete

27. © 2019 IBM Corporation
Pattern – Event sourcing with Kafka
Order2
Created
Order2
Updated
Order4
Created
Order2
Complete
Order4
Canceled
Order3
Created
Order3
Updated
Order6
Created
Order3
Complete
Order3
Created
Order6
Updated
Order1
Created
Order1
Updated
Order5
Created
Order1
Complete
Order7
Created
Order5
Canceled
Order7
Updated
Order7
Complete
TOPIC orders
PARTITION 0
TOPIC orders
PARTITION 1
TOPIC orders
PARTITION 2
CONSUMER GROUP A
p0, offset 7
p1, offset 3
p2, offset 5
CONSUMER
CONSUMER
CONSUMER
CONSUMER GROUP B
p0, offset 7
p1, offset 3
p2, offset 5
CONSUMER
CONSUMER

28. © 2019 IBM Corporation
Pattern – Command Query Responsibility Segregation (CQRS)
Atomic writes and efficient reads at the same time
API split into commands (change state) and queries (read only)
Complex in practice
EVENT BACKBONE
1 2 3 4 5
Fleet MS
WRITE
MODEL
Fleet MS
READ MODEL
Write-optimized
store
Read-optimized
store
Write API Read API

29. © 2019 IBM Corporation
Summary
Event-driven microservices offer an effective way to build loosely coupled applications
Techniques such as event storming can be used to derive the objects, commands and
events in an application
Patterns such as event sourcing and sagas make the complexity manageable

30. © 2019 IBM Corporation
Thank you
Andrew Schofield
Chief Architect, Event Streams
IBM Hursley Park
andrew_schofield@uk.ibm.com