What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Building event-driven (Micro)Services
with Apache Kafka
Guido Schmutz
Voxxed Days Banff – 27.10.2018
@gschmutz guidoschmutz.wordpress.com

2. Guido Schmutz
Working at Trivadis for more than 21 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz

3. COPENHAGEN
MUNICH
LAUSANNE
BERN
ZURICH
BRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than
600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 million
Financially self-supporting and
sustainably profitable
Experience from more than 1,900
projects per year at over 800
customers

4. Agenda
1. Where do we come from?
2. Using Microservices
3. Can we do (even) better?
4. Kafka for Event-Driven (Micro)Services
5. What about streaming sources?
6. What about integrating legacy applications?
7. What about (historical) data analytics?
8. Summary

5. Where do we come from?

6. Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
sync request/response

7. Shop UI App
Business
Activity Service
SOA Approach
Contract-first
Web Services
Technical layers
offer their own
interfaces
Reuse on each
level
Lower layer
often wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

8. Shop UI App
Business
Activity Service
Virtualized SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
X

9. Using Microservices

10. Microservice Approach
Tightly Scoped behind clear
interfaces
Responsible for managing their
own data (not necessarily the
infrastructure)
Should be highly decoupled
Independently deployable, self-
contained and autonomous
SOA done right ?!
Tightly Scoped
Responsible for managing their own data
Highly decoupled
Independently deployable, self-contained
and autonomous
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST

11. Synchronous World of Request-Response leads to tight,
point-to-point couplings
problem in lower end of chain have a ripple
effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State

12. Microservice Approach
with API Gateway
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway

13. Microservice Approach with Side Car (i.e. K8s & Istio)
Side-car can provide:
• retry
• load-balancing
• circuit switch, throttling
• security
• but still sync request/response
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Side
Car
Side
Car
Side
Car

14. Microservice Approach with Side Car (i.e. K8s & Istio)
Side-car and it’s advanced routing
capabilities can be used to switch to new
service version
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3 – v1
{ }
API Logic
State
Side
Car
Side
Car
Service 3 – v2
{ }
API Logic
State

15. Side Car provides abstraction ….
Service 1
{ }
API Logic
State
Service 2
{ }
API Logic
State
Service 4
Logic
State
Service 3
{ }
API Logic
State
{ }
API
New Service
Logic
State
{ }
API
but we still have to change the invoking
service …. If another service requires the
same information

16. Can we do (even) better?

17. Three mechanisms through which services can interact
Request-Driven (Imperative) Event Driven (Functional)
Service
Logic
State
Consume Event
“Event Ordered”
OrderEvent
Command
”Order IPad”
boolean order(IPad)
Publish Event
”Event Validated”
OrderValidatedEvent
Query
”Retrieve my Orders
List<Orders> getAllOrders(for)
Event Broker
Service
Logic
State

18. Event-Driven (Async)
Microservice Approach
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Event
Store
sync request/response
async request/response
async, event pub/sub

19. Kafka for Event-Driven
(Micro)Services

20. Apache Kafka – A Streaming Platform
High-Level Architecture
Distributed Log at the Core
Scale-Out Architecture
Logs do not (necessarily) forget

21. Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL)
log.retention.{ms | minutes | hours}
3. Size based
log.retention.bytes
4. Log compaction based
(entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181
--create --topic customers
--replication-factor 1
--partitions 1
--config cleanup.policy=compact

22. Keep Topics in Compacted Form
0 1 2 3 4 5 6 7 8 9 10 11
K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
Offset
Key
Value
3 4 6 8 9 10
K1 K3 K4 K5 K2 K6
V4 V5 V7 V9 V10 V11
Offset
Key
Value
Compaction
V1
V2
V3 V4
V5
V6
V7
V8 V9
V10
V11
K1
K3
K4
K5
K2
K6

23. Demo
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Microservice
{ }
Order API
Order
Order Logic
REST REST
Event
Hub
Customer
Schema
Registry
Schema
Customer
https://github.com/gschmutz/various-demos/tree/master/event-driven-microservices

24. Adding a new service by
bootstrapping from Event Hub
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Microservice
{ }
Order API
Order
Order Logic
REST REST
Event
Hub
Customer
Customer Microservice
{ }
Customer API CustomerCustomer Logic
REST
Schema
Registry
Schema
Customer

25. What about streaming sources?

26. How to work with
streaming data sources
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event
Store
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Event Stream

27. Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Streaming Processing & Microservices Architecture
BI Tools
SQ
L
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Analytics
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Service

28. Apache Kafka – scalable message processing and
more!
Source
Connector
trucking_
driver
Kafka Broker
Sink
Connector
Stream
Processing
Schema
Registry
Kafka Kafka

29. Kafka Connect - Overview
Source
Connector
Sink
Connector

30. Stream Processing Capability - Windowing
Computations over events done using
windows of data
Due to size and never-ending nature of it,
it’s not feasible to keep entire stream of
data in memory
A window of data represents a certain
amount of data where we can perform
computations on
Windows give the power to keep a
working memory and look back at recent
data efficiently
Time
Stream of Data Window of Data

31. Sliding Window (aka Hopping
Window) - uses eviction and
trigger policies that are based on
time: window length and sliding
interval length
Fixed Window (aka Tumbling
Window) - eviction policy always
based on the window being full and
trigger policy based on either the
count of items in the window or
time
Session Window – composed of
sequences of temporarily related
events terminated by a gap of
inactivity greater than some
timeout
Stream Processing Capability - Windowing
Time TimeTime

32. Stream Processing Capability - Joining
Challenges of joining streams
1. Data streams need to be aligned as they
come because they have different timestamps
2. since streams are never-ending, the joins
must be limited; otherwise join will never end
3. join needs to produce results continuously as
there is no end to the data
Stream to Static (Table) Join
Stream to Stream Join (one window join)
Stream to Stream Join (two window join)
Stream-to-
Static Join
Stream-to-
Stream
Join
Stream-to-
Stream
Join
Time
Time
Time

33. Stream Processing Capability - State Management
Necessary if stream processing use case
is dependent on previously seen data or
external data
Windowing, Joining and Pattern
Detection use State Management behind
the scenes
State Management services can be made
available for custom state handling logic
State needs to be managed as close to
the stream processor as possible
Options for State Management
How does it handle failures? If a machine
crashes and the/some state is lost?
In-Memory
Replicated,
Distributed
Store
Local,
Embedded
Store
Operational Complexity and Features
Low high

34. Kafka Streams - Overview
Designed as a simple and lightweight
library in Apache Kafka
no other dependencies than Kafka
Supports fault-tolerant local state
Supports Windowing (Fixed, Sliding and
Session) and Stream-Stream / Stream-
Table Joins
Millisecond processing latency, no
micro-batching
At-least-once and exactly-once
processing guarantees
KTable<Integer, Customer> customers =
builder.stream(”customer");
KStream<Integer, Order> orders =
builder.stream(”order");
KStream<Integer, String> enriched =
orders.leftJoin(customers, …);
joined.to(”orderEnriched");
trucking_
driver
Kafka Broker
Java Application
Kafka Streams

35. KSQL - Overview
STREAM and TABLE as first-class
citizens
• STREAM = data in motion
• TABLE = collected state of a stream
Stream Processing with zero coding
using SQL-like language
Built on top of Kafka Streams
Interactive (CLI) and headless (command
file)
ksql> CREATE STREAM customer_s
WITH (kafka_topic=‘customer',
value_format=‘AVRO');
Message
----------------
Stream created
ksql> SELECT * FROM customer_s
WHERE address->country = ‘Switzerland’;
...
trucking_
driver
Kafka Broker
KSQL Engine
Kafka Streams
KSQL CLI Commands

36. What about integrating legacy
applications?

37. Data Store
Integrate existing systems through CDC
Customer
Event Store
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
Results
Stream Processor
Reference /
Models
Dashboard
Capture changes directly on database
Change Data Capture (CDC) => think like a
global database trigger
Transform existing systems to event
producer

38. Legacy Microservice
Change Data Capture (CDC) with Kafka
RDBMS cdc-source trucking_
driver
Customer Topic
elasticsearch-
sink
NoSQL

39. Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Integrate existing systems through CDC
BI Tools
SQ
L
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Processor
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Service
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture

40. What about (historical) data
analytics?

41. Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
D
ata
Flow
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
SQL
Export
SearchEvent
Store
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture
File Import / SQL Import

42. Summary

43. Hadoop Clusterd
Hadoop Cluster
Big Data
Summary
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Event
Hub
D
ata
Flow
D
ata
Flow
Change
Data
Capture
Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Stream Processing
Microservices

44. Summary
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into
• commands
• events
• queries
• Kafka is well suited as an event broker / event store
• brings many more interesting features beyond just “message passing”

45. References
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc

46. Technology on its own won't help you.
You need to know how to use it properly.