1.
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Building event-driven (Micro)Services
with Apache Kafka
Guido Schmutz
Liverpool – 3.12.2018
@gschmutz guidoschmutz.wordpress.com

2.
Agenda
1. Where do we come from?
2. Using Microservices
3. Can we do even better?
4. How does Apache Kafka help?
5. What about streaming data sources?
6. What about integrating legacy applications?
7. What about (historical) data analytics?
8. Summary

3.
Guido Schmutz
Working at Trivadis for more than 22 years
Oracle Groundbreaker Ambassador & Oracle ACE Director
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
135th edition

4.
Where do we come from?

5.
Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
sync request/response

6.
Shop UI App
Business
Activity Service
SOA Approach
Contract-first
Web Services
Technical layers
offer their own
interfaces
Reuse on each
level
Lower layer
often wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

7.
Business
Activity Service
Virtualized SOA Approach
The raise of the
Enterprise Service
Bus (ESB)
Search BAS
Customer DAO
Order DAO
Order BAS
Business Entity
Service
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
Shop UI App
Shop UI UI Logic
GUI
Shop Web App
Shop UI UI Logic
GUI
X

8.
Shop UI App
Business
Activity Service
Orchestrated & Virtualized SOA Approach – Sync / Async
The raise of
orchestration engines
(BPEL & BPMN)
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
X
Orchestration

9.
Using Microservices

10.
Microservice Approach
Tightly Scoped behind clear
interfaces
Responsible for managing their
own data (not necessarily the
infrastructure)
Should be highly decoupled
Independently deployable, self-
contained and autonomous
Is it just SOA done right ?
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST

11.
Synchronous World of Request-Response leads to tight,
point-to-point couplings
problem in lower end of chain have a ripple
effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State
RESTRESTRESTREST
REST REST REST

12.
Microservice Approach
with API Gateway
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
X

13.
Microservice Approach with Side Car (i.e. K8s & Istio)
Side-car can provide:
• retry
• load-balancing
• circuit breaker, throttling
• security
• but still sync request/response
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Side
Car
Side
Car
Side
Car
REST
RESTRESTREST

14.
Microservice Approach with Side Car (i.e. K8s & Istio)
Side-car and it’s advanced routing
capabilities can be used to switch to new
service version
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3 – v1
{ }
API Logic
State
Side
Car
Side
Car
Service 3 – v2
{ }
API Logic
State
RESTREST REST
REST

15.
Side Car provides abstraction …. but
Service 1
{ }
API Logic
State
Service 2
{ }
API Logic
State
Service 4
Logic
State
Service 3
{ }
API Logic
State
{ }
API
New Service
Logic
State
{ }
API
we still have to change the invoking
service …. if another service requires the
same information
REST
REST REST REST REST

16.
Can we do even(t) better?

17.
Three mechanisms through which services can interact
Request-Driven (Imperative) Event Driven (Functional)
Service
Logic
State
Consume Event
“Event Ordered”
OrderEvent
Command
”Order IPad”
boolean order(IPad)
Publish Event
”Event Validated”
OrderValidatedEvent
Query
”Retrieve my Orders
List<Orders> getAllOrders(for)
Event Broker
Service
Logic
State

18.
Event-Driven (Async)
Microservice Approach
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Pub /
Sub
sync request/response
async request/response
async, event pub/sub
Customer
Mat View

19.
How does Apache Kafka help?

20.
Highly available, Pub/Sub infrastructure
Highly Scalable
Apache Kafka – A Streaming Platform
Distributed Log at the Core
Logs do not (necessarily) forget
• Never
• Time (TTL) or Size-based
• Log-Compacted based

21.
Example
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Microservice
{ }
Order API OrderOrder Logic
REST REST
Event
Hub
Customer
Mat View
Schema
Registry
Schema
Customer
(compacted)
https://github.com/gschmutz/various-demos/tree/master/event-driven-microservices

22.
Example
@Configuration
public class KafkaConfig {
private String bootstrapServers;
private String schemaRegistryURL;
@Bean
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryURL);
return props;
}
@Bean
public ProducerFactory<String, Customer> producerFactory() { .. }
@Bean
public KafkaTemplate<String, Customer> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
@Component
public class CustomerEventProducer {
@Autowired
private KafkaTemplate<String, Person> kafkaTemplate;
@Value("${kafka.topic.customer}")
String kafkaTopic;
public void produce(Customer customer) {
kafkaTemplate.send(kafkaTopic, customer.getId().toString(), customer);
}
}

23.
Adding a new service by
bootstrapping from Event Hub
Customer Search
Microservice
{ }
Customer API CustomerCustomer Logic
REST
Schema
Registry
Schema
Consume
from Offset 0
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Microservice
{ }
Order API OrderOrder Logic
REST REST
Event
Hub
Customer
Mat View
Customer
(compacted)

24.
What about streaming sources?

25.
How to work with
streaming data sources
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event
Hub
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Event Stream

26.
Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Streaming Processing & Microservices Architecture
BI Tools
SQ
L
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Analytics
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Hub
Service

27.
Apache Kafka – scalable message processing and
more!
Source
Connector
Kafka Broker
Sink
Connector
Stream
Processing
Schema
Registry
Kafka Kafka

28.
Kafka Connect - Overview
Source
Connector
Sink
Connector

29.
Kafka Streams - Overview
Designed as a simple and lightweight
library in Apache Kafka
no other dependencies than Kafka
Supports fault-tolerant local state
Supports Windowing (Fixed, Sliding and
Session) and Stream-Stream / Stream-
Table Joins
Millisecond processing latency, no
micro-batching
At-least-once and exactly-once
processing guarantees
KTable<Integer, Customer> customers =
builder.stream(”customer");
KStream<Integer, Order> orders =
builder.stream(”order");
KStream<Integer, String> enriched =
orders.leftJoin(customers, …);
joined.to(”orderEnriched");
trucking_
driver
Kafka Broker
Java Application
Kafka Streams

30.
KSQL - Overview
STREAM and TABLE as first-class
citizens
• STREAM = data in motion
• TABLE = collected state of a stream
Stream Processing with zero coding
using SQL-like language
Built on top of Kafka Streams
Interactive (CLI) and headless (command
file)
ksql> CREATE STREAM customer_s
WITH (kafka_topic=‘customer',
value_format=‘AVRO');
Message
----------------
Stream created
ksql> SELECT * FROM customer_s
WHERE address->country = ‘Switzerland’;
...
trucking_
driver
Kafka Broker
KSQL Engine
Kafka Streams
KSQL CLI Commands

31.
What about integrating legacy
applications?

32.
Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Integrate existing systems through CDC
BI Tools
SQ
L
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Processor
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Hub
Service
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture

33.
Data Store
Integrate existing systems through CDC
Customer
Event Hub
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
Results
Stream Processor
Reference /
Models
Dashboard
Capture changes directly on database
Change Data Capture (CDC) => think like a
global database trigger
Transform existing systems to event
producer

34.
Legacy Microservice
Change Data Capture (CDC) with
Kafka Broker and Kafka Connect
RDBMS cdc-source trucking_
driver
Kafka Broker
elasticsearch-
sink
NoSQL
Customer Topic

35.
Legacy
Microservice
Change Data Capture (CDC) with
Kafka Broker and Kafka Connect
RDBMS cdc-source
elasticsearch-
sink
NoSQL
Enhance
Kafka Broker
Customer Topic
CustomerEnhanced Topic
State

36.
What about (historical) data
analytics?

37.
Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
D
ata
Flow
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
SQL
Export
SearchEvent
Hub
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture
File Import / SQL Import

38.
Summary

39.
Hadoop Clusterd
Hadoop Cluster
Big Data
Summary
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Event
Hub
D
ata
Flow
D
ata
Flow
Change
Data
Capture
Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Stream Processing
Microservices

40.
Summary
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into
• commands
• events
• queries
• Kafka is well suited as an event store
• brings many more interesting features beyond just “message passing”
• Log compaction allows storing latest value per key for all time
• Does not provide all features for event sourcing (but maybe not needed)

41.
References
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc

42.
Technology on its own won't help you.
You need to know how to use it properly.