Event Driven Services come in many shapes and sizes from tiny event driven functions that dip into an event stream, right through to heavy, stateful services which can facilitate request response. This practical talk makes the case for building this style of system using Stream Processing tools. We also walk through a number of patterns for how we actually put these things together.
77. 77
Insert data Materialized View
Create via a query
(select * from Orders
Where Region == ‘USA’)
Query is rerun on data as it arrives
Base
Table
Materialized Views in a DB
78. 78
In practice often more complex
select Order.region,
count(order.quantity)
from Orders, Product, Customer
where Product.group = ‘Houshold’
and Customer.type = external
group by Order.region
Orders Product Customer
View
Read
Optimized
79. 79
Same problem but with services
select Order.region,
count(order.quantity)
from Orders, Product, Customer
where Product.group = ‘Houshold’
and Customer.type = external
group by Order.region
My Service View
Read
Optimized
Orders
Service
Product
Service
Customer
Service
86. 86
High Throughput Data Movement
Service
Service
Service
Service instance 1
Service instance 2
Service instance 3
Service instance 4
Kafka Cluster
(many machines)
92. 92
Take only the data you need
{“Stock Inventory”: {
“Id”: “Foo1234”,
“Vendor”: “Foo Industries”,
“Description”: “This is a …”,
“Delivery Category”: “ND”,
“Stock Status”: [
“Items in Stock”: 53,
“Items on Order”: 0
]…etc…}
93. 93
Data Movement
Be realistic:
• Network is no longer the bottleneck
• Indexing is:
• In memory indexes help
• Keep datasets focussed
94. 94
12. Use the log instead as a ‘database’
(for data-on-the-inside)
104. 104
(2) Stateless, Data Enabled Processor
Similar to star schema
• Facts are stream
• Dimensions are GKTables
e.g. Enrich Orders with Customer Info & Account details
Stream
GKTable
GKTable
Stream-Table
Join
105. 105
(3) Gates
e.g. rules engines or event correlation
When Order & Payment then …
Stream 1
Window
Window Stream-Stream
Join
Stream 2
106. 106
(4) Stateful Processor
e.g.
- Average order amount by region
- Posting Engine (reverse replace of previous position)
State store
backed up to Kafka
113. 113
All order logic,
stateless service
Orders-by-Customer view
(KStreams + Queryable State API)
UI Service
Stream
Maintenance &
Monitoring
Highly available, stateful service
UI uses Orders-by-
Customer View directly
via Queryable State
History Service pushes data to
a local Elastic Search Instance
Orders
Service
Derived
View
UI joins data
Tables & Streams
Fulfillment Service
Analytics
OrdersProduct Custo-
mers KTables
KTables KTables
Schemas
126. 126
Tips and Tricks
• Take only the data you need today
• Limit the scope of request response interfaces
• Build small services. Build big ones too.
• Don’t fear the network. Fear the index.
• The Sync-Async divide
• Pure event driven with event streams and views.
• Map request response to event driven (Leverage CQRS)
• Too fine-grained services.
• Too much data
• KStreams rebalancing
• Need for automation