76. 76
Materialized View
Create via a query
(select * from Orders
Where Region == âUSAâ)
Base
Table
Materialized Views in a DB
77. 77
Insert data Materialized View
Create via a query
(select * from Orders
Where Region == âUSAâ)
Query is rerun on data as it arrives
Base
Table
Materialized Views in a DB
78. 78
In practice often more complex
select Order.region,
count(order.quantity)
from Orders, Product, Customer
where Product.group = âHousholdâ
and Customer.type = external
group by Order.region
Orders Product Customer
View
Read
Optimized
79. 79
Same problem but with services
select Order.region,
count(order.quantity)
from Orders, Product, Customer
where Product.group = âHousholdâ
and Customer.type = external
group by Order.region
My Service View
Read
Optimized
Orders
Service
Product
Service
Customer
Service
86. 86
High Throughput Data Movement
Service
Service
Service
Service instance 1
Service instance 2
Service instance 3
Service instance 4
Kafka Cluster
(many machines)
92. 92
Take only the data you need
{âStock Inventoryâ: {
âIdâ: âFoo1234â,
âVendorâ: âFoo Industriesâ,
âDescriptionâ: âThis is a âŚâ,
âDelivery Categoryâ: âNDâ,
âStock Statusâ: [
âItems in Stockâ: 53,
âItems on Orderâ: 0
]âŚetcâŚ}
93. 93
Data Movement
Be realistic:
â˘âŻ Network is no longer the bottleneck
â˘âŻ Indexing is:
â˘âŻ In memory indexes help
â˘âŻ Keep datasets focussed
94. 94
12. Use the log instead as a âdatabaseâ
(for data-on-the-inside)
104. 104
(2) Stateless, Data Enabled Processor
Similar to star schema
â˘âŻ Facts are stream
â˘âŻ Dimensions are GKTables
e.g. Enrich Orders with Customer Info & Account details
Stream
GKTable
GKTable
Stream-Table
Join
105. 105
(3) Gates
e.g. rules engines or event correlation
When Order & Payment then âŚ
Stream 1
Window
Window Stream-Stream
Join
Stream 2
106. 106
(4) Stateful Processor
e.g.
- Average order amount by region
- Posting Engine (reverse replace of previous position)
State store
backed up to Kafka
113. 113
All order logic,
stateless service
Orders-by-Customer view
(KStreams + Queryable State API)
UI Service
Stream
Maintenance &
Monitoring
Highly available, stateful service
UI uses Orders-by-
Customer View directly
via Queryable State
History Service pushes data to
a local Elastic Search Instance
Orders
Service
Derived
View
UI joins data
Tables & Streams
Fulfillment Service
Analytics
OrdersProduct Custo-
mers KTables
KTables KTables
Schemas