You Put What in Your Stream?! Patterns and Practices for Event Design with Adam Bellemare

You Put WHAT in Your Stream?!
Patterns and Practices for Event Design

@AdamBellemare | developer.confluent.io
Who am I?
● Staff Technologist at Conﬂuent
● Previously a Data Platform Engineer
● Written a few Books
● Check out the Practical Data Mesh!
Released 2020
AVAILABLE NOW!

So you’re at an event-driven architectures talk…

“But what did you actually put in your event?”

The hand-waving event design technique
Source: NHLGIFs, apparently

Okay it didn’t work at all. And my arms are tired.

And no, not by going to 7th Normal Form

Schema-on-Read Doesn’t Work for Streams
A typical
“Schema-On-Read”
big data ﬁle system
Source: https://commons.wikimedia.org/wiki/File:Messy_storage_room_with_boxes.jpg

Bloopers
isScreenOn:

Bloopers
isScreenOn:
True, False

Bloopers
isScreenOn:
True, False, “Yes”, “No”

Bloopers
isScreenOn:
True, False, “Yes”, “No”, 1, 0

Bloopers
isScreenOn:
True, False, “Yes”, “No”, 1, 0, null

Bloopers
isScreenOn:
True, False, “Yes”, “No”, 1, 0, null
Source: Beaker

Per-Consumer Sensitivity to Bad Data
Stream
Stream

Bloopers
promotedContent:
True, False,

Bloopers
promotedContent:
True, False, 4291, null,

Bloopers
promotedContent:
True, False, 4291, null, “null”

Bloopers
promotedContent:
True, False, 4291, null, “null”
Source: Uh some kid?

Fixing the Bloopers
{
"type" : "record",
"name" : "SampleEvent",
"ﬁelds" : [
{ "name" : "isScreenOn" , "type" : "boolean" },
{ "name" : "promotedContent" , "type" : "boolean" }
]
}

Schemaless events
Source: Steve’s Disco and Dumpster Inferno Emporium

Use a Schema Registry
Conﬂuent Schema Registry
● Supports:
○ Avro
○ Protobuf
○ JSON Schema
● Supports Schema Evolution
● Serialization & Code Generators
● See the Schema Registry 101 Course on developer.conﬂuent.io

Factor 1:
Fact or Delta Event Types

Fact Events vs. Delta Events
Deltas Facts

Fact and Delta Event Types
Cart Fact Event item_added_to_cart
Delta Event
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"shipping": "express"
}
{
"cart_id": 3125123,
"item_id": 521,
"quantity": 1
}

When do you use Deltas, and when do you use Facts?

TIP: Ask yourself:
Are you transferring data?
OR
Are you notifying of an occurrence?

A lot of it depends on the boundaries
Internal World
External
World
Service
Boundary
(aka: Bounded
Context)
External
World
External
World

The Internal World
Internal World
Service
Boundary
(aka: Bounded
Context)

Data on the Inside
Tables
Stream Stream
Event Sourcing
Database-Backed System Event-Sourced System
Private and Purpose-Built for Internal Usage!

The External World
Internal World
External
World
External
World
External
World
Service
Boundary
(aka: Bounded
Context)

Data on the Outside
Stream
Stream
Stream
Stream
Stream Internal World
External
World
External
World
External
World

Events as Data Transfer Objects
Event Stream
Tables
Stream Stream
Database Backed
Event Sourcing
Create External Event
from Internal Model

Cart Fact Event (a DTO)
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
}
item_added_to_cart
{
"cart_id": 3125123,
"item_id": 521,
"quantity": 1
}
discount_code_applied
{
"cart_id": 3125123,
"discount_code": "SAVE-20-2022",
"discount_percent": "20"
}
Facts Model State – Deltas Model Change
Delta
Events

Deltas Are Used in Event Sourcing
(Data on the Inside)
Customer Product Quantity
Robert
Pants 1
T-Shirts 1
Hats 15
11:37 2 Pants added to cart
11:39 1 T-Shirt added to cart
11:41 1 Pants removed from cart
11:42 15 Hats added to cart
11:42 Apply Discount Code
Event
Time Events
Read
Apply in order to build state
Current Consumer State
Stream Table

Deltas Not Suitable for Building External State
(Data on the Outside)
11:37 2 Pants added to cart
11:39 1 T-Shirt added to cart
11:41 1 Pants removed from cart
11:42 15 Hats added to cart
11:42 Apply Discount Code
Event
Time Events
Read Duplicate
Cart Building
Logic
Independent Consumer
Services
Apply in order to build state
Apply in order
to build state
Stream

Use Case: General Alerting of Changes
item_added_to_cart
item_removed_from_cart
Consumers

What about communicating changes with facts?

Inferring Changes from Fact Events
{
"cart_id": 3125123,
"item_map": [ (521,1) ],
"shipping": "standard"
}
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
}
Cart Fact 1 Cart Fact 2

Facts with Before/After Fields
{
"before": {
"cart_id": 3125123,
"item_map": [ (521,1) ],
},
"after": {
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
}
}
Cart Fact Event

Fact Event Considerations
Event Size

Event Size Update Frequency

Event Size Update Frequency
Intent
?

Composite Events?

Composite Events: Facts with a Delta?
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"reason": "item_added_to_cart"
}
Composite Cart Event

Composite Events: Delta with some Facts?
{
"cart_id": 3125123,
"item_id": 923,
"quantity": 3,
"current_cart": {
"item_map": [ (521,1), (923,3) ],
}
}
item_added_to_cart

Facts are good for state transfer and ﬂexibility
Deltas are good for event sourcing
and notiﬁcations

Factor 2:
How normalized should your events be?

Highly Normalized Streams
Item Stream
Brand Stream
Rating Stream
Item
Brand
Rating
Connectors

Streaming Joins Can Be Costly
Item Stream
Brand Stream
Rating Stream
Same Joins!
Item
Brand
Rating
Consumers

Denormalizing Streams - Everybody’s doing it
Source: The 60s. Maybe the early 70s. That’s like 50 years ago…

Events are best served denormalized

Option 1: Outbox Table Pattern
Item
Brand
Connector
Outbox
Table
Enriched Item Stream
Transaction
Rating

Item
Brand
Rating
Connector
Outbox
Table
Transaction
Join
Triggers?

Item
Brand
Rating
Connector
Outbox
Table
Transaction
Extra Load
Join
Triggers?

Item
Brand
Rating
Connector
Outbox
Table
Transaction
Code
Extra Load
Join
Triggers?

Item
Brand
Rating
Connector
Outbox
Table
Transaction
Deletions
Extra Load
Join
Triggers?
Code

Option 2: Denormalize the Streams
Item Facts Stream
Brand Facts Stream
Rating Facts Stream
Enriched Item Facts Stream
Join events together
using ksqlDB

Option 2: Denormalize the Streams
Normalized
SELECT
I.ID,
I.PRICE,
I.NAME,
B.NAME
FROM ITEM I
INNER JOIN BRAND B
ON I.BRAND_ID = B.ID
EMIT CHANGES;
ksqlDB
Item Facts Stream
Brand Facts Stream
Enriched Item Facts Stream
Denormalized

Denormalization Considerations
Event Size

Denormalization Considerations
Event Size Join Trigger
Frequency

Factor 3:
Single vs. Multiple
Deﬁnitions per Stream

Use Case 1: Transferring State with Facts
Cart Fact Event
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
}
Single Type per Stream
Cart Facts
Consumers

Use Case: Transferring State with Facts
Cart Facts
Item Facts
Consumers

Single Fact Type per Stream
CREATE TABLE Items(
id BIGINT PRIMARY KEY,
price DECIMAL(10,2),
name STRING,
description STRING,
brand_id BIGINT
) WITH (
kafka_topic = 'items',
key_format = 'KAFKA',
value_format = 'AVRO');
Item Facts
ksqlDB

Best Practice: Record the Entire State in one Fact
Order
Cart
User
ksqlDB
Order_Cart
Cart
Order_User
User

Use Case 2: General Alerting of Changes
item_added_to_cart
Consumers

Remove
item
from
cart
Add
item
to
cart
Apply
discount
code
Remove
item
from
cart
Apply
discount
code
Remove
item
from
cart
Cart Events Stream
Use Case 3: Strict Order Processing

Multiple Types per Stream
Cart Events Stream
Consumer
item_added_to_cart
Use Case 3: Strict Order Processing

Can always split up the stream later
Cart Events Stream
ksqlDB
Application
item_added_to_cart

Facts:
Single Deﬁnition per Stream
Deltas:
Single or Multiple Deﬁnitions per Stream

Evolving your Data Model?

Versioned Streams for Breaking Changes
Cart Version 2
Cart Version 1
Producer
(Writes both for a period)

Versioned Streams for Breaking Changes
Cart Version 2
Producer

Deltas
Naming so far?
item_added_to_cart
Facts
Cart
Item
User

Streams Across the Business
Inventory
Orders
Shipments
Finance

Event Stream Naming
1) <domain>.<event-type>.<version>

Event Stream Naming
1) <domain>.<event-type>.<version>
Old Fact Stream Name:
Customers.Advertisement
New Fact Stream Names:
Customers.Advertisement.v2

2) <domain>.<service>.<event-type>.<version>
Event Stream Naming Including Service Name

2) <domain>.<service>.<event-type>.<version>
Event Stream Naming Including Service Name
Orders.Order_Placer_Service.Order.v1
Customers.Ad_Manager.Advertisement.v2
Customers.Ad_Budget_Manager.Ad_Promo_Budget.v2

Check out our
Data Mesh eBook!
AVAILABLE NOW!
Stay Tuned for
More Free Courses!

You Put What in Your Stream?! Patterns and Practices for Event Design with Adam Bellemare

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie You Put What in Your Stream?! Patterns and Practices for Event Design with Adam Bellemare

Ähnlich wie You Put What in Your Stream?! Patterns and Practices for Event Design with Adam Bellemare (20)

Mehr von HostedbyConfluent

Mehr von HostedbyConfluent (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)