Serverless brings with it a groundswell of interest in event-driven architectures. Many advocate FaaS as a natural fit for stream processing and event-driven design, but is that just marketing speak?
In this talk, I will dispel the myths behind serverless and show how it can form part of a data flow, event-driven streaming architecture that could power ‘eBay’. I dive into how the data flow model works with streams and stream processors interacting to provide a network of related data and essentially Kafka becomes a deconstructed database. I then elaborate on FaaS, the challenges, and uses and how it is becoming an essential part of being a cloud-native story that fits with Streaming architectures. This is all tied together by walking through data flow functionality like item-placement, item-bidding, item-search, auction analytics and where FaaS fits into these!
13. 1. Any (data) application can be a data flow
2. A series of events and reacting to those events
3. Kafka then becomes a database
● User signup
● Item bid
● Item sold
● Notify user
Neil bid: 100
Michael bid: 200
Ben bid: 200
>> SOLD!
store all events
replay all events
transactional
etc
15. Shoulders of Streaming Giants
subscribe(), poll(), send(),
flush(), beginTransaction(), …
KStream, KTable, filter(),
map(), flatMap(), join(),
aggregate(), transform(), …
CREATE STREAM, CREATE TABLE,
SELECT, JOIN, GROUP BY, SUM, …
KSQL UDFs
Ease of Use
Flexibility
16. Stream
processor
STREAM
/user-reg
FILTER
SELECT users > 18
PROJECT
SELECT user.name
JOIN
SELECT u.name, a.country
from user-reg u JOIN
address a WHERE u.id = a.id
GROUP BY (TABLE)
SELECT u.country,
count(u.name) FROM user-reg u
GROUP BY u.country
WINDOW (TABLE)
SELECT u.country, count(u.name)
FROM user-reg u WINDOW TUMBLING
(SIZE 1 MIN) group by u.country
STREAM
/address
20. Model events not commands,
-> streams of events,
-> series of streams to model the domain
{
item-id: 389*
user: 100
type: bid
cat: bikes/mtb
region: dc-east
}
/bikes/mtb by item-id*
/bikes/ by dc-east‘-’item-id*
-
-
-
-
-
-
-
31. - Enrich users on signup (address validation)
- Geo-enrich items on placement (city, state, lat-lon cell identification)
- Notify users on item sold or reserve not met
- Perform analytics on auction when item ‘completes’
- Notify user of items-of-interest from their history when browsing
- Ad-placement analytics (watched items, interested items, users-purchased
- Monte-carlo auction simulations to guide users and calc item trending scores