1.
1
FaaS Composition using Kafka and CloudEvents
Neil Avery,
Office of the CTO,
@avery_neil {...}

2.
What is FaaS?
●
●
●
●
●
●
●
DRIVEN BY EVENTS
{...}

3.
Origins of FaaS
●
●
●
●
“It’s easy to run without managing infrastructure. It will take care of
managing, scaling, monitoring and logging for you. Your applications run
milliseconds after the event has been triggered and thousands of
functions can work in parallel. You run the code only when needed.”

4.
44

5.
55

6.
66
Microservice/FaaS/Stream-Processor
Single atomic unit
does one thing
like

7.
77
Composition
<events don’t exist in isolation>

8.
88
Composition

9.
99
Composition styles
1. Wiring
2. Chaining
3. Orchestration / Workflow
(Step functions, Durable Fns,
Zeebe, Apache Airflow)
4. Choreography

10.
1010
1. Wiring: Infrastructure events
Triggered by
S3 bucket, DynamoDB, SNS or CloudWatch

11.
1111
1. Wiring: REST Endpoint
Http Request payload

12.
1212
2. Call chaining
Send event from one Fn to other(s)
Sync or Async(s)
What could go wrong?
< debug, logs, testing, portability>

13.
1313
3. Orchestration / Workflow
Vendor: AWS Step functions, Azure Durable Fns
OpenSource: Zeebe, Apache Airflow, CNCF Serverless workflow spec

14.
14

15.
15

16.
1616
3. Orchestration using a DataFlow

17.
1717

18.
1818
Rate of adoption
● Serverless is the top-growing extended cloud service for the
second year in a row, with a 50 percent growth over 2018
24 to 36 percent adoption.
● Stream processing is tied for first, increasing from
20 to 30 percent adoption.
Rightscale cloud report 2019

19.
1919
What is stream processing?
Event stream processing, or ESP, is a set of technologies designed to assist the
construction of event-driven information systems. ESP technologies include event
visualization, event databases, event-driven middleware, and event processing
languages, or complex event processing (CEP).
SELECT DataStream
Orders.TimeStamp, Orders.orderId,
Orders.ticker,Orders.amount, Trade.amount
FROM Orders
JOIN Trades OVER (RANGE INTERVAL '1' SECOND
FOLLOWING)
ON Orders.orderId = Trades.orderId;
https://en.wikipedia.org/wiki/Event_stream_processing

20.
20
Streaming processing as SQL
Stream
processor
STREAM
/user-reg
FILTER
SELECT users > 18
PROJECT
SELECT user.name
JOIN
SELECT u.name, a.country
from user-reg u JOIN
address a WHERE u.id = a.id
GROUP BY (TABLE)
SELECT u.country,
count(u.name) FROM user-reg u
GROUP BY u.country
WINDOW (TABLE)
SELECT u.country, count(u.name)
FROM user-reg u WINDOW TUMBLING
(SIZE 1 MIN) group by u.country
STREAM
/address

21.
21
1. Stream processor runs ‘future’ on
local-state of item-bids
auction items
3. Rejoined
removes item
4. Stream triggers FaaS
complete item processor
bid-history
Stream table join
5. FaaS Notify all bidders
bid-notifications
2.
Rewrite ‘completed’ item
status to retrigger join
item status

22.
22
Kafka Streams
● Data driven (dataflow)
● Partition aware
● Scale
● Resilient
● Delivery guarantees
● Replay events and time-travel
[Java, Scala]

23.
23
{
StreamsBuilder builder = new StreamsBuilder();
KStream<String, Payment> incoming = builder.stream(TASK_TOPIC);
incoming.filter(
(key, payment) -> payment.isComplete()).to(TASK_COMPLETE);
…
Topology gateTopology = builder.build();
streams = new KafkaStreams(gateTopology, streamsConfig);
streams.start();
}

24.
24
{
StreamsBuilder builder = new StreamsBuilder();
KStream<String, Payment> incoming = builder.stream(TASK_TOPIC);
incoming.flatMapValues(payment -> {
ArrayList<String> fanTasks = new ArrayList<>();
for (int i = 0; i < payment.getAmount(); i++) {
fanTasks.add("fan-" + i);
}
return fanTasks;
}
).to(TASK_FANOUT); flatMapValues: Many V to the same K
Or
flatMap: Many K,V to one K,V pair

25.
25
builder.stream(TASK_FANOUT)
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.windowedBy(SessionWindows.with(Duration.ofMinutes(1)))
.reduce(new Reducer<String>() {
public String apply(String agg1, String newValue) {
return agg1 + newValue;
}
})
.toStream()
.map((key, value) ->
new KeyValue<>(key.key() + "@" +
key.window().start() + "->" +
key.window().end(), value))
.to(TASK_REDUCE);
or: groupBy(KeyValueMapper)

26.
{...}
{...}
{...}
{ bid:100; sold:1000;}
/bid-history
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
Calculating percentiles using a ‘unit-of-work’ pattern
{
digest.merge(digest)
}
Stream processor with
a session window to
join related events by
key (k, v[1-10])
Stream
processor
fan out is natural (data-driven)
OR
use a Stream processor flatMap()

27.
27
Stream processing ‘FaaS’ Patterns
● Filter (gate)
● Transform
● Fan-out (one to many)
● Fan-in (many to one - join)
● Branch/Choose (if ‘n’ then)
● Aggregate
● ...more...

28.
2828
A Stream processing DataFlow is
both workflow and choreography
(EDA, SEDA, Event-driven, streaming)

29.
2929
Choreography
1. Fire and forget / Async events
2. Producers emit state (not commands)
3. Consumers react to events
4. There is no master plan, no central brain
LOOSE COUPLING

30.
30
Choreography

31.
31
Choreography and Evolvability

32.
32
An Event
records the fact that something happened
32
A good
was sold
An invoice
was issued
A payment
was made
A new customer
registered

33.
33
Capture behavior

34.
34
Evolvability
user experience?
how many users
affected?
has it happened
before?
new old
supports data change, logic change, logic extension, schema
evolution, loose coupling, add processors, A/B path

35.
35
is a distributed event streaming platform
Publish & Subscribe
to Events
Store
Events
Process & Analyze
Events

36.
3636
Choreography with Kafka

37.
37
payment.incoming.pre
2.
Payment
Inflight
payment.incoming
1.
{...}
Fraudulent
Check
“FaaS Connector”
a bridge between Kafka topic(s) and serverless function(s)

38.
3838
Confluent Hub
confluent.io/hub

39.
39
FaaS Connector modes
1. Sync:
● fire and queue response
● preserve order
2. Async:
● fire and forget
● order agnostic
https://www.confluent.io/hub
→ AWS Lambda Connector (preview)
→ Azure Functions Connector (preview)
→ GCP Functions Connector (preview)
<confluent-platform>
../bin/connect-standalone ./etc/json-standalone.properties ./etc/AwsLambdaSinkConnector.properties

40.
40
payment.incoming.pre
2.
Payment
Inflight
payment.incoming
1.
{...}
Fraudulent
Check
Connector
Bridge between Kaka topic
and Fn invocation
Producer
Write back to Kafka?

41.
{...}
{...}
{...}
{...}
bad stream

42.
42
Bad stream?
As we say in IT...

43.
4343

44.
44
Explain FaaS Connector modes
1. Sync: use case (ordered)
- state change analytics: add then remove values; i.e. CDC
- by topic partition
2. Async: use case (not ordered)
- compute intensive analytics; what-if scenario analysis
- machine learning training
- ETL
<confluent-platform>
../bin/connect-standalone ./etc/json-standalone.properties ./etc/AwsLambdaSinkConnector.properties

45.
45
Kafka Streams is a Java app….
It’s not very ‘Cloud native’ - without
45
Kubeless Cloud Foundry Knative GraalVm Quarkus

46.
4646
Composition at scale?

47.
{faas}
Central nervous system
appappappapp
Payments Department 2
{faas}appappappapp
Department 3 Department 4
Choreography
Choreography
Orchestration
Hierarchical topic naming

48.
4848
At Scale: CloudEvents

49.
{faas}
What is the message format James?
appappappapp
Payments Department 2
I’m so confused!

50.
50
https://cloudevents.io/

51.
51

52.
52
● CNCF
● Version 1.0 October 2019
● Specification (an Event envelope)
● Transport Bindings (Kafka, AMQP, HTTP others)
● SDKs: Java, .NET, Go-lang, Javascript etc
● Useful for exposing to external apps (Events as APIs)
CloudEvents is a specification for describing event data in common formats
to provide interoperability across services, platforms and systems.

53.
5353
CloudEvent semantics
● Self describing / self contained / stand alone
● Suppressed / non-intruisive
● Language, transport, data format independent
● Propagation
● Extensions
● Batteries included

54.
54
{
"specversion" : "0.4-wip",
"type" : "com.github.pull.create",
"source" :
"https://github.com/cloudevents/spec/pull",
"subject" : "123",
"id" : "A234-1234-1234",
"time" : "2018-04-05T17:31:00Z",
"comexampleextension1" : "value",
"comexampleextension2" : {
"othervalue": 5
},
"datacontenttype" : "text/xml",
"data" : "<much wow="xml"/>"
}

55.
55
CloudEvents Producer Consumer
SerDes
Kafka Streams and
KSQL
SerDes
Kafka Connect
Convertor
Cloud Function
Connectors
AWS, GCP, Azure
and the Kafka
Ecosystem

56.
56
KStream<String, CloudEvent<AttributesImpl, Payment>>
payments =
builder.stream(paymentTopic,
Consumed.with(StringSerde,
new CeBinarySerDes<>(Payment.class)));
payments.filter((k,v) -> {
return v.getData().get().getOrigin().contains("UK");
})
CloudEvents with Kafka Streams

57.
{faas}
Central nervous system
appappappapp
Payments Department 2
{faas}appappappapp
Department 3 Department 4
Choreography
Orchestration
Choreography
Orchestration
Zeebe
Hierarchical topic naming
Kafka Streams
Quarkus
GraalVM
KNative
CloudEvents
Confluent Cloud

58.
5858
Where to go from here
for more details on event-driven architectures with Kafka
@avery_neil

59.
59
Questions?
@avery_neil
“Journey to event driven” blog
1. Event-first thinking
2. Programming models
3. Serverless
4. Pillars of event-streaming ms’s
https://bit.ly/2tFfU84
or @avery_neil twitter profile

60.
60
@avery_neil

61.
61
Cloud Native Kafka Streams (wow!)
61
1. StreamsTopology.java → docker-compose up --build. → topology.q-native-container
2. Create a Knative service descriptor topology-java-quarkus.yaml
3. Run locally: ./mvnw compile quarkus:dev
4. Publish docker push {username}/topology-java-quarkus
5. Deploy kubectl apply --filename service.yaml
6. Observe:
kubectl get ksvc topology-java-quarkus
NAME URL :
topology-java-quarkus http://topology-java-quarkus.default.1.2.3.4.xip.io
https://quarkus.io/guides/kafka-streams-guide https://knative.dev/community/samples/serving/helloworld-java-quarkus/