Cloud Native Night March 2020, Mainz: Talk by Perry Krol (@perkrol, Confluent)
=== Please download slides if blurred! ===
Abstract: Proven approaches such as service-oriented and event-driven architectures are joined by newer techniques such as microservices, reactive architectures, DevOps, and stream processing. Many of these patterns are successful by themselves, but they provide a more holistic and compelling approach when applied together. In this session Confluent will provide insights how service-based architectures and stream processing tools such as Apache Kafka® can help you build business-critical systems. You will learn why streaming beats request-response based architectures in complex, contemporary use cases, and explain why replayable logs such as Kafka provide a backbone for both service communication and shared datasets.
Based on these principles, we will explore how event collaboration and event sourcing patterns increase safety and recoverability with functional, event-driven approaches, apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with microservices and SOA using patterns such as “inside out databases” and “event streams as a source of truth”.
2. 2
Confluent Community
Slack Channel
Over 10,000 Kafkateers are
collaborating every single day on the
Confluent Community Slack channel!
cnfl.io/community-slack
Subscribe to the
Confluent blog
Get frequent updates from key
names in Apache Kafka®
on best
practices, product updates & more!
cnfl.io/read
Welcome to the Mainz
Cloud Native Night Meetup
with Apache Kafka® !
Zoom open at 19:00
19:00PM - 19:10PM
Virtual Cheers and Networking
19:10PM - 19:15PM
Welcome and Intro
20:00PM - 20:00PM
Concepts & Patterns for
Streaming Services with Kafka
Perry Krol
11. 1111
Competitive Landscape
Non event-streaming
(TIBCO, IBM MQ, Solace, ...)
Event-streaming
Non-Kafka
(Kinesis, Pulsar)
Kafka
Open Source
(Apache Kafka)
Commercial Offerings
Non-Confluent
AWS MSK
Cloudera
Red Hat
Aiven
...
Confluent
DRAFT - For
Internal Enablement
Only
1414
Record Keys & Ordering
Record keys determine the partition with the default kafka partitioner
Keys are used in the default partitioning algorithm:
partition = hash(key) % numPartitions
19. 19
Confluent Hub
Online library of pre-packaged and
ready-to-install extensions or add-ons
for Confluent Platform and Apache
Kafka®:
● Connectors
● Transforms
● Converters
Easily install the components that
suit your needs into your local
environment with the Confluent Hub
client command line tool . https://hub.confuent.io
27. 27
CREATE STREAM vip_actions AS
SELECT userid,
page,
action
FROM clickstream c
LEFT JOIN users u
ON c.userid = u.user_id
WHERE u.level = 'Platinum’
EMIT CHANGES;
Simple SQL syntax for expressing reasoning along and
across data streams.
You can write user-defined functions in Java
Stream Processing with KSQL
29. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!
Do you think that’s a table
you are querying?
30. 30
Alice + €50
The Stream-Table Duality
Stream
(payments)
Table
(balance)
time
Alice €50
Bob + €18
Alice €50 Alice €50
Bob €18
Alice + €25
Alice €50
Bob €18
Alice €75
Bob €18
Alice – €60
Alice €75
Bob €18
Alice €15
Bob €18
31. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!
The truth is the log.
The database is a cache
of a subset of the log.
—Pat Helland
Immutability Changes Everything
http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
32. 32
KSQL for Real-Time Monitoring
32
• Log data monitoring, tracking and alerting
• syslog data
• Sensor / IoT data
CREATE TABLE error_counts AS
SELECT error_code, count(*)
FROM monitoring_stream
WINDOW TUMBLING (SIZE 1 MINUTE)
WHERE type = 'ERROR'
GROUP BY error_code;
33. 33
KSQL for Streaming ETL
33
CREATE STREAM engine_oil_pressure_readings AS
SELECT r.deviceid, r.reading, r.timestamp
d.sensor_type, d.uom, d.component
FROM sensor_readings r
LEFT JOIN device_master d
ON r.deviceid = d.id
WHERE d.component = ‘Engine’
AND d.sensor_type = ‘Oil Pressure’
EMIT CHANGES;
Joining, filtering, and aggregating streams of event data
35. 35
KSQL is a stream processing technology
As such it is not yet a great fit for:
Ad-hoc queries
● No indexes yet in KSQL
● Kafka often configured to retain
data for only a limited span of
time
BI reports (Tableau etc.)
● No indexes yet in KSQL
● No JDBC
● Most BI tools don’t understand
continuous, streaming results
37. 3737
What are microservices?
Microservices are a software development
technique - a variant of the service-oriented
architecture (SOA) architectural style that
structures an application as a collection of
loosely coupled services.
https://en.wikipedia.org/wiki/Microservices
41. 4141
● Scaling is hard
● Handling state is hard
● Sharing, coordinating is hard
● Run a database in each microservice - is hard
What have we learned about microservices?
46. 4646
Events
Why do you care?
Loose coupling, autonomy, evolvability, scalability, resilience, traceability, replayability
EVENT-FIRST CHANGES HOW YOU
THINK ABOUT WHAT YOU ARE BUILDING
...more importantly...
47. 4747
old world : event-driven architectures
new world: event-streaming
architectures
63. 63
Payments system: bounded context
[1] How much is being processed?
Expressed as:
- Count of payments inflight
- Total $ value processed
[2&3] Update the account balance
Expressed as:
- Debit
- Credit [4] Confirm successful payment
Expressed as:
- Total volume today
- Total $ amount today
72. 7272
Our mental model: Abstraction as an Art
Chained/Orchestrated
Bounded contexts
Stream processor
Stream
Event
Pillars
Business function Control plane Instrumentation Operations
Bounded context
73. 7373
Key takeaway (state)
Event streamingdriven microservices are the new atomic unit:
1. Provide simplicity (and time travel)
2. Handle state (via Kafka Streams)
3. Provide a new paradigm: convergent data and logic processing
Stream
processor
74. 7474
Key takeaway (complexity)
● Event-Streaming apps: model as bounded-context dataflows, handle
state & scaling
● Patterns: Build reusable dataflow patterns (instrumentation)
● Composition: Bounded contexts chaining and layering
● Composition: Choreography and Orchestration
75. 7575
Questions?
“Journey to event driven” blog
1. Event-first thinking
2. Programming models
3. Serverless
4. Pillars of event-streaming ms’s
79. 79Confluent are giving new users $50 of free usage per month for their first 3 months
Sign up for a Confluent Cloud
account
Please bear in mind that you will
be required to enter credit card
information but will not be charged
unless you go over the $50 usage
in any of the first 3 months or if
you don’t cancel your subscription
before the end of your promotion.
Here’s advice on how to use this promotion to try Confluent Cloud for free!
You won’t be charged if you don’t
go over the limit!
Get the benefits of Confluent
Cloud, but keep an eye on your
your account making sure that you
have enough remaining free
credits available for the rest of
your subscription month!!
Cancel before the 3 months end If
you don’t want to continue past
the promotion
If you fail to cancel within your first
three months you will start being
charged full price. To cancel,
immediately stop all streaming
and storing data in Confluent
Cloud and email cloud-
support@confluent.io
bit.ly/TryConfluentCloudAvailable on
bit.ly/TryConfluentCloud
80. 80
A Confluent community catalyst is
a person who invests relentlessly in
the Apache Kafka® and/or
Confluent communities.
Massive
bragging rights
Access to the
private MVP
Slack channel
Special swagThe recognition
of your peers
Direct interaction
with Apache Kafka
contributors as well as
the Confluent founders
at special events
Free pass for
Kafka Summit SF
Nominate yourself or a peer at
CONFLUENT.IO/NOMINATE
81. 8181
Want to host or speak at
one of our meetups?
Please contact community@confluent.io
and we will make it happen!