Speaker: Tim Vincent, Systems Engineer, Confluent

1. Apache Kafka and KSQL in Action : Let’s
Build a Streaming Data Pipeline!
timv@confluent.io
confluent.io/ksql

2. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 2
• Systems Engineer @ Confluent
• Worked with databases since 1995
• C, ESQL/C Developer (a rather poor one!)
• Informix RDBMS
• TimesTen In-memory RDBMS
• Oracle
• DataStax, Apache Cassandra
• Confluent, Apache Kafka
$ whoami

3. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 3
App App App App
search
HadoopDWH
monitoring security
MQ MQ
cache
cache
A bit of a mess…

4. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 4
Kafka is an Event Streaming Platform
KAFKA
DWH Hadoop
App
App App App App
App
App
App
request-response
messaging
OR
stream
processing
streaming data pipelines
changelogs

5. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 5
Analytics - Database Offload
HDFS / S3 /
BigQuery etc
RDBMS
CDC

6. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 6
Stream Processing with Apache Kafka and KSQL
order events
customer
customer orders
Stream
Processing
RDBMS CDC

7. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 7
Real-time Event Stream Enrichment
order events
customer
Stream
Processing
customer orders
RDBMS
<y>
CDC

8. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 8
Transform Once, Use Many
order events
customer
Stream
Processing
customer orders
RDBMS
<y>
New App
<x>
CDC

9. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 9
Transform Once, Use Many
order events
customer
Stream
Processing
customer orders
RDBMS
<y>
HDFS / S3 / etc
New App
<x>
CDC

10. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 10
Rating
events
Join events to
users, and filter
Push notification
to Slack
Operational
Dashboard
Data
Lake
User
data
Let’s Build It!

11. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 11
Rating
events
Join events to
users, and filter
Push notification
to Slack
Operational
Dashboard
Data
Lake
User
data
RDBMS
S3/HDFS/
SnowflakeDB
etc
Elasticsearch
App
App
Producer API
Consumer API
Let’s Build It!

12. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 12
Rating
events
Join events to
users, and filter
Push notification
to Slack
Operational
Dashboard
Data
Lake
User
data
RDBMS
S3/HDFS/
SnowflakeDB
etc
Elasticsearch
App
App
Producer API
Consumer API
Kafka Connect
Kafka
Connect
Kafka
Connect
Kafka
Connect

13. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 13
Streaming Integration with Kafka Connect
Kafka Brokers
Kafka Connect
Tasks Workers
Sources Sinks
Amazon S3
syslog
flat file
CSV
JSON MQTT
MQTT

14. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 14
✓ Fault tolerant and automatically load balanced
✓ Extensible API
✓ Single Message Transforms
✓ Part of Apache Kafka, included in
Confluent Open Source
Reliable and scalable integration of Kafka with other systems – no coding required.
{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:mysql://localhost:3306/demo?user=rmoff&password=foo",
"table.whitelist": "sales,orders,customers"
}
https://docs.confluent.io/current/connect/
✓ Centralized management and configuration
✓ Support for hundreds of technologies
including RDBMS, Elasticsearch, HDFS, S3
✓ Supports CDC ingest of events from RDBMS
✓ Preserves data schema
Kafka Connect

15. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 15
Kafka Connect + Schema Registry = WIN
RDBMS
Avro
Message
Elasticsearch
Schema
RegistryAvro
Schema
Kafka
Connect
Kafka
Connect

16. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 16
Kafka Connect + Schema Registry = WIN
RDBMS
Elasticsearch
Schema
RegistryAvro
Schema
Kafka
Connect
Kafka
Connect
Avro
Message

17. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 17
Confluent Hub
hub.confluent.io
• One-stop place to discover and
download :
• Connectors
• Transformations
• Converters

18. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 18
MySQL DebeziumKafka Connect
Producer API
Demo Time!

19. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 19
Rating
events
Join events to
users, and filter
Push notification
to Slack
Operational
Dashboard
Data
Lake
User
data
RDBMS
S3/HDFS/
SnowflakeDB
etc
Elasticsearch
App
App
Producer API
Consumer API
Let’s Build It!
Kafka
Connect
Kafka
Connect
Kafka
Connect

20. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 20
Rating
events
Join events to
users, and filter
Push notification
to Slack
Operational
Dashboard
Data
Lake
User
data
RDBMS
S3/HDFS/
SnowflakeDB
etc
Elasticsearch
App
App
Producer API
Consumer API
KSQL
Kafka
Connect
Kafka
Connect
Kafka
Connect
KSQL

21. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql
Declarative
Stream
Language
Processing
KSQLis a

22. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql
KSQLis the
Streaming
SQL Enginefor
Apache Kafka

23. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql
KSQL in Development and Production
Interactive KSQL
for development and testing
Headless KSQL
for Production
Desired KSQL queries
have been identified
REST
“Hmm, let me try
out this idea...”

24. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql
KSQL for Streaming ETL
CREATE STREAM vip_actions AS
SELECT userid, page, action
FROM clickstream c
LEFT JOIN users u
ON c.userid = u.user_id
WHERE u.level = 'Platinum';
Joining, filtering, and aggregating streams of event data

25. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql
KSQL for Anomaly Detection
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;
Identifying patterns or anomalies in real-time data,
surfaced in milliseconds

26. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql
KSQL for Real-Time Monitoring
• Log data monitoring, tracking and alerting
• syslog data
• Sensor / IoT data
CREATE STREAM SYSLOG_INVALID_USERS AS
SELECT HOST, MESSAGE
FROM SYSLOG
WHERE MESSAGE LIKE '%Invalid user%';
http://cnfl.io/syslogs-filtering / http://cnfl.io/syslog-alerting

27. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql
CREATE STREAM views_by_userid
WITH (PARTITIONS=6, REPLICAS=5,
VALUE_FORMAT='AVRO',
TIMESTAMP='view_time') AS
SELECT * FROM clickstream
PARTITION BY user_id;
KSQL for Data Transformation
Make simple derivations of existing topics from the command line

28. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 28
MySQL DebeziumKafka Connect
Producer API
Elasticsearch
Kafka Connect

29. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 29
Producer API
{
"rating_id": 5313,
"user_id": 3,
"stars": 4,
"route_id": 6975,
"rating_time": 1519304105213,
"channel": "web",
"message": "worst. flight. ever. #neveragain"
}
POOR_RATINGS
Filter all ratings where STARS<3
CREATE STREAM POOR_RATINGS AS
SELECT * FROM ratings WHERE STARS <3
Demo Time!

30. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 30
Do you think that’s a table
you are querying?

31. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 31
The Table Stream Duality
Account ID Balance
12345 €50Account ID Amount
12345 + €50
12345 + €25
12345 -€60
Account ID Balance
12345 €75
Account ID Balance
12345 €15
Time
Stream Table

32. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 32
The truth is the log.
The database is a cache
of a subset of the log.
—Pat Helland
Immutability Changes Everything
http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Photo by Bobby Burch on Unsplash

33. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 33
Kafka Connect
Producer API
{
"rating_id": 5313,
"user_id": 3,
"stars": 4,
"route_id": 6975,
"rating_time": 1519304105213,
"channel": "web",
"message": "worst. flight. ever. #neveragain"
}
{
"id": 3,
"first_name": "Merilyn",
"last_name": "Doughartie",
"email": "mdoughartie1@dedecms.com",
"gender": "Female",
"club_status": "platinum",
"comments": "none"
}
RATINGS_WITH_CUSTOMER_DATA
Join each rating to customer data
CREATE STREAM RATINGS_WITH_CUSTOMER_DATA AS
SELECT * FROM RATINGS LEFT JOIN CUSTOMERS
ON R.ID=C.ID;

34. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 34
Kafka Connect
Producer API
{
"rating_id": 5313,
"user_id": 3,
"stars": 4,
"route_id": 6975,
"rating_time": 1519304105213,
"channel": "web",
"message": "worst. flight. ever. #neveragain"
}
{
"id": 3,
"first_name": "Merilyn",
"last_name": "Doughartie",
"email": "mdoughartie1@dedecms.com",
"gender": "Female",
"club_status": "platinum",
"comments": "none"
}
RATINGS_WITH_CUSTOMER_DATA
Join each rating to customer data
UNHAPPY_PLATINUM_CUSTOMERS
Filter for just PLATINUM customers
CREATE STREAM UNHAPPY_PLATINUM_CUSTOMERS AS
SELECT * FROM RATINGS_WITH_CUSTOMER_DATA
WHERE STARS < 3
Demo Time!

35. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 35
Kafka Connect
Producer API
{
"rating_id": 5313,
"user_id": 3,
"stars": 4,
"route_id": 6975,
"rating_time": 1519304105213,
"channel": "web",
"message": "worst. flight. ever. #neveragain"
}
{
"id": 3,
"first_name": "Merilyn",
"last_name": "Doughartie",
"email": "mdoughartie1@dedecms.com",
"gender": "Female",
"club_status": "platinum",
"comments": "none"
}
RATINGS_WITH_CUSTOMER_DATA
Join each rating to customer data
RATINGS_BY_CLUB_STATUS_1MIN
Aggregate per-minute by CLUB_STATUS
CREATE TABLE RATINGS_BY_CLUB_STATUS AS
SELECT CLUB_STATUS, COUNT(*)
FROM RATINGS_WITH_CUSTOMER_DATA
WINDOW TUMBLING (SIZE 1 MINUTES)
GROUP BY CLUB_STATUS;

36. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 36

37. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 37
Confluent Community :
Apache Kafka with a bunch of cool stuff! For free!
Database Changes Log Events loT Data Web Events …
CRM
Data Warehouse
Database
Hadoop
Data
Integration
…
Monitoring
Analytics
Custom Apps
Transformations
Real-time Applications
…
Apache Open Source Confluent Open Source Confluent Enterprise
Confluent Platform
Confluent Platform
Apache Kafka®
Core | Connect API | Streams API
Data Compatibility
Schema Registry
Monitoring & Administration
Confluent Control Center | Security
Operations
Replicator | Auto Data Balancing
Development and Connectivity
Clients | Connectors | REST Proxy | CLI
Apache Open Source Confluent Open Source Confluent Enterprise
SQL Stream Processing
KSQL

38. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 38
Free Books!
https://www.confluent.io/apache-kafka-stream-processing-book-
bundle

39. timv@confluent.io
https://www.confluent.io/ksql
http://cnfl.io/slack

40. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 40
• Embrace the Anarchy : Apache Kafka's Role in Modern Data Architectures Recording & Slides
• Look Ma, no Code! Building Streaming Data Pipelines with Apache Kafka and KSQL
• Steps to Building a Streaming ETL Pipeline with Apache Kafka and KSQL Recording & Slides
• https://www.confluent.io/blog/ksql-in-action-real-time-streaming-etl-from-oracle-transactional-data
• https://github.com/confluentinc/ksql/
Useful links

41. @rmoff / Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! http://cnfl.io/ksql 41
• CDC Spreadsheet
• Blog: No More Silos: How to Integrate your Databases with Apache Kafka and CDC
• #partner-engineering on Slack for questions
• BD team (#partners / partners@confluent.io) can help with introductions on a given sales op
Resources
#EOF