Join us as we build a complete streaming application with KSQL. There will be plenty of hands-on action, plus a description of our thought process and design choices along the way. Look out for advice on best practices and handy tips and tricks as we go. This is part 2 out of 3 in the Empowering Streams through KSQL series.

1. 1

2. 2
Nick is one of the masterminds and co-creators of KSQL. He is also a technology
and product leader at Confluent, where he enjoys leveraging many years of
experience in the world of data and analytic systems to help design and explain
the power of a streaming platform for every business. Prior to Confluent, he led
the data platform group for a leading online real-estate seller and was chief
architect for a cloud-based financial analytics platform. His early career stretches
all the way back through multiple data warehouse and business intelligence
adventures to the green-screen days of mainframe banking systems.
Nick Dearden
Director of Engineering, Confluent
Hojjat started the KSQL project at Confluent and is the lead engineer for
the project. Before joining Confluent he has worked at NEC Labs,
Informatica, Quantcast and Tidemark on various big data management
projects. He has a PhD in computer science from UC Irvine where he
worked on scalable stream processing and publish/subscribe systems.
Hojjat Jafarpour
KSQL Project Lead, Confluent

3. 3
Housekeeping Items
● This session will last about an hour.
● This session will be recorded.
● You can submit your questions by entering them into the GoToWebinar panel.
● The last 10-15 minutes will consist of Q&A.
● The slides and recording will be available after the talk.

4. KSQLis the
Streaming
SQL Enginefor
Apache Kafka

5. KSQL Concepts
• Streams are first-class citizens
• Tables are first-class citizens
• All queries run until terminated
• Some queries are persistent

6. 6
What is it for ?
● Streaming ETL
○ Kafka is popular for data pipelines.
○ KSQL enables easy transformations of data within the pipe
CREATE STREAM vip_actions AS
SELECT userid, page, action FROM
clickstream c LEFT JOIN users u ON c.userid
= u.user_id
WHERE u.level = 'Platinum';

7. 7
What is it for ?
● Anomaly Detection
○ Identifying patterns or anomalies in real-time data, surfaced in milliseconds
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5
SECONDS)
GROUP BY card_number
HAVING count(*) > 3;

8. 8
What is it for ?
● Real Time Monitoring
○ Log data monitoring, tracking and alerting
○ Sensor / IoT data
CREATE TABLE error_counts AS
SELECT error_code, count(*)
FROM monitoring_stream
WINDOW TUMBLING (SIZE 1
MINUTE)
WHERE type = 'ERROR'
GROUP BY error_code;

9. 9
Running Example
Ecommerce Site HadoopEvent Stream Support

10. 10
Standing On the Shoulders of (Streaming) Giants
• Native, 100%-compatible Kafka integration
• Secure stream processing using Kafka’s security features
• Elastic and highly scalable
• Fault-tolerant
• Stateful and stateless computations
• Interactive queries
• Time model
• Supports late-arriving and out-of-order data
• Windowing
• Millisecond processing latency, no micro-batching
• At-least-once and exactly-once processing
guarantees

11. 11
Q & A

12. 12
Thank you for joining us!