Streaming SQL on Apache Kafka using Flink SQL. We demo Eventador SQLStreamBuilder application and discuss processing streams of data using SQL.

1. Streaming SQL w/ Kafka and Flink
Kenny Gorman
Co-Founder and CEO
www.eventador.io

2. Intro
● Co-Founded Eventador, we are team of streaming data nerds
● 20 years of data experience, especially SQL and relational databases
● Co-Founded a MongoDB as a service company (ObjectRocket)
● Eventador.io: Streaming SQL Engine—SQLStreamBuilder and Fully Managed Apache Flink
● SQL is experiencing a resurgence—and is an important part to our future

3. Use Case - Aircraft ADSB data
Let’s use a abstract use case to discuss Streaming SQL shall we?
Aircraft emit radio signals with data about their flight. Automatic dependent surveillance – broadcast
(ADS–B) is a standardized data format that allows for the transmission of data in real time from aircraft to
ground, and from aircraft to aircraft. Aircraft transmit and receive this data as a constant stream of data
once a second.
Check out: https://eventador.io/blog/planestream-the-ads-b-datasource/

4. Erik’s serious about this..
Me
Erik

5. Dump1090
antirez

6. Use Case - Aircraft ADSB data
{
"speed": "390",
"lon": "",
"flight": "",
"timestamp": 1566145812,
"lat": "",
"counter": 61,
"icao": "A5739A",
"msg_type": "4",
"timestamp_verbose": "2019-08-18
16:30:12.846211",
"vr": "1728",
"track": "285",
"altitude": ""
}
{
"speed": "",
"lon": "",
"flight": "",
"timestamp": 1566145821,
"lat": "",
"counter": 118,
"icao": "A5739A",
"msg_type": "5",
"timestamp_verbose": "2019-08-18
16:30:21.314870",
"vr": "",
"track": "",
"altitude": "15300"
}
{
"speed": "",
"lon": "-98.24491",
"flight": "",
"timestamp": 1566145821,
"lat": "30.27925",
"counter": 121,
"icao": "A5739A",
"msg_type": "3",
"timestamp_verbose": "2019-08-18
16:30:21.769869",
"vr": "",
"track": "",
"altitude": "15300"
}

7. Use Case - Aircraft ADSB data
{
"speed": "390",
"lon": "",
"flight": "",
"timestamp": 1566145812,
"lat": "",
"counter": 61,
"icao": "A5739A",
"msg_type": "4",
"timestamp_verbose": "2019-08-18
16:30:12.846211",
"vr": "1728",
"track": "285",
"altitude": ""
}
{
"speed": "",
"lon": "",
"flight": "",
"timestamp": 1566145821,
"lat": "",
"counter": 118,
"icao": "A5739A",
"msg_type": "5",
"timestamp_verbose": "2019-08-18
16:30:21.314870",
"vr": "",
"track": "",
"altitude": "15300"
}
{
"speed": "",
"lon": "-98.24491",
"flight": "",
"timestamp": 1566145821,
"lat": "30.27925",
"counter": 121,
"icao": "A5739A",
"msg_type": "3",
"timestamp_verbose": "2019-08-18
16:30:21.769869",
"vr": "",
"track": "",
"altitude": "15300"
}
Nulls
Key
Kafka
TimeStamp

8. Data flow
Some sort of
processing
apparatus

9. Stream processor
We like Apache Flink because….
● Killer state management
● Rich API
● Production grade
● Killer scalability model
● Uses Apache Calcite for SQL
● Good community
https://eventador.io/blog/apache_flink_checkpoints_and_savepoints/

10. Data flow

11. Table API + SQL

12. SQL on streams
Streaming SQL is very similar to the SQL you know and love, but has some fundamental differences:
● Relations are expressed over time, not point in time
● Results don’t use a cursor, rather, a endless query emitting results
● Time bounded queries matter
○ Event Time vs Processing Time
● It’s super easy to iterate and reason about your data!
● SELECT * FROM foo WHERE 1=0;

13. Creating processors - java/scala
https://github.com/kgorman/TrafficAnalyzer/

14. Creating processors - the Schema meat
// define a schema
String[] fieldNames = { "flight", "timestamp_verbose", "msg_type", "track",
"timestamp", "altitude", "counter", "lon",
"icao", "vr", "lat", "speed" };
TypeInformation<?>[] dataTypes = { Types.INT, Types.SQL_TIMESTAMP, Types.STRING, Types.STRING,
Types.SQL_TIMESTAMP, Types.STRING, Types.STRING, Types.STRING,
Types.STRING, Types.STRING, Types.STRING, Types.STRING };

15. Creating processors - the SQL meat
// define a simple filtering SQL statement
String sql = "SELECT icao, lat, lon, altitude FROM flights WHERE altitude <> ‘’";
// or maybe something more complicated..
String sql = “SELECT icao, max(altitude)
FROM flights
GROUP BY tumble(timestamp, INTERVAL ‘5’ SECOND), icao”;
// apply that statement to the table
tableEnv.registerTableSource("flights", kafkaTableSource);
Table result = tableEnv.sql(sql);

16. Data flow

17. Using SQL against aircraft data
{
"speed": "390",
"lon": "",
"flight": "",
"timestamp": 1566145812,
"lat": "",
"counter": 61,
"icao": "A5739A",
"msg_type": "4",
"timestamp_verbose": "2019-08-18
16:30:12.846211",
"vr": "1728",
"track": "285",
"altitude": ""
}
{
"speed": "",
"lon": "",
"flight": "",
"timestamp": 1566145821,
"lat": "",
"counter": 118,
"icao": "A5739A",
"msg_type": "5",
"timestamp_verbose": "2019-08-18
16:30:21.314870",
"vr": "",
"track": "",
"altitude": "15300"
}
{
"speed": "",
"lon": "-98.24491",
"flight": "",
"timestamp": 1566145821,
"lat": "30.27925",
"counter": 121,
"icao": "A5739A",
"msg_type": "3",
"timestamp_verbose": "2019-08-18
16:30:21.769869",
"vr": "",
"track": "",
"altitude": "15300"
}

18. Using SQL against aircraft data
{
"output_timestamp": 1566145821,
"icao": "A5739A",
"speed": "390",
"altitude": "15300"
}

19. Streaming SQL - some hints
-- eventTimestamp is the Kafka timestamp
-- as unix timestamp. Magically added to every schema.
SELECT max(eventTimestamp) FROM solar_inputs;
-- make it human readable
SELECT CAST(max(eventTimestamp) AS varchar) as TS FROM
solar_inputs;
-- dete math with interval
SELECT * FROM payments
WHERE eventTimestamp > CURRENT_TIMESTAMP-interval '10' second;
-- detect multiple auths in a short window and
-- send to lock account topic/microservice
SELECT card,
MAX(amount) as theamount,
TUMBLE_END(eventTimestamp, interval '5' minute) as ts
FROM payments
WHERE lat IS NOT NULL
AND lon IS NOT NULL
GROUP BY card, TUMBLE(eventTimestamp, interval '5' minute)
HAVING COUNT(*) > 4 -- >4==fraud
-- unnest each array element as separate row
SELECT b.*, u.*
FROM bgp_avro b,
UNNEST(b.path) AS u(pathitem)
-- union two different virtual tables
SELECT * FROM clickstream
WHERE useragent = 'Chrome/62.0.3202.84 Mobile Safari/537.36'
UNION ALL
SELECT * FROM clickstream
WHERE useragent = 'Version/4.0 Chrome/58.0.3029.83 Mobile Safari/537.36'
-- inline math
SELECT (amount+10)*upcharge AS total_amount
FROM payments
WHERE account_type = 'merchant'
-- convert C to F
SELECT (temp-32)/1.8 AS temp_fahrenheit
FROM reactor_core_sensors;
-- join multiple streams
SELECT o.name,
SUM(d.clicks),
HOP_END(r.eventTimestamp, interval '20' second, interval '40'
second)
FROM click_stream o JOIN orgs r ON o.org_id = r.org_id
JOIN models d ON d.org_id = r.org_id
GROUP BY o.name,
HOP(r.eventTimestamp, interval '20' second, interval '40' second)

20. BUT!
● Java coding required, easy to get started, gets tricky as you grow
● CICD pipeline
● Not iterative
● It’s a bit like punch cards eh?

21. Demo

22. We are hiring!
● We are hiring!
○ Full stack engineers
○ Java/Scala engineers
● Eventador.io SQLStreamBuilder Early Access ← join!
kgorman@eventador.io
If you liked this, you might also like: Secrets of the sky

23. Thank You
hello@eventador.io