During this session we’ll discuss the pros and cons of a new structured streaming data processing model in Spark and a nifty way of enhancing Spark with SnappyData, an open-source framework providing great features for both persistent and in-motion data analysis. Based on a real-life use case, where we designed and implemented a streaming application filtering, consuming and aggregating tons of events, we will talk the role of the persistent back-end and stream processing integration in the real-time applications in terms of performance, robustness and scalability of the solution.

1. FEBRUARY 22, 2018, WARSAW
ENHANCING SPARK
INCREASE STREAMING CAPABILITIES OF YOU APPLICATIONS
Kamil Folkert, Tomasz Mirowski

2. BUSINESS RULES - ALERTS
ANALYTICAL MODEL

3. SOURCE LAYER
• SECURITY LOGS
MESSAGE LAYER
• TIME WINDOW
PROCESSING LAYER
DATA
STREAM
FILTERING
AGGREGATED
WINDOW
PERPETUALLY
GROWING
REFERENCE
DATA
EXTERNAL
ACCESS

4. SOURCE
SYSTEM
EVENT
EVENT
EVENT

5. SOURCE
SYSTEM
EVENT
EVENT
EVENT

6. JOINING
STREAMS
TEMPORARY
FILES
TIME
OUTPUT
METHODS

7. EVENT
EVENT
EVENT
SNAPPYDATA

8. SNAPPYDATA: REAL TIME + OLTP

9. SNAPPYDATA
SNAPPYDATA

12. DATA
STREAM
FILTERING
AGGREGATED
WINDOW
PERPETUALLY
GROWING
REFERENCE
DATA
EXTERNAL
ACCESS

13. val eventsDF = jedis.hgetAll(”eventDictionary")
.toList.toDF.withColumnRenamed("_1","key").withColumnRenamed("_2", "value")
application_name log_type source_host ldap_group status
application1 application_server server1 userGroup security_breach
… … … … …
DATA
STREAM
FILTERING
AGGREGATED
WINDOW
PERPETUALLY
GROWING
REFERENCE
DATA
EXTERNAL
ACCESS

14. snsc.sql("create stream table applicationStream (”time_stamp timestamp," +
" application_name string, log_type string,” +
…
) " +
" using directkafka_stream options(" +
" rowConverter 'io.streaming.RowsConverter' ," +
s" kafkaParams 'metadata.broker.list->$brokerList;auto.offset.reset->largest'," +
s" topics '$kafkaTopic')"
)
DATA
STREAM
FILTERING
AGGREGATED
WINDOW
PERPETUALLY
GROWING
REFERENCE
DATA
EXTERNAL
ACCESS

15. val resultStream: SchemaDStream = snsc.registerCQ(
"select application_name, log_type" +
"count(source_host) as source" +
"from applicationStream window (duration 1 seconds, slide 1 seconds)" +
"group by application_name, log_type"
)
DATA
STREAM
FILTERING
AGGREGATED
WINDOW
PERPETUALLY
GROWING
REFERENCE
DATA
EXTERNAL
ACCESS

16. resultStream.foreachDataFrame(df => {
val dfApplications = df.groupBy("application_name")
.agg(sum(”log_type"))
dfApplications.write.insertInto("application_name")
val resultStream: SchemaDStream = snsc.registerCQ(
"select application_name, log_type" +
"count(source_host) as source" +
"from applicationStream window (duration 1 seconds, slide 1 seconds)" +
"group by application_name, log_type"
)
DATA
STREAM
FILTERING
AGGREGATED
WINDOW
PERPETUALLY
GROWING
REFERENCE
DATA
EXTERNAL
ACCESS

17. snsc.sql("create table eventTable(application string, event string)
using row")
resultStream.foreachDataFrame(df => {
val result = df.collect()
...
val stmt = conn.prepareStatement("put into eventTable
(application_name, log_type) values (?,?+(nvl(
(select application_name from eventTable where
application_name= ?),0)))")
DATA
STREAM
FILTERING
AGGREGATED
WINDOW
PERPETUALLY
GROWING
REFERENCE
DATA
EXTERNAL
ACCESS

18. DATA
STREAM
FILTERING
AGGREGATED
WINDOW
PERPETUALLY
GROWING
REFERENCE
DATA
EXTERNAL
ACCESS

19. FEBRUARY 22, 2018, WARSAW
ENHANCING SPARK
INCREASE STREAMING CAPABILITIES OF YOU APPLICATIONS
Kamil Folkert, Tomasz Mirowski