1.
Feed Your SIEM Smart
with Kafka Connect
Vitalii Rudenskyi
Security Architect, McKesson Corp

2.
Motivation and Background

3.
Why We Started
It’s not a mistake
to make a mistake,
but it’s a mistake
to repeat the
same mistake

4.
Requirements
★ Logs collection to be vendor agnostic
★ Feed different analytics tools
★ Data filtering and cleaning
★ Existing data retention requirements
★ Scalable and highly available

5.
Anticipated Challenges
★ Amount of data and number of sources
★ Variety of different formats
★ SaaS/PaaS applications and public Clouds
★ Cloud SIEM

6.
Kafka Connect

7.
The Three Keys That Open The Door

8.
How to Collect Data

9.
The Three Keys
‘Push’ Connector ‘Pull’ Connector
Transformations Library

10.
Push - NettySource Connector
★ Unified: “all-in-one” different transport and protocols
★ Scalable: multiple tasks supported
★ Available: works behind a LB (health checks supported)
★ Configurable: multiple protocols (plain text, syslog, http, snmp,
netflow)
★ Customizable: custom implementations supported

11.
NettySource Connector
Common deployment model

12.
Pull - PollableAPIClient Connector
★ Simple: easy new connectors development
★ Scalable: multiple tasks/partitions supported
★ Configurable: interval and scheduled (cron-style) polls,
retry/backoff, resettable offsets
★ Customizable: custom implementations supported

13.
PollableAPIClient Connector

14.
PollableAPIClient Connector
25+ ApiClient implementations

15.
Transformations Library
★ Can transform different parts of a kafka message
★ Supports “if” conditions

16.
Takeaways

17.
Highly Available NettySource
Connector

18.
Headers All The Way
★ Track origin of the data
★ Conditional routing
★ Tagging in ‘chained’ transformations
★ SIEM specific metadata

19.
Syslog: Take Port 514 Under Control
Multi-rules RegexRouter config

20.
Data archiving solution
★ Pair of source and sink connectors
★ One connector for all topics
★ Compressed
★ Easy to restore

21.
★ Monitoring is essential
★ Keep original data
★ Be cautious when using “heavy” transformations in
source connectors

22.
5 kafka clusters
20+ kafka connect clusters
530+ deployed connectors
7+TB of data daily

23.
What is the next...

24.
We share!
★ NettySource Connector
https://github.com/vrudenskyi/kafka-connect-netty-source
★ PollableAPIClient Connector
https://github.com/vrudenskyi/kafka-connect-pollable-source
https://github.com/vrudenskyi/kafka-connect-api-clients
★ Transformations Library
https://github.com/vrudenskyi/kafka-connect-transform

25.
Q & Avrudenskyi@gmail.com
https://www.linkedin.com/in/vrudenskyi/