Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 25 Anzeige

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020

Herunterladen, um offline zu lesen

SIEM platforms are essential to the new cybersecurity paradigm and data collection layer is a very important piece of it.

When you deliver a new platform, you can easily get lost in a variety of different vendors and solutions, too many challenges are facing. What if I change vendors, will I keep my data? How to feed multiple tools with the same data? How to collect data from custom apps and services? How to pay less for an expensive platform? How to keep data without a huge cost?

Join us if you are looking for the answers. In this session, you will learn how we replaced the vendor-provided data collection layer with kafka connect and the lessons we learnt. After the talk you will know:
- architecture and real-life examples of the flexible and highly available data collection platform
- custom connectors that do most of the work for us and how to extend the connectors to consume new data, we made them open sourced
- easy way to receive data from thousands of servers and many cloud services
- how to archive data at low cost

You will leave armed with a set of free tools and recipes to build a truly vendor-agnostic data collection platform. It will allow you to take you SIEM costs under control. You will feed your analytics tools with what they need and archive the rest at low cost. You will feed your SIEM smart!

SIEM platforms are essential to the new cybersecurity paradigm and data collection layer is a very important piece of it.

When you deliver a new platform, you can easily get lost in a variety of different vendors and solutions, too many challenges are facing. What if I change vendors, will I keep my data? How to feed multiple tools with the same data? How to collect data from custom apps and services? How to pay less for an expensive platform? How to keep data without a huge cost?

Join us if you are looking for the answers. In this session, you will learn how we replaced the vendor-provided data collection layer with kafka connect and the lessons we learnt. After the talk you will know:
- architecture and real-life examples of the flexible and highly available data collection platform
- custom connectors that do most of the work for us and how to extend the connectors to consume new data, we made them open sourced
- easy way to receive data from thousands of servers and many cloud services
- how to archive data at low cost

You will leave armed with a set of free tools and recipes to build a truly vendor-agnostic data collection platform. It will allow you to take you SIEM costs under control. You will feed your analytics tools with what they need and archive the rest at low cost. You will feed your SIEM smart!

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020 (20)

Anzeige

Weitere von HostedbyConfluent (20)

Aktuellste (20)

Anzeige

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020

  1. 1. Feed Your SIEM Smart with Kafka Connect Vitalii Rudenskyi Security Architect, McKesson Corp
  2. 2. Motivation and Background
  3. 3. Why We Started It’s not a mistake to make a mistake, but it’s a mistake to repeat the same mistake
  4. 4. Requirements ★ Logs collection to be vendor agnostic ★ Feed different analytics tools ★ Data filtering and cleaning ★ Existing data retention requirements ★ Scalable and highly available
  5. 5. Anticipated Challenges ★ Amount of data and number of sources ★ Variety of different formats ★ SaaS/PaaS applications and public Clouds ★ Cloud SIEM
  6. 6. Kafka Connect
  7. 7. The Three Keys That Open The Door
  8. 8. How to Collect Data
  9. 9. The Three Keys ‘Push’ Connector ‘Pull’ Connector Transformations Library
  10. 10. Push - NettySource Connector ★ Unified: “all-in-one” different transport and protocols ★ Scalable: multiple tasks supported ★ Available: works behind a LB (health checks supported) ★ Configurable: multiple protocols (plain text, syslog, http, snmp, netflow) ★ Customizable: custom implementations supported
  11. 11. NettySource Connector Common deployment model
  12. 12. Pull - PollableAPIClient Connector ★ Simple: easy new connectors development ★ Scalable: multiple tasks/partitions supported ★ Configurable: interval and scheduled (cron-style) polls, retry/backoff, resettable offsets ★ Customizable: custom implementations supported
  13. 13. PollableAPIClient Connector
  14. 14. PollableAPIClient Connector 25+ ApiClient implementations
  15. 15. Transformations Library ★ Can transform different parts of a kafka message ★ Supports “if” conditions
  16. 16. Takeaways
  17. 17. Highly Available NettySource Connector
  18. 18. Headers All The Way ★ Track origin of the data ★ Conditional routing ★ Tagging in ‘chained’ transformations ★ SIEM specific metadata
  19. 19. Syslog: Take Port 514 Under Control Multi-rules RegexRouter config
  20. 20. Data archiving solution ★ Pair of source and sink connectors ★ One connector for all topics ★ Compressed ★ Easy to restore
  21. 21. ★ Monitoring is essential ★ Keep original data ★ Be cautious when using “heavy” transformations in source connectors
  22. 22. 5 kafka clusters 20+ kafka connect clusters 530+ deployed connectors 7+TB of data daily
  23. 23. What is the next...
  24. 24. We share! ★ NettySource Connector https://github.com/vrudenskyi/kafka-connect-netty-source ★ PollableAPIClient Connector https://github.com/vrudenskyi/kafka-connect-pollable-source https://github.com/vrudenskyi/kafka-connect-api-clients ★ Transformations Library https://github.com/vrudenskyi/kafka-connect-transform
  25. 25. Q & Avrudenskyi@gmail.com https://www.linkedin.com/in/vrudenskyi/

×