1.
Building a Scalable, Modern Cyber Intelligence
Platform with Apache Kafka®
Presenter: Jac Noel
Kafka Summit Europe – May 2021

2.
IT@Intel 2
Notices and Disclaimers
This presentation is for informationalpurposes only. INTEL MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.
Intel, the Intel logo, Intel Core, Intel Optane and Xeon are trademarks of Intel Corporation or its subsidiaries.
Other names and brands may be claimed as the propertyof others.
Copyright © 2021, Intel Corporation.All rights reserved.
2

3.
IT@Intel 3
Jac Noel has over 25 years of Information Technology and
Cyber Security experience across the military, government,
and corporate environments.
He started his technical career in the United States Air Force
supporting defense intelligence systems for the AF mission in
EMEA. He has spent the past 20 years serving in various
technical roles in Intel’s IT organization. He’s currently serving
as a Security Solutions Architect focusing on security
intelligence and response capabilities. He’s the lead architect
for Intel’s Cyber Intelligence Platform (CIP), which is a next-
gen architecture combining a data lake, message bus, stream
processing, machine-learning, orchestration, and workflow
automation into a single platform.
Jac holds a Bachelor of Science degree from Chico State
University and has earned numerous professional certifications
over the years, including CISSP, GCFW, CCNA, and MCSE.
He’s also a proud inventor, patent holder, and author of several
white papers.
Jac Noel
Security Solutions Architect

4.
IT@Intel 4
Intel Information Security’s Mission
4
Our mission is to keep Intel
legal and secure.
This mission is never
“done.”
Best ways to measure our success:
 Reduce Mean Time to Detect (MTTD)
and Mean Time to Respond (MTTR)
 Identify and implement more effective
preventative controls
 Improve our agility to respond to new and
changing threats and regulations

5.
IT@Intel 5
API Data Virtualization Layer
Information Security
Business Role
Incident Response
Vulnerability
Management
Compliance
Enforcement
Data Protection
Threat Intelligence
Common Work
Surface Layer
Query
Search
Reporting
Dashboards
Visualizations
Analytics Workbench
Workflow Automation
Infrastructure
Clients
Servers
Network
Infrastructure
Other Data
Sources
Data
Blueprint
Security
Data Lake
Control Layer
Security Event Management
User Event Behavior Analytics
Vulnerability Scanning
Threat Intelligence
Advanced Analytics
Deceptions
Intrusion Detection
Firewalls
Intrusion Prevention
Endpoint Detection and Response
Data Loss Prevention
Intrusion Scanning
Connectors
Enterprise Security Message Bus
Topics, Publish/Subscribe, Transform, Enrich, Filter, Join
CyberIntelligencePlatform-ReferenceArchitecture
A platform that supports our entire InfoSec organization
5

6.
IT@Intel 6
High Performance Compute & Storage
BU
Partners
IT
Ops
Partners
Confluent Platform
Message Bus
Stream Processing
Cyber Intelligence Platform - Solution Stack
Our partners produce and consume data, too!
6

7.
7
Cyber Intelligence Platform – Solution Stack (cont)
Built with industry leading technologies Splunk and Kafka

8.
IT@Intel 8
The Power of the Kafka Bus
No Message Bus
 Point to point, complex
 Slow to implement
 Increased technical debt due to tightly-coupled solutions and brittle integrations
 No orchestration (custom-code it, multiple times)
 No transformation (custom-code it, multiple times)
 Slow to move data between multiple capabilities
 Harder to monitor and govern
With Message Bus
 Data Transformation (enrich, aggregate, normalize)
 Near real-time integration (streaming)
 Resilient, robust, scalable, available
 Orchestrate multiple activities in one place
 Cross-capability consumption
 Platform independent, plug and play
 Apps loosely coupled but tightly integrated
 Common architectural element for large enterprises
App App App App App App
App App App App App App
App App App App App App
App App App App App App
Message Bus
Abstraction, Resiliency, Scalability, Availability
Transform Orchestrate

9.
IT@Intel 9
Improving Data Availability with Confluent MRC
9
Single Cluster
Data Center 3
Producers Consumers
Streaming Apps
Consumers Producers
Data Center 1
Leaders (ISR)
Zookeeper 1
Zookeeper 2
Broker n
Broker 2
Broker 1
Broker 3
…
Mirroring
Data Center 2
Observers
Zookeeper 3
Zookeeper 4
Broker n
Broker 2
Broker 1
Broker 3
…
Zookeeper 5

10.
IT@Intel 10
Asynchronous Replication for Faster Recovery
10
Single Cluster
Data Center 3
Producers Consumers
Streaming Apps
Consumers Producers
Data Center 1
Zookeeper 1
Zookeeper 2
Broker n
Broker 2
Broker 1
Broker 3
…
Mirroring
Data Center 2
Leaders (ISR)
Zookeeper 3
Zookeeper 4
Broker n
Broker 2
Broker 1
Broker 3
…
Zookeeper 5
Confluent Platform with Multi Region Clusters

11.
IT@Intel 11
TLS
Confluent Control
Center
LDAP/TLS Schema
Registry
SASL
Digest MD5
Admin User SASL
TLS Digest MD5
Zookeeper 1
Broker Cluster
TLS
Zookeeper 2
Connectors
SASL
Digest MD5 Zookeeper 3
Authorization
ACL Zookeeper
Broker 1
Producers
(Client App) Broker 2
TLS Stream Processor 1
Broker 3 Stream Processor 2
… TLS
Consumers Stream Processor 3
Broker n
(Client App) TLS
Stream
Processor
Securing Our Confluent Platform
11

12.
IT@Intel 12
Monitoring Our Kafka Clusters
12
Our C3 server requires Intel 2nd gen Xeon processors for high-performance compute
and Intel Optane DC SSDs for low latency and high-endurance storage.
Kafka
Admins
All-in-One Kafka Cluster
Confluent Control Center Server (C3)
(Broker, ZooKeeper, Connect, Kafka Streams)
Kafka Streams App
“Stream Processor”
C3 Web App
Consumers UI
Trouble-
shooting
Producers
Producers
Kafka
Production
Monitoring Data
Metrics Data
Metrics
Reporter
Monitoring
Interceptor
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Topics
Consumers
Metrics Topic
Monitoring Topic Transformed Topics
Health
Monitoring

13.
IT@Intel 13
Managing Vulnerabilities with Stream Processing
13
Confluent Platform
Producers Kafka Streams API
Stream Processing
Kafka Bus
Vulnerability
Topic Filter
Vulnerabilities by
Business Unit
IP Address
Range Topic
Join Asset
Asset Inventory
Topic
Ownership with Consumers
Vulnerable Assets
BU #1’s
Vulnerabilities Topic
Data Lake
BU Partners
BU #2’s
Vulnerabilities Topic
IT Partners
BU #3’s
Vulnerabilities Topic
SIEM
Vulnerabilities
with Owners Topic Enforcement
SOAR
Scanning
Engine
IP Address
Management
Asset Management
Inventory
Vulnerabilities
Asset configuration, CVEs, CVSS
IP Address Ranges
Ownership, Business Units
Asset Ownership

14.
IT@Intel 14
Kafka Maturity
Timeline
14
Acquire once-consume many
Integration efficiency
Remove the noise, and
duplication
Cost savings for downstream consumers
Join multiple sources
Contextually rich + clean data downstream
ACQUIRE
DATA
FILTERING
ENRICHMENT
SUMMARIZATION
ADVANCED Autonomous Actions
e.g. Cluster analysis, ML
Produce summary statistics
State information, performance benefit
and downstream cost savings

15.
IT@Intel 15
Kafka By The Numbers
15
20+
TB/DAY
135+
32+
CONSUMERS DATA
SOURCES
320+
TOPICS
90+
PRODUCERS
>18B
EVENTS/DAY
Kafka
by the
Numbers
~8 trillion events indexed by Splunk in 2020

16.
IT@Intel 16
Kafka - Benefits to Intel
16
KAFKA LEADERSHIP
THROUGH CONFLUENT
EXPERTISE
GENERATES
CONTEXTUALLY RICH
DATA
MODERN
ARCHITECTURE WITH
THRIVING COMMUNITY
GLOBAL
SCALE AND REACH
OPERATE ON DATA
IN STREAM
ECONOMIES
OF SCALE
REDUCE TECHNICAL
DEBT AND
DOWNSTREAM COSTS
ALWAYS
ON

17.
IT@Intel 17
People + Technology + Data
Transforming How Information Security Works
17
Reduced Risk
to Intel
Greater Insight
and Tighter
Collaboration
Highly
Integrated
and
Automated
A Force
Multiplier
Faster
Detection and
Response
Speaking a
Common
Language
A Platform
for the Future

18.
IT@Intel 18
Additional Resources
18
Solution Brief and
Reference
Architecture

19.
19
IT@Intel
Questions & Answers