1.
Network Traffic Analytics with
Hive LLAP and Druid

2.
Network Traffic Analytics
Deliver ultra-fast SQL analytics that can be consumed from the BI tool by security engineering team to get accelerated business
results
Opportunity for business users to explore and visualize streaming datasets with integration for various data sources and build
dashboards for different slices
Capability to run BI queries in just milliseconds over 1TB dataset
2
3
4
Data ingestion, integration and aggregation of network security traffic in Hadoop at near Real-Time1
High granular permission model on security datasets that allow intricate rules on accessibility for the datasets5
Near Real time monitoring of IP security traffic to identify and alert the unusual network activities across interfaces
within an organization.
Context:
Framework provides an unified platform for data storage and data enrichment by integrating the network traffic with
multiple sources (like Geo Location and Threat IP’s). Hadoop performs complex pre-computations and aggregations
to report and alert on IP threats.
Solution &
Benefits:

3.
• NetFlow routers to Kafka
• Enrich Kafka messages
with multiple Data Feeds
• Store the single flat table
for OLAP analysis
• Analyze network traffic
• Identify Threat IP’s
• User anomaly detection
• Create data visuals for
streaming datasets
• Provide near real time
network traffic
monitoring dashboards
and alerting
Network Traffic Analytics
3
Multi-step workflow to perform faster Analytics
on network traffic data
Data load and
preparation
Data Analytics Visualization and
Publish

4.
Current Challenges
4
Fragile and manually reliant process
No unified platform to perform data cleansing, transformation, analytics and reporting
Inadequate storage to account for data scalability
Data Analysis done using custom data structures which are specific to use cases
Lack of ability to perform monitoring and alerting on streaming datasets.
Lack of ability to enrich data and aggregate at different levels

5.
Enterprise Solution
5
Modern Big Data Platform provides the necessary tool and infrastructure to land, cleanse, process
Real time data streams
Spark streaming to enrich the data with multiple source system feeds
Faster OLAP analytics using Hive LLAP and Druid Integration
Capability to run BI queries in just milliseconds over 1TB dataset
Modernization of Application stack
Explore and visualize streaming datasets and build dashboards for different slices

6.
NetFlow Architecture
Consumers
Threat IP
Sources Data Storage and Processing
6

7.
Q & A