2. Agenda
Introduction
The Splunk App for Stream Overview
Customer Successes
Architecture and Deployment
Key Features in the Splunk App for Stream
Summary
2
5. 5
Why Wire Data?
Deep insights across use cases
IT, security and business data transmit over the wire
Non-intrusive and passive
No impact to workloads
No need for instrumentation and tagging of applications
Holistic and comprehensive
Real-time communication across various protocols
Correlate with logs, events and metrics for comprehensive analytics
7. 7
Enable New Operational Insights
• Add information about application, infrastructure, security and
business activity, without needing instrumentation
• Support new and extends existing Splunk use cases across IT, security
and the business with wire data capture
Enhanced Operational
Intelligence
Efficient, Cloud-Ready Wire
Data Collection
Fast Time to Value
• Gain visibility into any public, private or hybrid cloud infrastructures
with a software solution
• Control data collection volumes with fine-grained protocol and
attribute filtering
• Deploy quickly from interface-driven install
• Enable rapid incident response
• Easily scale out with centralized management
8. Examples of What’s Available From the Wire
8
Performance Metrics
Round Trip Time
Client Request Time
Server Reply Time
Server Send Time
Total Time Taken
Base HTML Load Time
Page Content Load Time
Total Page Load Time
Application Data
POST Content
AJAX Data
Section
Sub-Section
Page Title
Session Cookie
Proxied IP Address
Error Message
Business Data
Product ID
Customer ID
Shopping Cart ID
Cart Items
Cart Values
Discounts
Order ID
Abandoned?
10. Stream at CanDeal: Breaking the Silos
Kris Laxdal,
IT Manager & Security Analyst
“You cannot show up with traditional packet
captures tool in the boardroom. Stream and
Splunk help us understand issues at the high
level and if exec team wants to see the details
we can drill down easily. That is what's great
about Stream!”
IT Operations
• High level view with contextual drill-down ability
• Easy access and visibility into production MySQL
environment helps app developers troubleshoot
issues and roll out releases quicker
• Improved collaboration between teams: IT
operations, QA (pre-production testing),security and
development
• Improved customer response times due to real-time
visibility into app issues
Security
• Correlation against indicators of compromise helps
investigate and mitigate APTs, potential data
exfiltration & other risks
Key Customer Benefits
10
11. Applications Visibility for Easy Capacity Planning
AVP of Networks and Communications,
Large National Bank
“I enjoyed using the Splunk App for Stream as it's
giving us a bunch of different perspectives on
our traffic and better granularity compared to
some of the other tools we used.”
• Granular application and network visibility drives
easy remediation
• Proactive applications and network traffic
monitoring enables better capacity reporting and
planning
• Powerful analytical engine enables data analyses by
novice users
Key Customer Benefits
• Quick host-based deployment at critical network
segments
– Ability to observe both client and server traffic
11
12. Wire Data Intelligence Improves Security
Security Analyst,
Payment Processing Company
“The thing that makes Stream better than any
other packet analysis solution out there is the
statistical analysis from Splunk Enterprise. You
can apply it freely to all of the wire data, which
enables me to analyze this data in ways not
possible before. This visibility help us prevents
external infiltration and avoid malicious
attacks.”
• Real-time security intelligence to prevent attacks and
infiltrations
• Baselining, trending and applying analytics to detect
anomalies in traffic (mySQL, postgres, etc.)
• Centralized management of all wire data results in
operational cost savings
• Efficient monitoring of user authentications for audit
and security
• Non-intrusive and easy monitoring of server
communication
• Flexible and easy integration with Splunk security
dashboards
Key Customer Benefits
12
13. Wire Data Speeds Up Forensics
Security Engineer,
Financial Services Institution
“The biggest value of Stream is how fast we can
resolve and close security cases. Before Stream,
I had to collect data from multiple systems and
it would take me an hour. With Stream,
information is already there and I can get
answers within 5 minutes. “
• 90% reduction in incident triage and investigation time
• Deeper, quicker and easier understanding of traffic and
user activity
• Immediate insights and improved data collection
– Elimination of moving pcap files around between
several tools
• Flexible and easy deployment on key network locations
Key Customer Benefits
13
16. Architecture: Run on Servers
16
End Users
Firewall
Splunk
Indexers
Search Head
Physical or Virtual Servers
Universal Forwarder
Splunk_TA_stream
Internet
Physical Datacenter,
Public or Private Cloud
17. Stream Forwarder Architecture
Protocol
Decoder
(Deep Packet
Inspection)
EventsDecryption
Request/Re
sponse
Network
Interface
(eth1)
Standard Out
(To Splunk Forwarder)
Packets
Streams
Request/Re
sponse
Request/Re
sponse
Protocol
Decoder
(Deep Packet
Inspection)
EventsDecryption Standard Out
(To Splunk Forwarder)
Protocol
Decoder
(Deep Packet
Inspection)
EventsDecryption Standard Out
(To Splunk Forwarder)
Network
Interface
(ethN)
Packets
…
Threads
17
24. (Micro) Manage Wire Data
24
Attribute Filters Stream Filters
• Only send data for specific events
• Example: HTTP with status=404 (File
Not Found)
25. (Micro) Manage Wire Data
25
Aggregations summarize events (many to one)
I want one event
every 60 seconds,
using these key attributes to
uniquely identify buckets,
and generating summary
attributes for these metrics.
28. Enterprise Security
Use event actions to acquire new
data for ad hoc investigations
28
Data, context and extractions
from captures are immediately
available in Protocol Intelligence
dashboards
Selectively acquire data from a
specific protocol or for a duration
of time
32. Better Insights for IT Operations
• Get real-time granular insights to
reduce MTTR without costly
appliances
• Analyze all applications and user
behavior, measure application
response times and trace
transaction paths
• Identify infrastructure performance
issues, capacity constraints,
changes and establish baselines
Value
+
Contextual
Data
Application logs,
infrastructure (storage,
network, server) logs,
performance metrics,
events
32
SQL queries, DNS records, IP
conversations,
transaction traces, ICA
latency, response times
Wire Data
33. Better Insights for App Management
Protocol conversations on
database performance,
DNS lookups, client data,
business transaction
paths… Measure application response
times, deeper insights for root-
cause diagnostics, trace
transactions paths, establish
baselines, etc.
Enriched View
Wire Data
+
Contextual
Data
Application logs,
monitoring data,
metrics, events
33
34. Better Insights for Security
• Real-time DPI with analytics enables easier
forensics analyses and quicker incident
response
• Analyze user and applications behavior
• Respond timely to threats with cost-efficient
real-time header and payload field extraction
• Baseline network traffic and understand
anomalies associated with APTs and insider
threats
• Quick install at endpoints, on-premises and
cloud infrastructures without expensive
appliances
Value
+
Contextual
Data
Firewall logs, application
logs, IDS logs, network logs,
perf. metrics, events
34
User and application traffic,
protocol identification (TCP,
DNS, HTTP, etc.), protocol
headers & payload
extraction, SSL decryption
Wire Data
35. Better Insights for Digital Marketing
Browser-level customer
interactions
Customer Experience – analyze
website and application bottlenecks
to improve customer experience
and online revenues
Customer Support (online, call
center) – faster root-cause analysis
and resolution of customer issues
with website or apps
Enriched View
Wire Data
+
Contextual
Data
Website log activity,
clickstream data, metrics
35
36. The 6th Annual Splunk Worldwide Users’ Conference
September 21-24, 2015 The MGM Grand Hotel, Las Vegas
• 50+ Customer Speakers
• 50+ Splunk Speakers
• 35+ Apps in Splunk Apps Showcase
• 65 Technology Partners
• 4,000+ IT & Business Professionals
• 2 Keynote Sessions
• 3 days of technical content (150+ Sessions)
• 3 days of Splunk University
– Get Splunk Certified
– Get CPE credits for CISSP, CAP, SSCP, etc.
– Save thousands on Splunk education!
36
Register at: conf.splunk.com
37. We Want to Hear your Feedback!
After the Breakout Sessions conclude
Text Splunk to 878787
And be entered for a chance to win a $100 AMEX gift card!
Please skip the first section “Intro to wire data’ if the customer is familiar with the wire data collection. Typically this section may not needed to be explained for network or security teams.
If your customer is network engineer or admin or in network security and is familiar with wire data, please feel free to skip this segment
Wire Data is machine data, recorded as events, that we capture from the network using packet sniffing technology from a host’s network interface for a variety of standard protocols. It is an authoritative record of what is happening with and to your operations in real-time. It is a record of all communication between machines and applications We say that wire data is poly structured since certain protocols are more rigid than others. For example, DNS has little to no variance within the fields/attributes within the protocol while HTTP may have a great degree of variance or additional information within its fields.
When you capture this wire data, you can get very deep insights across various use cases including transaction payloads, application performance, infrastructure bottlenecks, security vulnerabilities, customer payloads and usage metrics, troubleshooting and analytics. Second, capturing wire data has no impact on workloads as it Is passive and non intrusive and it does not require semantic logging by customer or byte-code instrumentation. Finally it is comprehensive as we get real-time insights into everything where we can correlate it with log data, database, Hadoop and systems data.
With this app users can capture application transaction times, transaction paths, network performance, and even database queries. Correlating wire data with other application and infrastructure data in Splunk software such as logs, metrics and events, As a result users are getting insights about app, service or network availability, performance and usage of their services. IT admins can pinpoint root-cause, proactively monitor the performance and availability of their individual technology silos, map dependencies of infrastructure to applications and trend performance to establish baselines. For security, wire data extends itself into rapid incident investigation. more complete threat detection, expanded monitoring and compliance. For business, wire data also captures user interactions and process insights for a deeper understanding of the user experience to support multiple business analytics use cases.
The Splunk App for stream enables efficient, cloud-ready wire data collection with a single software solution. This provides real-time visibility into any public, private or hybrid cloud infrastructure through insights from wire data. Additionally, customers can now securely decrypt SSL encrypted data for data completeness. Capture only the relevant wire data for analytics, through filters and aggregation rules. The app provides the ability to control and manage wire data volumes with fine-grained precision by selecting or deselecting protocols and associated attributes within the App interface
Lastly, can be rapidly deployed to collect wire data in real time to gain network visibility that is otherwise unavailable from cloud implementations and hard to achieve with traditional datacenters. Now, customers can quickly respond to any issue with a simple interface-driven installation, centralized deployment and configuration across IT environments of all sizes.
What can you get out of wire data that you don’t already get from other machine data? Many different things as shown here much more than what specific application chose to log. Anything from data that appeal to the admin level user – the things as how long it takes for this page to load or round trip time. Than application owners can get information valuable for them, what are the error messages we are getting from particular application so that they can further investigate the applications issues. Finally, wire data contains information relevent for business users, what are customers buying, are they abandoning carts, where are this purchases coming from. And this is just a small example….there is way more. There is a small amount of overlap between wire data and other data that we’ve captured so far but it requires deeper and more intrusive instrumentation
Optional text
For example, web server logs typically record status codes such as HTTP 200 response, indicating whether a web page was rendered properly to a client. However, what is missing is transaction payload information – that means, it will not be able to show which of these HTTP 200 responses were for pages with a “service unavailable” message. This information is contained in wire data or transaction payload and is not logged by the server. Can you get this from log data – yes, if you instrument the code. And that is the beauty of wire data – it does not require any instrumentation of the application.
Let me go over Splunk Stream utilization in CanDeal. CanDeal is a Canadian online exchange for Canadian dollar debt securities. They provides their investors access to liquidity for Canadian Government Bonds and money market instruments. Stream is deployed at CanDeal across variety of different use cases – security, IT operations even application development. Their teams can collaborate together at CanDeal – in the past, due to strict restriction to who has access to financial data, developers could not get to production MySQL environment as raw visibility for packet data was something they never had access before. Now security team gives them visibility and they can control and they can access any time without the need to wait which significantly improves turnaround times and visibility into issues. preprod testing can also be quickly done. As a result they have improved collaboration among all different teams. In the past, they spent hours just collecting data, shuttling pcap files which created tremendous lag time.
Customer satisfaction: In real time they can detect proxy issues, SSL mismatching, misconfigured routes,
[Security]Splunk Stream helps Candeal to get huge value in their security practice. They now able to get indicators of compromise by bringing data from STIX into Splunk (utilizing Splice) and cross-correlate against data they are getting from Stream (HTTP, DNS, etc).Since they have a full user and applications behavior, they are now able to quickly investigate and mitigate ATS, analyze potential data exfiltration and other risks in their environment. In the past it was very hard and time consuming to grab data from various pcap and it was fragmented and further it was not indexed in Splunk.
[Executive] They are able to create executive reports and present to executives which they could not do with tools they had in practice in the past.
In this example, the Stream is deployed in of the large national banks out of Texas. They had acquired branches around the country and in the process integrating them with the hq datacenters. They have several months to do the integration. They are using Stream to better understand the traffic that is going across key links not only within the country but also international. Stream gives them very granular visibilty into any traffic, they can understadn top talkers vs top communicators. They can apply analysis to trigger an alert if the traffic utilization is over specific threshold. And the data is used by new IT personnel. What they are getting from Stream that they cannot get from these other tools Is Splunk analytics behind. With other tools they can get some data but the granularity is not there. And many of the tools don’t look at client perspective.
Example: With Stream and Splunk this customer can perform granular analytics they could not do with other tools. “ With other tools I can look at my conversations or all my bytes coming across are, you know, 50 percent of that is, you know, one host, you have thrown a load on that. I can alert when the bandwidth is 85 percent, right? I can do that all day long with other tools But I can't necessarily go look at the traffic and alert on, "Hey, this is I.P. address is taking all the bandwidth. That and much more I can do with Stream”.
This is a company that has deployed Splunk in financial industry and specifically in SaaS based payment processing. They are deploying Stream to monitor wire data traffic in their internal communication as they can easily detect anomalies in traffic. For example, they are able to look into database traffic mySQL and postgres traffic and detect issues with user authentication and more. They are looking at what type of data is being sent at their SQL and postgres servers. One of the biggest value for them is that they are able to apply Splunk statistical analysis on wire data and normalize the quiries so that they can prevent external infiltration and avoid malicious attacks. Both in real-time and historically, they are able to set baselines in the amount and type of their database communication . By doing that they were able prevent injection of malicious queries, ensuring there were no attacks on their servers. They were able to integrate wire data in existing security dashboards and proactively look for any abnormalities in communication. They are also able to look for unexpected traffic such as IRC communication or look for exposed passwords in the user authentication. Protocols: MySQL, postgres , LDAP, RADIUS, IRC, SMB, FTP.
This is a customer from one of the banking institutions in US. They have deployed Stream to monitor data on DMZ and on egress at the points where there is visibility across all the traffic. They wanted to simplify the data collection for forensics purposes. They did not want to search multiple tools to get the data they are looking for. The value for Stream is how fast can they resolve and close security cases. They got Stream because they wanted to get to the so called “higher level” data. For example, logs from firewalls offered them a very basic info example such as this user tried to connect to this or that external website or that external user wanted to connect to this resource from the outside. They get IP destination port and that is it. From Stream we are getting better understanding of the traffic. Now they can answer these question: This user from the outside tried to issue an SQL injection. Once they have the IP address from firewall they can search the Stream and they can get the better view of what the user did. [The way they did it before was to get the pcap from the user based on the firewall log IP information. Now they don’t need to go and get the pcap to get into very minor detail. We can just look into Splunk and see that is actually what happened.] They are looking into lots of things from their IDS including alerts and things . SQL injection, exploit attempt, etc. If it is something new, we go and check Stream out for more details.
Before Stream one example would be as we would be going into IDS alert and bring that into a pcap and then look at pcap into another tool to see what happened, it would take me an hour. With Stream, if get data, enter source and destination IP the get this instantly. Then they can further determine whether I need to investigate more or not. With Stream it goes down to 5 min which is 90% reduction. It is much easier to get data now. ”
For them the ability to look at meta data for HTTP level data, and see the things such as the user agent, the response is valuable and very useful for someone in security domain
We can get wire data directly from the “wire” by installing our wire data collector (the TA) on a dedicated, physical server. This server then receives a passive network copy from a SPAN/(TAP) or packet broker which would transport the “real” wire data of interest to the software.
Alternatively, the data collector can live directly on the systems of interest as a lightweight agent, where the systems can be either physical or virtual. In both cases the data collectors are actually TAs and therefore need to cohabitate with a forwarder.
And finally, events are generated based on the Stream configuration from “App for Stream” and passed on to the UF as modular input data (streaming standard output) in JSON format.
[Terminology]
Here is the current list of protocols that are supported. We also now support Windows OS and also have improved performance. Here we see currently supported protocols and platforms. Talk with your customers and them if there is any other protocol they find extremely useful that they would like to be added. And also ask them why would need particular protocol to be added.
[
[Terminology]
And finally after all that, we’re back in familiar territory as the data is simply data and the rest is just good old core Splunk.
[Terminology]
Thank you. Open up for Questions
So let’s start with IT Operations – You can capture IT relevant data set from network and enrich it with existing data in Splunk such as infrastructure and application logs and events .You capture the content of database queries, granular IP conversations, transaction traces, applications response times. As a result, they will have granular visibility into infrastructure performance, resources utilization, or solve capacity bottlenecks. They can have visibility into applications availability, performance and usage and relation of it to underlying infrastructure components. IT admins can establish better baselines and trending for application performance and usage, and enable better IT and business decision making. This all results in faster resolutions of problems with fewer people.
With the Splunk App for Stream, customers can now unlock the full potential of their machine data by adding wire data to the Splunk software platform. Correlate application and infrastructure data such as logs, events, metrics with wire data to gain valuable insights into application and infrastructure performance, find the root cause of operational issues, understand transaction paths, resolve system downtime, identify infrastructure relationships, assess security threats and understand customer behavior. Enhance operational intelligence for IT, security and the business with wire data analytics, enabled by Splunk software.
The Splunk App for Stream captures wire data from endpoints and key network locations to provide additional insight into how applications are performing, without requiring any instrumentation. Wire data collected by the Splunk App for Stream provides granular data on transaction response times, transaction traces, transaction paths, network performance and even database queries. Wire data effectively complements the kind of metrics often gathered by traditional APM tools, which often focus on specific transaction components. Also, the Splunk App for Stream does not require instrumentation of the application itself, so you can gather performance information across the application without developers instrumenting the application or modifying application logs.
Stream brings huge benefits for your security practitioners.. It is particularly interesting as you are most likely used to packet sniffing for forensic and real time analysis. Data captured contains all user activity and behavior as well as applications behavior. With Stream security customers can perform deep protocol inspection understanding at a very granular level what is going in. This can be used both in real time to understand risks or to perform response to an incident. In addition, security investigators can observe daily or seasonal traffic patterns so that they can immediately react when these become anomalous– they can respond to insider threats. See when someone is emailing IP out or if someone is trying to mimic the database queries to trying to gain access to your internal databases. Stream extracts both header and payload information for very deep granular insights for incident response and threat prevention. It is very important to mention that it can be deployed anywhere into end points, without you need to buy having to by expensive appliances. Very important when customer is a breach conditions.
Backup
Protocol header and data decoding: HTTP, DNS and email protocols (e.g. IMAP, POP3 and SMTP) are the dominant attack and exfiltration vectors for some of the most damaging breaches. Streams can be deployed to acquire header information (HTTP and email) and payload information (DNS) to drive sophisticated analytics for threat detection, incident response, intelligence gathering and threat prevention.
Rapid deployment and response: When incident investigation or analysis or tracking down malware requires additional real-time information from network traffic, threat responders can leverage Stream’s simple and rapid deployment via Splunk to start getting wire data from the system of interest to Splunk. This is useful under breach conditions – where a known infiltration may be in progress.
Customer Experience & Digital Analytics: The Splunk App for Stream allows organizations to capture all web interactions for a deeper understanding of user experience, to improve customer satisfaction, prevent drop-offs, improve conversions and boost online revenues.
Wire data provides insights into key metrics such as time spent on page, bounce rates, time on site, navigation paths, product performance etc., without the need to tag individual pages. This is especially valuable to ensure the success of marketing campaigns.
Business Process Analytics: Business processes such as order management in retail, provisioning in telecoms, trade execution in financial services etc. span many different applications. Collecting relevant data across applications and correlating it is critical for end-to-end process visibility. Wire data implicitly has this information, without requiring specific instrumentation. With the Splunk App for Stream, business operations teams can easily access this data and use Splunk Enterprise to gain real-time business insights across the complete process.
And finally, I would like to encourage all of you to attend our user conference in September.
The energy level and passion that our customers bring to this event is simply electrifying.
Combined with inspirational keynotes and 150+ breakout session across all areas of operational intelligence,
It is simply the best forum to bring our Splunk community together, to learn about new and advanced Splunk offerings, and most of all to learn from one another.
