3. 3
Safe Harbor Statement
During the course of this presentation, we may make forward looking statements regarding future events
or the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC. The forward-looking statements
made in this presentation are being made as of the time and date of its live presentation. If reviewed
after its live presentation, this presentation may not contain current or accurate information. We do not
assume any obligation to update any forward looking statements we may make. In addition, any
information about our roadmap outlines our general product direction and is subject to change at any
time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described orto includeany suchfeatureor functionalityina futurerelease.
4. 4
Agenda
What is the Splunk App for Enterprise Security?
Guided Tour
â General Overview
â Common Information Model
â Incident Response Exercise
â Creating a Correlation Search
Questions?
7. Machine Data contains a definitive record of all
Human <-> Machine
&
Machine <-> Machine
Interaction
Splunk is a very effective platform to collect,
store, and analyze all of that data.
8. Mainframe
Data
VMware
Platform for Machine Data
Splunk Solutions > Easy to Adopt
Exchange PCISecurity
DB Connect MobileForwarders
Syslog /
TCP / Other
Sensors &
Control Systems
Rich Ecosystem of Apps
Across Data Sources, Use Cases and Consumption Models
Stream
8
10. 10
ES Fast Facts
âą Version 3.3 of the product is shipping now
âą We release at least twice a year and add lots of new content
âą Content ideas come from industry experts, market analysis, focus groups, internal
brainstorming, but most importantly YOU
âą All of the great things about Splunk carry through into ES â this makes it flexible,
scalable, fast, and customizable. It leverages everything cool about Splunk.
âą ES has its own development team, dedicated support, services practice, and
training courses
39. 39
Searches that rely on this data
model
How much of ES can I use?
What else could I onboard?
(more) searches that rely on this
data model
Instructor Only
82. 82
New search for unique
pattern in the dataâŠ
Click âDestinationIpâ
Click
83. 83
Thereâs our malicious IP!
We now know that something calling itself âsvchost.exeâ
dropped by something calling itself âcalc.exeâ which was in
turn dropped by our PDF reader, upon opening weapolized
PDF, is communicating to a âknown badâ IP address.
Scroll downâŠ
Scroll
84. 84
Click
âthreat_intel_sourceâ
Click
Thereâs the threat source it
maps to
We could take this further by investigation of
email logs, or wire data from Chrisâs laptop, or
access logs to determine how this PDF got stolen,
but in the interest of time letâs update our
eventâŠ
Click back to
Incident Review
105. 105
Second half of the form after
scroll down
How to assign risk
Other actions of interest
(like Stream Capture)
106. 106
Correlation Search Cheatsheet 1
Search Name: <your user name>â Brute Force Against Web Portal
App Context: SA-zeus-demo
Search: Paste in from your clipboard
Start: -7d@d End: now
Cron Schedule: */2 * * * *
Window Duration: 600
Group By: clientip threat_intel_source
107. 107
Correlation Search Cheatsheet 2
Notable Event: (check the checkbox)
Title: <your username here>- Brute Force on Web Portal from $src$ detected
Description: There have been $logonattempts$ logon attempts and $adminloads$
admin page loads from an $threat$ ip
Security Domain: Threat
Severity: Critical
Default Owner: <your persona>
Default Status: New
When done, click Save
112. The 6th Annual Splunk Worldwide Usersâ Conference
September 21-24, 2015 ïŒ The MGM Grand Hotel, Las Vegas
âą 50+ Customer Speakers
âą 50+ Splunk Speakers
âą 35+ Apps in Splunk Apps Showcase
âą 65 Technology Partners
âą 4,000+ IT & Business Professionals
âą 2 Keynote Sessions
âą 3 days of technical content (150+ Sessions)
âą 3 days of Splunk University
â Get Splunk Certified
â Get CPE credits for CISSP, CAP, SSCP, etc.
â Save thousands on Splunk education!
112
Register at: conf.splunk.com
113. 113
We Want to Hear your Feedback!
After the Breakout Sessions conclude
Text Splunk to 878787
And be entered for a chance to win a $100 AMEX gift card!
Hinweis der Redaktion
Introduce yourself
We donât have a ton of time and ES is quite a feature-rich product. It would take many hours to go through everything the app can do. So weâll spend only a few minutes on some intro slides, and then the great bulk of this session will be hands-on.
Now unfortunately, you do need a modern laptop with a modern browser to participate. You can probably get away with a Surface or something like that, but iPads, old browsers, and especially IBM PCjrâs will not work. (donât laugh â I actually had one of those.)
Everything Iâm going through up here has been pretty well documented in a word doc. You can use the link here to get that doc, or if youâre really interested in it later come see me. You wonât need it right now though.
Each of you has creds â there are 10 fairly large Amazon EC2 instances that have been provisioned for this exercise and if weâre at capacity there will be 12 of you on each. Nowâs a good time to try hitting that URL and logging into Splunk.
Splunk excels at creating a data fabric
Machine data: Anything with a timestamp, regardless of incoming format.
Throw it all in there!
Collect it. Store it in one place. Make it accessible for search/analytics/reporting/alerting.
DETECTION NOT PREVENTION! ASSUME BREACH!
So we need a place we can go to DETECT attacks. DETECT breaches. DETECT the âweird.â
So if you had a place to see âeverythingâ that happenedâŠ
âŠ.what would that mean for your SOC and IR teams?
The Splunk platform consists of multiple products and deployment models to fit your needs.
Splunk Enterprise â for on-premise deployment
Splunk Cloud â Fully managed service with 100% SLA and all the capabilities of Splunk EnterpriseâŠin the Cloud
Hunk â for analytics on data in Hadoop
Splunk Mint â to get insights into data from Mobile devices
The products can pull in data from virtually any source to support multiple use cases.
Splunk Apps extend and simplify deployments by providing pre-packaged content designed for specific use cases and data types.
Weâre pretty proud of this â and by the way for 2014 we were right around the same ranking. When Gartner compares SIEM technologies it is the Splunk App for Enterprise Security they are looking at.
We have some good company up there â all of those products are decent solutions.
But theyâre all quite security-focused, and any other use cases like IT ops, app dev, internet of things, business analytics â all of that is either nonexistent or secondary.
42%
Field focus â Haiyan Song, security marketing, security practice, security development, security field expertise
Used to see complement, now see replace â the 90% of the time in Splunk â why maintain the old technology if youâre not using it?
A huge driver â the main driver of this, is the app for enterprise securityâŠ
3.3. 3.0 was the first release done against Splunk 6 and that was a huge step forward â mainly because of the use of CIM and accelerated data models.
Unlike other competitive solutions ES is constantly evolving â on average twice a year. Upgrades are pretty seamless.
Where does content come from? All of the typical places but most importantly it comes from YOU. We take the best ideas that you give us, and we productize them and make them scalable and supportable.
Splunk is more than a product â it is a wide open platform that inspires. None of this is lost in ES â splunk with ES is just as flexible and customizable. And it leverages technology in the core product like mapreduce and data models. You need ES to scale to the security intelligence needs of a huge enterprise? No problem.
ES has its own dev team and roadmap, dedicated support individuals, a services practice schooled in it and other complementary infosec. Also lots of training is available.
This should look familiar to you. What weâre doing here is giving a starting point for any Security Analyst to understand at a high level whatâs going on in the environment. A single pane of glass, if you will, for all security data.
Â
Everything we are seeing here is customizable â the panels, the indicators, via standard Splunk functionality.
Â
Most of the data on this dashboard is centered on Notable Events. Notable Events are a concept unique to Splunk with ES â thereâs an entire Notable Event framework that allows us to perform simple or complex correlations, and then create events by analyzing disparate events from disparate sources.
Â
Notable Events in ES are categorized into various high-level security domains: access, audit, identity, network, and threat. Weâll see those categories throughout the app.
Â
You can see Splunk Sparklines here â these little green lines. These are great for detecting quick trends in the security events â a continuous line means something constant, which could be a heartbeat or a scripted attack. A spike could be a single attack or maybe just someone fat-fingering their password a few times.
Â
Weâll drill into some of these incidents in a few minutes, but letâs continue on with our tour. How does all this data get into Splunk?
Weâve just discussed that there are various security domains at play within ES. And we know Splunk can onboard a ton of data from many disparate sources, all security relevant.
Â
But in an organization of any size, you have a lot of different sources â sometimes multiple vendors providing the same kind of data. We need a way of standardizing all of this data, and that published model (which is available outside of ES, by the way) is Splunkâs Common Information Model, or CIM.
The common information model, or CIM, is absolutely key to how ES works. Case in point: you might have four different endpoint protection solutions in your environment: Symantec, Sophos, McAfee, and Trend. Each can send data to Splunk in different ways, and each identifies key data in a certain way. However they will all have similar data â in this case things like infected host, how many infections per host, the name of the infections found, etc. Well, Sophos might call the name of an infection âsignatureâ while Symantec might call it âinfectionâ and Trend might call it âmalware name.â CIM allows ES, and any other Splunk app that leverages it, to process those fields in a standard way AT SEARCH TIME so that itâs easy to correlate disparate system data and onboard new data.
Â
CIM does this by mapping the incoming data to fields found in a published Data Model. Data Models exist in CIM for all sorts of security relevant data: IDS, firewalls, endpoint protection, email, DNS, you name it.
Â
CIM is free. You donât need to buy ES to use it. And we encourage our partners to use it too: if we go back to Splunkbase and search for âfireeye add-onâ you will see that the latest FireEye add-on for Splunk is, in fact, CIM compliant:
So what does the data look like once itâs onboarded into Splunk in a CIM-compatible format?
Â
Letâs look at one example in ES: Malware Center.
Here we have a simple dashboard showing us all sorts of detail about recent malware activity in the environment. Like Security Posture, this is high level information, but more granular about a certain security domain (Malware, which is under Endpoint). We have these âcentersâ throughout ES for things like Access, Traffic, Intrusions, Updates, Vulnerabilities, and many other security-relevant areas, and you can investigate them later.
Â
For now, letâs drill into two of the âtop infectionsâ to see CIM at work. Looking at this dashboard we canât tell that we actually have at least two different endpoint protection systems feeding data into Splunk: Sophos, Trend Micro, and Symantec Endpoint Protection. Splunk normalizes the data on search time, according to CIM, to create this (and the other) dashboards.
Â
Click on Mal/Packer, and youâll see that this infection was detected by Sophos. The raw logs are literally a click away:
Â
Note that in this data, Sophos calls the âsignatureâ âEventNameâ but Splunk is normalizing that to âsignatureâ upon search time, which is why we can search on it as âsignature.â
Two different Symantec products are feeding data to Splunk: SEP and SAV. And both call the âsignatureâ something different, but again, Splunk normalizes this upon search time.
Â
So now we understand a little more about CIM. What are the various data models in CIM that ES uses? To figure that out, letâs look at Pivot, which is a core Splunk feature allowing non-technical users to interact with a data model:
Â
This will bring up the 29 data models that ES leverages:
And as an example, letâs see what kind of fields are defined for Malware by clicking on Malware and then the down arrow next to Malware Attacks to see all of the fields in the Malware data model:
Letâs do a quick pivot to show what we can do with these fields. First weâll load up the Malware Attacks data model and change the time to last 60 minutes. Then weâll go to an area chart which by default shows us this time period stretched out on an X axisâŠ
So these are overall malware attacks over the last 60 minutes in our environment. Letâs split out by the signaturesâŠ.
And once youâve gotten the report looking the way you like it. You can save it as a report or dashboard panel.
We donât have time to go through each and every one of the dashboards in the ES app. However, letâs just see that up here under Security Domains, we have the four major ones:
AccessâŠ
EndpointâŠ
NetworkâŠ
IdentityâŠ.
The more data you have flowing into Splunk and into ES, the more useful it becomes. And ES is self auditing to tell you which data sources you are missing:
In version 3.1 of Enterprise Security we introduced a full Risk Analysis framework. This is unique because we allow you to assign an arbitrary risk number, that means something to you, based on a notable event. You can assign risk to a user, or to a system, or to some other object that you see in the environment â perhaps a particular piece of malware is considered risky to you so you elevate the risk on the malware âobjectâ itself.
Â
Letâs bring up the Risk Analysis page associated with Advanced Threat:
The main reason why this risk framework is important is that it gets you away from writing specific rules for specific threats or assets. You donât need 1,000 correlation rules anymore â you simply can elevate risk scores on whatever object you want, based on the behavior youâre seeing in the environment. So the idea here is, a correlation rule fires, and then a risk modifier takes effect and changes the risk score based on cumulative scoring of whatever else has happened to that user, or system, or other object.
Â
On the dashboard, we can define filters to find a particular system or user or timeframe.
Â
Note the natural language descriptions (in the screenshot they are medium and low). We track how your overall risk scoring is doing over time, and constantly re-calculate the baseline. Got a lot of activity going on that isnât ânormalâ for that timeframe and you might see things going from âincreasing minimallyâ to âextremely increasingâ â all based on what the historical norm is.
Â
We can of course see which objects have the highest risk and which correlation rules are contributing the most to the highest risk.
Note we can assign risk ad-hoc by clicking the âCreate Ad Hoc Risk Entryâ button in the upper right.
Now, how does the risk get assigned in a correlation search? When we go to build a correlation search, we will see that, so stay tuned.
Threat intelligence is a growing field in infosec these days. There are entire companies that just offer customized threat feeds that you can subscribe to in order to understand threat artifacts in your environment and how they affect your security posture. There are open-source and community sources of threat intelligence. You may create your own threat intelligence. And you may be a member of an ISAC that offers a feed of threat intelligence specific to your industry.
Â
Letâs go to the Threat Activity dashboard and talk a little about how Splunk handles external threat intelligence.
On the dashboard we can see that weâre using the power of Splunk search to match artifacts in our incoming data against IoCâs we find in our threat feeds. Splunk de-duplicates the threat feeds so that if an artifact shows up in multiple feeds you donât get duplicate notifications.
Â
We can filter the display by threat_group, which is essentially the source of the IoCs. This could be something commercial like ThreatStream or ThreatConnect or Norse, something open-source like Sans or iblocklist, or something from your ISAC that is delivered over a TAXII feed in STIX format.
Â
The threat collection shows that we can use various IoCs to match up against artifacts in our data â IP addresses, domain names, URLs, filenames, certificate common names and organizations, email addresses, registry keys â as long as it can be defined in your incoming feed or locally, you can use it as an IoC.
Â
You can see the most active threat sources, and if you scroll down, you can see the most recent matches against your threat feeds.
Â
How are these configured? Letâs go to the configuration, and see.
This is the main configuration page for the threat intelligence downloads. Most of these are simply URL grabs of files that are regularly updated, and then thereâs some parsing that occurs to put the data into a format that Splunk can leverage it as a lookup from the KV store.
Â
As of 3.3 Splunk natively supports TAXII feeds containing data in STIX format. It also supports OpenIOC documents.
Â
Letâs look at the sans blocklist entryâŠ
There are a lot of options here, some of the more basic ones are the URL to grab the data, the interval (this one is every 12 hours) and the weight. Weight is used during risk scoring â if you increase the weight here, then when IoCs from this source match in your data the risk score assigned to that object will be higher.
Â
Note that you can create your own local threat lists and keep them updated automatically, or edit them manually. Thereâs one called âbad_ipsâ in this demo environment that we will use shortly.
Â
Rounding out the Threat Intelligence capabilities are the Threat Artifacts browser, which allows us to search through all of the artifacts stored in ES:
And the Threat Intelligence Audit, which shows us how up-to-date our threat intelligence is and if there are any issues in downloading content:
We donât have time to go through each and every one of the advanced threat capabilities in the ES app. However, letâs just see that up here under Advanced Threat we have some very interesting capabilities:
Some of the most useful ones are the Protocol Intelligence that leverages wire data from things like Splunk Stream, Netflow, and Bro. Also the Access Anomalies and User Activity, which are very useful to detect possible insider threat. And the New Domain Analysis, which analyzes traffic patterns and DNS queries to domains, and then tells you if you have devices communicating with recently registered garbage domains (that are often associated with DGA). Again â this is something you can go through on your own time.
ES isnât limited to just dashboards. Thereâs over 300 reports that come in the product that range from simple to complex, and each can be used as is or modified as you see fit. Again, they are typically mapped to security domains.
Letâs go back to close to where we started with this tour, and interact with a notable event. Weâll pretend that we have an infected system â a workstation â in our environment that has been infected with Zeus and now it is communicating with known Zeus C2 servers.
Weâll start back on Security Posture.
On the Top Notable Events panel (bottom left) find the event âThreat Activity Seen from Endpoint â Zeus Demoâ and click it. This will lead you to Incident Review.
Now what weâve done here is adjusted the throttling. Normally you would not have the same notable event happening over and over again every 10 minutes â you might throttle so that this event happens only once per day. But for this exercise we need to have lots of the same event to play with.
We will see all of the details of the event, including our most recent comments and ownership activity.
So we know from the title of the event that we have a device on our network communicating out to a known bad IP address thatâs a Zeus C2 address. But Splunk has enriched this event with some very useful info. We can see here that this particular machine is a laptop, and that it is owned by someone in Sales named Chris Gilbert. We see the IP addresses associated with the communication. We see the locations that this person Chris Gilbert works from. This correlation happens automatically against our ES Asset and Identity frameworks â we get the information an incident responder needs right up front.
Â
Everything we see here is pivotable. We can go to places within ES, within Core Splunk and outside of Splunk too, and use that field as an argument. As an example, letâs drill into the arrow next to âDestinationâ and see what Domain Dossier has to say about this external IP address:
Â
We can see that this netblock is assigned to an organization in China.
Â
While there are a lot of these âworkflow actionsâ associated with Notable Events configured already in the product, you can feel free to create custom ones.
Â
Next, letâs understand what else has been going on with this laptop.
One thing that we assume is that traffic from laptops outbound to C2 servers occurs via web proxy, at least when the laptops are on our corporate network. So we can look in our proxy logs to verify.
Note that we have only one source machine (Chris Gilbertâs laptop at 192.168.56.102) communicating with this known bad IP. Thatâs good at least â this doesnât appear to be a widespread infection.
Â
Some other interesting things about this data â notice a fairly large transaction in terms of bytes. Notice also that the connection is âtcpâ over port 443 not âhttpsâ which would be considered normal.
Go back to the notable event and letâs look at Asset Investigator to get a more detailed view of this possibly-infected asset:
Asset Investigator shows us, at the top, all of the things we know about this asset from sources such as CMDBs or Active Directory. It also has multiple âswimlanesâ that visually show you whatâs been going on with the asset:
We can see Threat List, Exec File, IDS, and Notable Events associated with this asset, most of those happening right around the same time (this was likely the time of infection).
Â
Click on the Exec File Activity orange vertical bar and youâll see details about that swimlane appear in the right panel (note you can select multiple bars by holding down Ctrl (CMD) or clicking and dragging). Note that we have a strange file here â calc.exe shows that it was running at this time, but itâs running out of the user profile and not where it normally runs, so thatâs a bit concerning.
Â
Click the magnifying glass to see the events underneath.
These are all Microsoft Sysmon events. Sysmon is a great, free utility from Microsoft that is lightweight and runs on all modern Windows variants. Weâre simply collecting this data from Sysmon into Splunk, in real time, from our workstations. It gives us granular process data that includes parent/child relationships, hash data, and network connections, among other things.
Â
Note that the second event is that strange calc.exe event. Letâs click the small arrow to the far left of the event:
And here we can see that something calling itself calc.exe was actually dropped by the PDF Reader, when it opened up a file, delivered by Outlook, called â2nd_qtr_2014_report.pdf.â Now, weâre assuming that Chris Gilbert has been spearphished in some way, and opened up a weaponized PDF attachment.
Â
But whatâs communicating out to that malicious IP we saw in the Notable Event: 115. 29.46.99? You could always just search through Sysmon data for that IP, but instead, letâs point and click our way through.
Here you can see the parent-child relationships quite easily. Next click on the Image field that contains âsvchost.exeâ and do a ânew searchâ from the popup that comes up:
Â
And if you click on âDestinationIpâ in the extracted fields on the left, youâll see our malicious IP address.
Â
We now can confirm how Chrisâs machine was infected, and what processes are responsible for communicating to known-bad systems.
Â
There are further investigations we can do in this data, for example, we can investigate email logs to figure out where the email came from, and we can update our threat intelligence feeds to block the senders and domains responsible for this spearphishing. However, in the interest of time, letâs update our incident and move on.
Go back to the incident, and edit it again. This time, change the status to Pending, the urgency to low, and add a comment that youâll open a ticket so that the laptop can be re-imaged.
This will lead you to a customized page where you can open a ticket. These types of integrations are relatively easy to build due to Splunkâs flexibility.
Finally, letâs see some of the auditing that ES does of the activity carried out against Notable Events.
Â
The recent activity that you have carried out should appear in the panels. Clicking on a reviewerâs name will bring you detail about that reviewerâs activity.
To finish up, letâs do a little exercise to understand how correlation searches are created. They can be very simple, or very complex, or somewhere in between. They can be standard Splunk searches against the raw data collected by ES, or they can be against accelerated data models (and many of the standard ones are).
Â
We learned from the exercise above that we had a brute-force attack on our web portal that resulted in the exfiltration of a sales report that was then weaponized. How about if we have a correlation search that looks for that behavior and alerts us to it?
This is a search thatâs been created that returns any IP address where we see, over the timeframe selected, a lot of login attempts (greater than 10) and then loading of the admin pages of the portal from that same IP. If any IP address returns from the search, we can consider this an alertable event.
Note that you could turn this into a simple Splunk alert by just doing a âsave asâ alert and running it regularly. But we want to see how to turn this into a Notable Event in ES.
Â
Using your mouse, select the entire text of the search and copy it to the clipboard.
And finally, I would like to encourage all of you to attend our user conference in September.
Â
The energy level and passion that our customers bring to this event is simply electrifying.
Â
Combined with inspirational keynotes and 150+ breakout session across all areas of operational intelligence,
Â
It is simply the best forum to bring our Splunk community together, to learn about new and advanced Splunk offerings, and most of all to learn from one another.