Hands on Security - Disrupting the Kill Chain Breakout Session

1. Copyright © 2015 Splunk Inc.
Hands-On Security
Disrupting the Cyber
Kill Chain using Splunk

2. 2
Safe Harbor Statement
During the course of this presentation,we may make forward looking statements regarding future events
or the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC. The forward-looking statements
made in this presentation are being made as of the time and date of its live presentation. If reviewed
after its live presentation, this presentation may not contain current or accurate information. We do not
assume any obligation to update any forward looking statements we may make. In addition, any
information about our roadmap outlines our general product direction and is subject to change at any
time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described orto includeany suchfeatureor functionalityina futurerelease.

3. 3
Agenda
Splunk & Security
– Unknown Threats
– Connect the Dots across All Data
Kill Chain* Disruption
– Overview
Exercise/Demo
– Security Investigation Example

5. No hard copy?
http://bit.ly/1FCSmxa
pw: splunklive

6. Machine Data contains a definitive record of all
Human <-> Machine
&
Machine <-> Machine
Interaction.
Splunk is a very effective platform to collect,
store, and analyze all of that data.

7. Servers
Storage
DesktopsEmail Web
Transaction
Records
Network
Flows
DHCP/ DNS
Hypervisor
Custom
Apps
Physical
Access
Badges
Threat
Intelligence
Mobile
CMDB
Intrusion
Detection
Firewall
Data Loss
Prevention
Anti-
Malware
Vulnerability
Scans
Traditional
Authentication
Connect the Dots acrossAll Data
7

8. 8
Splunk software complements, replaces and goes beyond traditional SIEMs.
Moving Past SIEM to Security Intelligence
Small Data. Big Data. Huge Data.
SECURITY &
COMPLIANCE
REPORTING
REAL-TIME
MONITORING OF
KNOWN THREATS
MONITORING
OF UNKNOWN
THREATS
INCIDENT
INVESTIGATIONS
& FORENSICS
FRAUD
DETECTION
INSIDER
THREAT

9. 9
Splunk software complements, replaces and goes beyond traditional SIEMs.
Moving Past SIEM to Security Intelligence
Small Data. Big Data. Huge Data.
SECURITY &
COMPLIANCE
REPORTING
REAL-TIME
MONITORING OF
KNOWN THREATS
MONITORING
OF UNKNOWN
THREATS
INCIDENT
INVESTIGATIONS
& FORENSICS
FRAUD
DETECTION
INSIDER
THREAT

10. Hands-OnSession: Kill Chain*Disruption
11
Your system is compromised and the adversary begins its work
Exploitation
The adversary works to understand your organization looking for opportunities
Reconnaissance
The attacker steals data, disrupts your operations or causes damage
Act on Intent
*mostly….

11. • Q. How can the security analysts at Buttercup Games, Inc. discover that their systems
have been compromised by way of a stolen document from their web portal?
• A. They would want to discover and disrupt the kill chain:
• Where did the adversary start? (Recon)
• How did they get a foothold? (Exploitation)
• What was their motive and what did they take?
(Actions on Intent)
Security InvestigationExample
12
bu tercup
games
Let’s get hands-on!

12. 13 1
Reconnaissance
Weaponization
Delivery
Exploitation
Installation
Command & Control
Act on Objectives
Web
Kill Chain Demo Data Source - Activity
Email and Endpoint
Endpoint
Endpoint, DNS, Proxy
Endpoint, DNS, Proxy
A brute force attack takes place on the
customer web site, access is gained, and a
sensitive pdf file is downloaded and
weaponized with malware.
A convincing phishing email is crafted and
sent to an internal target
The pdf document is opened then exploits
the vulnerable pdf reader app creating a
dropper which installs the malware.
Command/Control activity is highlighted by
it’s association with Threat Intelligence
Demo Story line
Threat Intelligence Integration

13. 14
APT Transaction Flow Across Data Sources
1
http (proxy) session
to
command & control
server
Remote control
Steal data
Persist in company
Rent as botnet
Proxy
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
Threat
Intelligence
Endpoint
Network
Email, Proxy,
DNS, and Web
Data Sources
.pdf
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Svchost.exe
(malware)
Calc.exe
(dropper)
Attacker hacks website
Steals .pdf files
Web
Portal.pdf
Attacker creates
malware, embed in .pdf,
emails
to the target
MAIL
Read email, open attachment
Our Investigation begins by
detecting high risk
communications through the
proxy, at the endpoint, and
even a DNS call.

14. System Monitor (SYSMON) is the application programming interface (API) that you use to configure the
Microsoft System Monitor ActiveX control. The System Monitor control lets you view real-time and previously
logged performance counter data such as memory, disk, and processor counter data. SYSMON is available
starting with Microsoft Windows 2000.
Provides a data input and CIM-compliant field extractions for Microsoft Sysmon data input. The Microsoft
SYSMON utility provides data on process creation (including parent process ID) and network connections.
MicrosoftSystem Monitor (SYSMON)

15. 16
Demo

16. 17
To begin our
investigation, we will
start with a quick search
to familiarize ourselves
with the data sources.
In this demo
environment, we have a
variety of security
relevant data including…
Web
DNS
Proxy
Firewall
Endpoint
Email

17. 18
Take a look at the
endpoint data source.
We are using the
Microsoft Sysmon TA.
We have endpoint
visibility into all network
communication and can
map each connection
back to a process.
}
We also have detailed
info on each process and
can map it back to the
user and parent process.}
Lets get our day started by looking
using threat intel to prioritize our
efforts and focus on communication
with known high risk entities.

18. 19
We have multiple source
IPs communicating to
high risk entities
identified by these 2
threat sources.
We are seeing high risk
communication from
multiple data sources.
We see multiple threat intel related
events across multiple source types
associated with the IP Address of
Chris Gilbert. Let’s take closer look
at the IP Address.
We can now see the owner of the system
(Chris Gilbert) and that it isn’t a PII or PCI
related asset, so there are no immediate
business implications that would require
informing agencies or external customers
within a certain timeframe.
This dashboard is based on event
data that contains a threat intel
based indicator match( IP Address,
domain, etc.). The data is further
enriched with CMDB based
Asset/identity information.

19. 20
We are now looking at only threat
intel related activity for the IP
Address associated with Chris
Gilbert and see activity spanning
endpoint, proxy, and DNS data
sources.
These trend lines tell a very
interesting visual story. It appears
that the asset makes a DNS query
involving a threat intel related
domain or IP Address.
ScrollDown
Scroll down the dashboard to
examine these threat intel events
associated with the IP Address.
We then see threat intel related
endpoint and proxy events
occurring periodically and likely
communicating with a known Zeus
botnet based on the threat intel
source (zeus_c2s).

20. 21
It’s worth mentioning that at this point
you could create a ticket to have
someone re-image the machine to
prevent further damage as we continue
our investigation within Splunk.
Within the same dashboard, we have
access to very high fidelity endpoint
data that allows an analyst to continue
the investigation in a very efficient
manner. It is important to note that
near real-time access to this type of
endpoint data is not not common within
the traditional SOC.
The initial goal of the investigation is
to determine whether this
communication is malicious or a
potential false positive. Expand the
endpoint event to continue the
investigation.
Proxy related threat intel matches are
important for helping us to prioritize our
efforts toward initiating an
investigation. Further investigation into
the endpoint is often very time
consuming and often involves multiple
internal hand-offs to other teams or
needing to access additional systems.
This encrypted proxy traffic is concerning
because of the large amount of data
(~1.5MB) being transferred which is
common when data is being exfiltrated.

21. 22
Exfiltration of data is a serious
concern and outbound
communication to external entity
that has a known threat intel
indicator, especially when it is
encrypted as in this case.
Lets continue the investigation.
Another clue. We also see that
svchost.exe should be located in a
Windows system directory but this is
being run in the user space. Not
good.
We immediately see the outbound
communication with 115.29.46.99 via
https is associated with the svchost.exe
process on the windows endpoint. The
process id is 4768. There is a great deal
more information from the endpoint as
you scroll down such as the user ID that
started the process and the associated
CMDB enrichment information.

22. 23
We have a workflow action that will
link us to a Process Explorer
dashboard and populate it with the
process id extracted from the event
(4768).

23. 24
This is a standard Windows app, but
not in its usual directory, telling us
that the malware has again spoofed
a common file name.
We also can see that the parent
process that created this
suspicuous svchost.exe process is
called calc.exe.
This has brought us to the Process
Explorer dashboard which lets us
view Windows Sysmon endpoint
data.
Suspected Malware
Lets continue the investigation by
examining the parent process as this
is almost certainly a genuine threat
and we are now working toward a
root cause.
This is very consistent with Zeus
behavior. The initial exploitation
generally creates a downloader or
dropper that will then download the
Zeus malware. It seems like calc.exe
may be that downloader/dropper.
Suspected Downloader/Dropper
This process calls itself “svchost.exe,”
a common Windows process, but the
path is not the normal path for
svchost.exe.
…which is a common trait of
malware attempting to evade
detection. We also see it making a
DNS query (port 53) then
communicating via port 443.

24. 25
The Parent Process of our suspected
downloader/dropper is the legitimate PDF
Reader program. This will likely turn out to
be the vulnerable app that was exploited
in this attack.
Suspected Downloader/Dropper
Suspected Vulnerable AppWe have very quickly moved from
threat intel related network and
endpoint activity to the likely
exploitation of a vulnerable app.
Click on the parent process to keep
investigating.

25. 26
We can see that the PDF
Reader process has no
identified parent and is the
root of the infection.
ScrollDown
Scroll down the dashboard to
examine activity related to the PDF
reader process.

26. 27
Chris opened 2nd_qtr_2014_report.pdf
which was an attachment to an email!
We have our root cause! Chris opened a
weaponized .pdf file which contained the Zeus
malware. It appears to have been delivered via
email and we have access to our email logs as one
of our important data sources. Lets copy the
filename 2nd_qtr_2014_report.pdf and search a
bit further to determine the scope of this
compromise.

27. 28
Lets search though multiple data sources to
quickly get a sense for who else may have
have been exposed to this file.
We will come back to the web
activity that contains reference to
the pdf file but lets first look at the
email event to determine the scope
of this apparent phishing attack.

28. 29
We have access to the email
body and can see why this was
such a convincing attack. The
sender apparently had access to
sensitive insider knowledge and
hinted at quarterly results.
There is our attachment.
Hold On! That’s not our
Domain Name! The spelling is
close but it’s missing a “t”. The
attacker likely registered a
domain name that is very close
to the company domain hoping
Chris would not notice.
This looks to be a very
targeted spear phishing
attack as it was sent to
only one employee (Chris).

29. 30
Root Cause Recap
3
Data Sources
.pdf executes & unpacks malware
overwriting and running “allowed” programs
http (proxy) session
to
command & control
server
Remote control
Steal data
Persist in company
Rent as botnet
Proxy
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
Threat
Intelligence
Endpoint
Network
Email, Proxy,
DNS, and Web
.pdf
Svchost.exe
(malware)
Calc.exe
(dropper)
Attacker hacks website
Steals .pdf files
Web
Portal.pdf
Attacker creates
malware, embed in .pdf,
emails
to the target
MAIL
Read email, open attachment
We utilized threat intel to detect
communication with known high risk
indicators and kick off our investigation
then worked backward through the kill
chain toward a root cause.
Key to this investigative process is the
ability to associate network
communications with endpoint process
data.
This high value and very relevant ability to
work a malware related investigation
through to root cause translates into a very
streamlined investigative process compared
to the legacy SIEM based approach.

30. 31 3
Lets revisit the search for additional
information on the 2nd_qtr_2014-
_report.pdf file.
We understand that the file was delivered
via email and opened at the endpoint. Why
do we see a reference to the file in the
access_combined (web server) logs?
Select the access_combined
sourcetype to investigate
further.

31. 32 3
The results show 54.211.114.134 has
accessed this file from the web portal
of buttergames.com.
There is also a known threat intel
association with the source IP
Address downloading (HTTP GET)
the file.

32. 33 3
Select the IP Address, left-click, then
select “New search”. We would like to
understand what else this IP Address
has accessed in the environment.

33. 34 3
That’s an abnormally large
number of requests sourced
from a single IP Address in a
~90 minute window.
This looks like a scripted
action given the constant
high rate of requests over
the below window.
ScrollDown
Scroll down the dashboard to
examine other interesting fields to
further investigate.
Notice the Googlebot
useragent string which is
another attempt to avoid
raising attention..

34. 35 3
The requests from 52.211.114.134 are
dominated by requests to the login page
(wp-login.php). It’s clearly not possible to
attempt a login this many times in a short
period of time – this is clearly a scripted
brute force attack.
After successfully gaining access to our
website, the attacker downloaded the
pdf file, weaponized it with the zeus
malware, then delivered it to Chris
Gilbert as a phishing email.
The attacker is also accessing admin
pages which may be an attempt to
establish persistence via a backdoor into
the web site.

35. 36
Kill Chain Analysis Across Data Sources
3
http (proxy) session
to
command & control
server
Remote control
Steal data
Persist in company
Rent as botnet
Proxy
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
Threat
Intelligence
Endpoint
Network
Email, Proxy,
DNS, and Web
Data Sources
.pdf
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Svchost.exe
(malware)
Calc.exe
(dropper)
Attacker hacks website
Steals .pdf files
Web
Portal.pdf
Attacker creates
malware, embed in .pdf,
emails
to the target
MAIL
Read email, open attachment
We continued the investigation
by pivoting into the endpoint
data source and used a
workflow action to determine
which process on the endpoint
was responsible for the
outbound communication.
We Began by reviewing
threat intel related events
for a particular IP address
and observed DNS, Proxy,
and Endpoint events for a
user in Sales.
Investigation complete! Lets get this
turned over to Incident Reponse team.
We traced the svchost.exe
Zeus malware back to it’s
parent process ID which was
the calc.exe
downloader/dropper.
Once our root cause analysis
was complete, we shifted out
focus into the web logs to
determine that the sensitive pdf
file was obtained via a brute
force attack against the
company website.
We were able to see which
file was opened by the
vulnerable app and
determined that the
malicious file was delivered
to the user via email.
A quick search into the mail
logs revealed the details
behind the phishing attack
and revealed that the scope
of the compromise was
limited to just the one user.
We traced calc.exe back to
the vulnerable application
PDF Reader.

36. Questions?

37. 38
Prizes in Exchange for Your Survey Feedback!
Text Splunk to 878787
OR
Scan this QR Code
Then stop by our registration desk for a free gift and a chance to win a
$100 AMEX gift cards

38. Thank You