The document discusses using Splunk to monitor network activity and detect potential security threats. It proposes using Splunk to profile VPN usage and detect abnormal remote access patterns that could indicate security compromises. It also proposes using Splunk to monitor network "jumping" where devices switch between the corporate network and guest network, to detect attempts to bypass security controls or access external websites hosting malware. The approach involves analyzing trends in network activity over time and drilling down on individual users as needed to investigate anomalous behaviors in more depth.
2. Disclaimer
During the course of this presentation, we may make forward looking statements regarding future events or the
expected performance of the company. We caution you that such statements reflect our current expectations and
estimates based on factors currently known to us and that actual events or results could differ materially. For important
factors that may cause actual results to differ from those contained in our forward-looking statements, please review
our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time
and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or
accurate information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change at any
time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other
commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include
any such feature or functionality in a future release.
2
3. About
Andrew Gerber is a managing information security consultant at Wipro. Over the last ten
years he has focused on security information and event management (SIEM), security
analytics, and security operations center (SOC) design. Andrew additionally has experience
evaluating information security program maturity and building effective managed security
service offerings. Andrew has worked with clients in North America, Europe, and Asia,
including several Fortune 100 and Fortune Global 100 industry leaders in financial services,
healthcare, manufacturing, retail, and law enforcement. Andrew holds a B.S. in computer
science and an M.B.A. from Purdue University.
Wipro Ltd. (NYSE:WIT) is a global information technology, consulting, and outsourcing company
with over 145,000 employees across 6 continents and over 175 cities. Wipro posted revenues
of $7.3 billion for the financial year ended March 31, 2014. Wipro helps customers do business
better by leveraging our industry-wide experience, deep technology expertise, comprehensive
portfolio of services, and vertically aligned business model. Wipro is proud of its strategic
partnership with Splunk and the value Wipro delivers using Splunk as a platform across
industries and applications, with a focus in enterprise information security managed services.
3
4. Agenda
New approach to Enterprise Security
– Situational Awareness
– Kill Chain
Techniques using this new approach
– Looking for threat behavior – Profiling VPN access
– Looking for an attacker trying to get out of environment as well as
identifying potential delivery vectors – Profiling Network Jumpers
– A framework for developing additional techniques
Recommendations and best practices for further development and
implementation of this approach
4
5. The Enterprise Security Landscape
Attacks and breaches on the rise, threat actors motivated by previous attacks’ successes
Attackers still have a remarkably easy time getting in
– Organizations are still not implementing basic controls (i.e. geographic restrictions, segmentation,
account lockouts)
A LOT CAN BE DONE WITH BASIC CONTROLS
– Organizations are still not monitoring/responding to IOCs (Indicator of Compromise); a recent breach
analysis showed - multiple alerts on potential malware and malicious activity completely missed
INFORMATION AND ALERTS FROM ALL SOURCES MUST BE ANALYZED
Don’t focus solely on alerts for denied or failure events
– FOCUS ON PROFILING BEHAVIOR OVER TIME & ACROSS PLATFORMS TO DISTINGUISH ANOMALIES
5
6. Threats
Threats are increasing, attacker dwell time still well over 200
days on average.
Move from generic malware targeting everyone to deliberate,
smart attackers targeting you, with a specific objective.
With attackers identifying high-value objectives, the
investment they are willing to make increases.
We can see attackers’ methodology evolving over time to
adapt to organizations’ actions and responses.
People are being targeted more, resulting in more valid-credential
6
based attacks and less need for vulnerability
exploits of network/security devices.
Threat actors now look more like legitimate users. You can
still tell them apart, just not with legacy tools/strategies.
Breaches by Asset Category over Time
From Verizon’s 2014 Data Breach Investigations Report
7. Threats: Who Attacks and Why?
Categories of Attackers Attacker Motivation
7
From IBM’s 2013 Cyber Security Intelligence Index
8. Risks: Clear and Present Danger
Brand / Revenue / Financial Data / Product Data / Customer & Patient Records / Financial Theft /
Blackmail / Job Loss / Operations Disruption and Manipulation / Competitive Espionage / …
8
9. Situational Awareness
Changing threat environments demand enhanced security monitoring, often called “situational
awareness”
Advanced targeted threats have increased the requirement for the proactive detection of
potential incidents above standard due diligence levels.
Situational awareness expands on security information and event management (SIEM)
processes, and requires a combination of asset and threat information and activity data, in
combination with analysis and reporting capabilities.
Advanced analysis capabilities to support “human in the loop” investigation and decision
making are critical requirements. From Gartner’s note “Delivering Situational Awareness” (G00214313)
9
Tech
Process
People
To deliver situational awareness, we need to add a process/approach/model to the
people (us) and the technology (Splunk) deployed to provide enterprise security.
10. Kill Chain
Model to identify threat behavior across the lifecycle of an attack
– Move from looking at single alert or single aspect of the attack
– Must look at entire spectrum of activities (all data) to determine
10
attack/threat
Detection earlier in kill chain = lower impact and mitigation cost
Detection later in kill chain = greater impact, must look back in time
to determine infection/impact and how to contain/mitigate
11. Beyond SIEM – True Security Analytics:
Brings together information that would be time consuming or impossible to
manually analyze (goes beyond centralized logging)
Enables a deep investigation of what otherwise could only be aggregated
and/or ignored
Allows dynamic correlation – visual representation makes anomalies obvious
Enables exploration of loose relationships between events, driven by “human-in-
the-loop” processes, leading to a “hypothesis test findings” approach
instead of an “event evaluate” approach.
Accelerates analyst decision trees around behavior
Is cohesive and behaviorally driven, with a monitoring/response posture based
on knowing your users, assets, and environment
11
12. Use cases to implement with Splunk
Use Case 1 - Detect inappropriate or malicious remote access
– VPN profiling of employees, contractors, vendors, and other insiders
– Useful to identify following kill chain stages
C2, Exfiltration
– Also useful to identify employee/insider Fraud, Theft, & Abuse (FTA)
Use Case 2 - Detect attempted and actual bypass of network controls
– Detect network jumping and off-network activity
– Useful to identify following kill chain stages
Delivery, C2, Exfiltration
– Also useful to identify employee/insider Fraud, Theft, & Abuse (FTA)
12
14. What & Why?
Find abnormal remote access usage pattern in remote access
– VPN access with valid credentials used in major attacks, including recent healthcare
14
industry breach
Profile remote usage by employees, contractors, vendors, and other insiders
Look for:
– Indicators of Delivery, C2, Exfiltration, as well as employee or insider FTA
– Identify potentially compromised credentials
Key points to look for:
– Increase in login frequency
– Odd times/locations
– Improbable travel distance between logins or login attempts
(velocity requirements between consecutive geographical login locations too high)
16. Design & Approach - Workflow
Geographic & Network VPN Trends
At-a-glance profiling of VPN login success and failures
Geolocation and domain charting identify normal vs. abnormal access
• Top Level Domains and other domain names to find anomalies,
i.e. connections from .edu TLD or external VPN services
User level VPN Trends
Multiple login failures by count and over time and successful logins
provide insight into VPN behavior.
Identify repeat VPN login failure trends by user
Easy to spot outlier and clustered events
16
17. Design & Approach - Workflow
Geographic Analysis with “Traveler” identification
Per-country trends & users with multiple locations in a given time period
Also identify relative distances for users from a relevant fixed location
“Traveler” mapping & improbable behavior analysis
Determine unlikely distance/time combinations between VPN logins
17
18. Key Events – VPN Authentication Success/Failure
The key searches are looking for VPN authentication success and failure, which we will expand on throughout this use case.
18
19. Overview – Geographic & Network VPN Trends
19
index=firewall sourcetype=ACMEvpn "Security Negotiation Complete"
| iplocation IP
| geostats count by Username globallimit=0
index=vpn sourcetype=ACMEvpn "Login failed"
| eval userinfo=user.":".user_bunit
| iplocation src_ip
| geostats count by userinfo globallimit=0
index=firewall sourcetype=ACMEvpn "Security Negotiation Complete"
| stats count by IP
| lookup dnslookup clientip as IP
| rex field=clienthost ".*(?P<toplevel>.w+)$"
| stats count by toplevel
index=firewall sourcetype=ACMEvpn "Security Negotiation Complete"
| stats count by IP
| lookup dnslookup clientip as IP
| rex field=clienthost ".*.(?P<midlevel>w+).(?P<toplevel>w+)$“
| eval thedomain=midlevel.".".toplevel
| eval lendomain=len(thedomain)
| where lendomain>0
| stats count by thedomain
| sort -thedomain
| sort -count
20. Overview – User-based VPN Trends
20
index=firewall (sourcetype=ACMEvpn AND
"AAA user authentication Rejected" AND user=*) OR
(sourcetype=ACMEtraffic AND src_user=* AND to=VPN
AND action!="allowed")
| rename src_user AS fulluser
| rex "users=s(?<fulluser>.*)"
| stats count by fulluser
| search count>3
index=firewall (sourcetype=ACMEvpn AND
"AAA user authentication Rejected" AND user=*) OR
(sourcetype=ACMEtraffic AND src_user=* AND to=VPN
AND action!="allowed")
| rename src_user AS fulluser
| rex "users=s(?<fulluser>.*)"
| top fulluser
index=firewall sourcetype=ACMEvpn
"Security Negotiation Complete"
| stats sparkline(count), count by Username | sort -count
21. Overview – User-based VPN Trends
index=firewall sourcetype=ACMEvpn "AAA user authentication Rejected" user=*
| rex "users=s(?<fulluser>.*)"
| timechart count by fulluser useother=f limit=25
21
22. Geographic Analysis with “Traveler” identification
22
index=firewall sourcetype=ACMEvpn "Security Negotiation Complete"
| iplocation IP
| eval regionlen=len(Region)
| where regionlen>0
| eval regioncity=City.",".Region
| stats sparkline(dc(IP)),dc(IP) as howmanyIP,dc(regioncity) as howmanyRegion,
values(regioncity) as Locations by Username
| sort -howmanyip
| where howmanyRegion>1
index=firewall index=firewall sourcetype=ACMEvpn "Security Negotiation Complete"
|dedup IP
| iplocation allfields=true IP
|eval citylen=len(City)
| eval short_lon=round(lon,2)
| eval short_lat=round(lat,2)
| strcat short_lat "," short_lon as latlon
| eval HQ="37.235,-115.811"
| where citylen>0
| haversine originField=HQ latlon units=mi | table _time,Username,City,Region,distance
| sort -distance | eval distance=round(distance,0)
24. Design & Extension Notes
Additional panels:
– Simultaneous logins (often rare as a legitimate scenario)
– Increase in data volume over connection (sign of exfiltration, data collection)
– Potential to add algorithms to refine results and accelerate analysis
Additional Information about user access patterns
– “Out-of-Office” information - Integrate with Exchange
– PTO/Absence/etc. - Integrate with HR/Time management systems
24
26. What & Why?
Find assets & users jumping from corporate LAN, WLAN to Guest Network
– Detect attempts to bypass security controls
– Detect malware vector of “benign” off-network browsing
1 in 566 websites host malware (Symantec 2014 Internet Security Threat Report)
– If controls exist around Guest network usage, still implement this for attestation
Profile jumping behavior to look for patterns and anomalies
– Identify the User, IP address, MAC address
– Identify activity before and after jumping
– Filter out insider Fraud, Thief, Abuse from possible
26
Indicators of Compromise
Key points to look for include
– Assets and users jumping periodically –
Normal business users should be on corporate network
– Network jumps which don’t appear to be pre-meditated
(i.e. looking for programmatic jumps)
– Volume, periodicity, destination, traffic type can all be
indicators of potential Exfiltration
“40% [of companies] reported
that they had been exposed to a
security threat as a direct
consequence of an off-network
user’s laptop getting compromised
within the last twelve months.”
From Google report, “Off-Network Workers –
The Weakest Link to Corporate Web Security”
27. Design & Approach
Overview – Long/Short Term Off-Net Jumping Trends
Identify a user of interest and drill-down to investigate
Behavior investigation – longitudinal trending
Behavior investigation – Pre-Jump Activity
Behavior investigation – Guest Network Activity
27
28. Design & Approach - Workflow
Long/Short Term Off-Net Jumping Trends
Visual analysis to determine what look abnormal
At-a-glance profiling of corporate credentials used on guest
network – activity for today, 7-days, 14-days
Rapid investigation to identify users of interest
Selection enables deep investigation via initial drilldown into user activity/details
28
Selection to
lookup user
Dynamic drilldown begins at this point on this dashboard:
When you click on the row, the IP, Hostname, MAC is
passed on the following subpanels, this is based on
drilldown parameters being set in this panel’s XML source.
Selection determines drill down
29. Design & Approach - Workflow
Behavior Investigation – Longitudinal Trending
Patterns identify potential repeat offender, or possible C2/exfiltration
look at guest network activity to clarify – compare these two trends
29
30. Design & Approach - Workflow
Behavior Investigation – Pre-Jump Activity
• Does the jump make sense? – driven by business logic or “benign” behavior
• Does the jump look like attacker trying to get out? – more “random” patterns
• Does the jump look like insider threat? – exfiltration, etc.
Looking back in time from the jump
User activity on the corporate network preceding the jump
Looking back in time to the jump
User device to IP address mapping of jumper
Looking in time after the jump
User activity on the guest network after the jump
30
31. Key Event – Guest network DHCP request
Key search to identify this activity
• Look at guest network firewall logs which logs DHCP requests (IP MAC hostname)
• Look at DHCP requests using IP address of one of our corporate networks, and the MAC address.
• Eliminate mobile devices, limit results to our corporate hostname naming convention
• Database of internal IP space, hostnames, and associated MAC addresses is being built to further refine this.
31
32. Trending – How it’s Done
32
index=firewall sourcetype=“ACMEguestFW"
(hostname!=*phone* AND hostname!=*pad* AND hostname!=*android*)
dhcp_msg=Request ip=“ACMEipSpace”
| regex hostname=“ACMEnamingConvention"
| timechart span=4h limit=30 count by hostname
index=firewall sourcetype=“ACMEguestFW”
(hostname!=*phone* AND hostname!=*pad* AND hostname!=*android*)
dhcp_msg=Request ip=“ACMEipSpace" earliest=-14d latest=-1d
| regex hostname=“ACMEnamingConvention"
| dedup hostname
| timechart span=1h count
| eval StartTime=relative_time(now(),"-48h@h")
| eval Series=if(_time>=StartTime, "Yesterday’s Count", “2 Week Average")
| eval Hour = strftime(_time,"%H") | chart max(count) by Hour Series
33. Trending – How it’s Done
index=firewall
sourcetype=“ACMEguestFW" ip=“ACMEipSpace"
(hostname!=*phone* AND hostname!=*pad* AND hostname!=*android*)
dhcp_msg="Request"
| regex hostname=“ACMEipSpace"
| timechart span=1h count by hostname
33
34. Identify User, present additional data – How it’s Done
34
index=firewall
(hostname!=*phone* AND hostname!=*pad* AND hostname!=*android*)
sourcetype=“ACMEguestFW" ip=“ACMEipSpace" dhcp_msg="Request"
| regex hostname=“ACMEipSpace"
| stats count by ip,_time,hostname,mac| sort _time
View the XML Source for the
Dashboard (“Edit Source”),
find the panel, and add:
<drilldown>
<set token="source_ip">$row.ip$</set>
<set token="mac">$row.mac$</set>
<set token="hostname">$row.hostname$</set>
</drilldown>
Make this panel only appear when the drilldown is activated:
<panel><single id="jumpername" depends="$source_ip$">
1
Search uses $source_ip$ based on click and searches the internal firewall logs to find the most recent user from that IP address:
index=firewall sourcetype=ACMEfw src=$source_ip$ | rex field=src_user "w+(<browseusername>w+)" | dedup browseusername | table browseusername
2
3
4
Drill-down
to lookup
user
35. Longitudinal Trending – How It’s Done
This panel is driven by the same drill-down we’ve been using, based on $hostname$ from the guest network firewall logs.
The search simply returns the jumping pattern over the past week and charts it in 15-minute spans.
index=firewall hostname=$hostname$ dhcp_msg=Request sourcetype=ACMEguestFW | timechart span=15m count
35
36. Behavior Investigation – Pre-Jump Activity
36
Select “Edit Panels” for the Dashboard and then
“Add Input”, select “Radio”, drag the input to the
panel, and customize in the GUI, or add the XML
code directly in “Edit Source”. This dropdown input
sets the token $category$ to the value selected:
<input type="dropdown" token="category“
searchWhenChanged="true">
<label>Select Category</label>
<populatingSearch earliest="@d" latest="now"
fieldForLabel="category"
fieldForValue="category">index=firewall sourcetype=pan*
src_ip=$source_ip$ | stats count by
category</populatingSearch>
<choice value="*">ALL</choice>
</input>
3
Search the Windows DNS logs for requests and responses triggered by the Jumper on the
corporate network. Still using the same drilldown from before for source_ip:
index=winevents sourcetype="MSAD:NT6:DNS" src_ip=$source_ip$ | stats count by
questionname,questiontype,response,src_ip | rex mode=sed field=questionname
s/(d+)/./g | sort –count
This is a basic filtering search | stats to take a count of queries made, type and the
response by the source ip | regex to use sed to change format of DNS queries to exclude
(<digits>) | sort by count
1
Selection determines drill down
Combined Static & Dynamic Dropdown input.
Static (default) vaue of ALL maps to a value of
“*”, dynamic options populated by a search:
index=firewall sourcetype=ACMEfw
src_ip=$source_ip$ | stats count by category 2
37. Guest Network Sessions for Jumper
Get a list of IP addresses for the identified jumper based on MAC address from the Guest network firewall logs.
Again going back to the same drill-down, use the MAC address identified and list guest network IPs associated with the MAC we’ve tied to a
corporate asset:
index=firewall sourcetype=“ACMEguestFW” (ip!=“ACMEipSpace" AND ip!="0.0.0.0") mac=$mac$| stats count by mac,ip | fields - count
37
38. Behavior investigation – Guest Network Activity
List hosts accessed by the jumper on the guest network, filtered by pass/block/all as per the station radio
input above and using the source selected in the original drilldown on the dashboard:
index=network sourcetype=ACMEguestWLC srcip=$source$ action=$action$ | stats count by
srcip,hostname,action,msg,dstip | sort -count
38
3
Static form input defined to filter the panel’s
search on action field (block, pass, all)
View the XML Source for the
Dashboard (“Add Input”), select
“Radio”, drag the input to the panel, and customize in the GUI, or add the XML code
directly in “Edit Source”. This radio input sets the token $action$ to the value selected:
<input type="radio" token="action" searchWhenChanged="true">
<choice value="pass">pass</choice>
<choice value="block">block</choice>
<choice value="*">all</choice>
<default>*</default>
</input>
2
1
39. Design & Extension Notes
Areas to continue the investigation
– Select user of interest to drive additional panels – including additional historical trending
– Additional review of DNS requests
– Data volume on guest network
– Threat list mapping for known C2 servers, site hosting malware/malvertising
Practical integrations
– Capture page, walled garden for jumpers with training and/or restriction on Guest Network
Potential to add algorithms to refine results and accelerate analysis
– High level charts – 14 day, 7 day, today
– Integrate additional data sources to further identify behavior
39
41. Developing Additional Use Cases
Have a disciplined approach
Start with a behavior, choose a point on the kill chain
Identify what logs sources you have
Think about and try different visualizations
Use statistics and simple algorithms to clarify the data
Find related log sources
Think longitudinally
Find outliers, shift your parameters, and let more outliers emerge
41
42. Additional Examples
Identifying Pass-the-Hash (PtH) Attacks and other Credential Theft Techniques
– Look for lateral movement, then get specific in your search for specific techniques. Methods include RDP and other
remote access tools, the use of PsExec, as well as Windows Management Instrumentation (WMI).
– The NSA report “Spotting the Adversary with Windows Event Log Monitoring” provides many good ideas to build on. For
PtH:
“The successful use of PtH for lateral movement between workstations would trigger event ID 4624, with an event
level of Information, from the security log. This behavior would be a LogonType of 3 using NTLM authentication where
it is not a domain logon and not the ANONYMOUS LOGON account.”
“A failed logon attempt when trying to move laterally using PtH would trigger an event ID 4625. This would have a
LogonType of 3 using NTLM authentication where it is not a domain logon and not the ANONYMOUS LOGON account.”
Validating and Monitoring Mitigation Actions (Closed-Loop Management)
– When mitigating risks and threats in your environment, you need to validate that your measures take effect while
monitoring and minimizing disruption to mission-critical business operations.
– Look for metrics that are leading indicators to help validate progress
– Look for trailing indicators that show potential disruption
– One example would be forced password expiry impairing users who only use applications with integrated authentication
42
that do not support password resets
44. Security Controls
The average enterprise today has decent but incomplete
coverage via a collection of security controls
In addition to gaps in security controls there is usually an
even larger gap in which security controls are centrally
logged and monitored
Multi-control correlation is rarely done, and even more
rarely done right
Security controls in silos are not enough
Approach to analysis needs to be cohesive and behaviorally
driven, with a monitoring/response posture based on
knowing your users, network, and environment
Need to evolve:
– From compliance reporting to threat detection
– From finding/neutralizing malware to dissecting/disrupting
attack
– From static views of data to longitudinal data analytics
44
45. Security Control Frameworks
45
Security Control
Monitoring Priorities:
• Perimeter-in
• Critical assets/crown jewels
• Kill chain/behavior-based
• Quick wins
SANS Critical Security Controls V5 – SANS
Top 20
(ISC)2 Common Body of
Knowledge
(10 Domains)
ISO 27001:2013
(114 Controls in 14 Groups)
NIST Special Publication
800-53 Rev. 4
(224 controls in 18 families)
1. Inventory of Authorized and Unauthorized
Devices
2. Inventory of Authorized and Unauthorized
Software
3. Secure Configurations for Hardware and
Software on Mobile Devices, Laptops,
Workstations, and Servers
4. Continuous Vulnerability Assessment and
Remediation
5. Malware Defenses
6. Application Software Security
7. Wireless Access Control
8. Data Recovery Capability
9. Security Skills Assessment and Appropriate
Training to Fill Gaps
10. Secure Configurations for Network Devices
such as Firewalls, Routers, and Switches
11. Limitation and Control of Network Ports,
Protocols, and Services
12. Controlled Use of Administrative Privileges
13. Boundary Defense
14. Maintenance, Monitoring, and Analysis of
Audit Logs
15. Controlled Access Based on the Need to
Know
16. Account Monitoring and Control
17. Data Protection
18. Incident Response and Management
19. Secure Network Engineering
20. Penetration Tests and Red Team Exercises
1. Access Control
2. Telecommunications
and Network
Security
3. Information Security
Governance and Risk
Management
4. Software
Development
Security
5. Cryptography
6. Security Architecture
and Design
7. Operations Security
8. Business Continuity
and Disaster
Recovery Planning
9. Legal, Regulations,
Investigations and
Compliance
10. Physical
(Environmental)
Security
1. Information security policies (2
controls)
2. Organization of information security
(7 controls)
3. Human resource security - 6
controls that are applied before,
during, or after employment
4. Asset management (10 controls)
5. Access control (14 controls)
6. Cryptography (2 controls)
7. Physical and environmental security
(15 controls)
8. Operations security (14 controls)
9. Communications security (7
controls)
10. System acquisition, development
and maintenance (13 controls)
11. Supplier relationships (5 controls)
12. Information security incident
management (7 controls)
13. Information security aspects of
business continuity management (4
controls)
14. Compliance; with internal
requirements, such as policies, and
with external requirements, such as
laws (8 controls)
1. Access Control
2. Awareness & Training
3. Audit & Accountability
4. Certification,
Accreditation & Security
Assessments
5. Configuration
Management
6. Contingency Planning
7. Identification And
Authentication
8. Incident Response
9. Maintenance
10. Media Protection
11. Physical & Environmental
Protection
12. Planning
13. Personnel Security
14. Risk Assessment
15. System & Services
Acquisition
16. System &
Communication
Protection
17. System & Information
Integrity
18. Program Management