Ensuring security and compliance across a globally distributed, large-scale AWS deployment requires a scalable process and a comprehensive set of technologies. In this session, Adobe will deep-dive into the AWS native monitoring and security services and some Splunk technologies leveraged globally to perform security monitoring across a large number of AWS accounts. You will learn about Adobe’s collection plumbing including components of S3, Kinesis, CloudWatch, SNS, Dynamo DB and Lambda, as well as the tooling and processes used at Adobe to deliver scalable monitoring without managing an unwieldy number of API keys and input stanzas. Session sponsored by Splunk. AWS Competency Partner

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Randy Young, Splunk
Scott Pack, Adobe
November 29, 2016
SAC309
You Can’t Protect
What You Can’t See
AWS Security Monitoring & Compliance Validation

2. What to expect from the session
•Learn how to automate data collection for security
monitoring and validate compliance for large numbers of
AWS accounts.
•Learn how Splunk & the Splunk App for AWS can enable
you to managing your AWS environment.

3. Presenters
• Scott Pack
• Security Engineer @ Adobe
• SLC, UT
• 2 Year AWS User
• 4 Year Splunker
• Proudly DQd at 3 Pinewood Derbies
• Randy Young
• Principal Product Manager @ Splunk
• Bezerkly, CA
• 8 Year AWS User
• 3 ½ Years a Splunker
• Proud Dubs Season Ticket Holder
R

4. The background
Digital Marketing
~55k physical hosts across 30 sites
Collection of ~20 admin teams.
• Different tech stacks, but mostly *nix
Monitoring Toolset:
• Netflow, FPC, IDS, Network Transaction
S

5. Security monitoring
5
Security Engineering:
• Build & Maintain Monitoring Toolset
• Define (w/ SOC) “Security Notables”
• Work with Internal Audit to gauge compliance
Security Operations:
• Event Analysis
• “Hunting”
• Investigation
• Incident Response
S

6. What is Splunk?
Platform for Machine Data
Correlation &
Enrichment
Field
Extraction
Reporting
& Alerting
Data Collection &
Field Extraction
Multiple use cases across one platform
R

7. What can Splunk do for your AWS environment?
7
Splunk App for AWS
EC2
EMR
Amazon
Kinesis
Route 53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Amazon
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
R

8. Shift to the cloud
8
Lots of accounts … > 200
Dozens of teams, thousands of instances
Missing data to:
• Detect/respond to incidents
• Making assurances to Compliance
We received a mandate: Fix this
• Get whatever visibility you can
• Minimize risk of operations impact
• Be cost sensitive
S

9. AWS security incidents
9
1. Infrastructure Impact
Baddie impacts the infrastructure as
an external user (DDOS)
2. Host Compromise
Baddie has some control of a host.
(Command Injection)
3. Account Compromise
Baddie interacts as an authenticated
AWS user. (Account Takeover)
S

10. Initiative goals
Identify & collect security relevant data
Analysis the same as on-premises
Data -> Splunk ES -> SOC
Minimize operations impact
Limit IAM users
No risk to services
Quick setup
10
S

11. Data sources
S

12. AWS native sources
11/30/201612
CloudTrail
API Usage &
Logging
VPC Flow Logs
Virtual Interface
Connectivity
AWS Config
Account Configuration &
Inventory
ELB Access
Logs
Load Balancer
Logging
Trusted Advisor
Security Practice Checks
Identity & Access
Management
Credential Report
R

13. Data examples
13
CloudTrail
VPC Flow Logs
ELB Access Logs
Config Credential Report
R

14. Cross-account authentication
14
IAM users
• Use API Keys directly
Roles
• AWS Security Token Service
• Can be “assumed” by a specified principal
• Authenticate to an aggregation account user
• Assume the cross-account role
• Retrieve temporary access keys
• Make calls with temporary keys
Tutorial: Delegating Access using IAM Roles - http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html
Shon Sha re:Invent 2014 - https://www.youtube.com/watch?v=0zJuULHFS6A
S

15. A few more AWS services
15
S3 –
File/Object
Storage
Lambda – Code
without
Instances
Amazon
Kinesis – Data
Streaming
CloudWatch
Logs
SNS –
Notification
Service
DynamoDB –
NoSQL Database
S

16. Collection plumbing: S3
S3 Buckets:
• ELB (1 per region)
• Permit PutObject from ELB IAM Roles
• Config
• Permit PutObject from config.amazonaws.com
• Config Parsed
• CloudTrail
• Permit PutObject from cloudtrail.amazonaws.com
• Trusted Advisor Results
• Permit PutObject from Lambda execution IAM role
11/30/201616
AWS ELB Account IDs for Log Delivery: http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html#attach-bucket-policy
S

17. Collection plumbing: VPC flows
Amazon Kinesis stream:
• 1 per region
CloudWatch log destinations
• 1 per region
• Directs to region-local Amazon Kinesis stream
17
S

18. 18
Aggregation
18
CloudTrail
VPC Flow
Logs
Config
ELB Access
Logs
Trusted
Advisor
IAM
Amazon
S3
Per Region
CloudWatch
Per Region
CloudWatch
Destination
Monitored
Account
Aggregation
Account
S

19. Registration
S

20. 20
CloudFormation
Resources:
Config Role
FlowLogs Role
SecEng Role
SNS
Notification
Role’s
Done!
Inputs:
Description
Jira Queue
Registration
Lambda
Registration
DynamoDB
Monitoring registration
S

21. Registration through web UI
11/30/201621
S

22. 22
Scheduled delivery
enforcement
Distributor Handler
Config
STS
Config
Handler
IAM
Credential Report
STS
Distributor
CloudWatch CloudWatch
Scheduled retrieval &
storage
S

23. Dashboards & analysis
S

24. Splunk apps & add-ons
• Input Methods: S3
• Input Sourcetypes: CloudTrail, VPC
Flows, ELB Access Logs
• Parsing Handler:
GZIPMessageHandler
11/30/201624
Aggregation reduces amount of Splunk inputs: 26 Total Inputs
• S3: 14
• Amazon Kinesis Inputs: 10
• Additional Logging: 2
Currently running on a dedicated Heavy Forwarder.
• If needed, split regions to different forwarders.
S

25. Sourcetypes, lookups, and other fun
25
Sourcetypes: Cheated off the Splunk App for AWS.
• Set JSON KV format and check line-breaks
Use HTTP Event Collector FOR DynamoDB Registrations
• Scheduled lookup-generating search
• Auto lookups on each sourcetype
Tagging into Enterprise Security data models
• ELB Access Logs & VPC Flow Logs right out of the box
S

26. Onboarding dashboard
26
S

27. Account overview
S

28. Compliance checks
Inspect Config + Credential Reports
+ Bunches more
Query per Standard/Compliance Requirement
S

29. Resource lookup
S

30. Example ES correlation rules
30
• Console logins from outside org IP space
• Flows to/from threat actors
• Instance increase by X% within 24-hours
• AMI sharing to non-org AWS account
• URI/user agent web application attacks
• Multiple service API denies for 1 API key within X mins
• (Nimbostratus – Andres Riancho, BlackHat 2014)
S

31. Things that can go wrong:
S

32. Splunk hints
32
Amazon Kinesis Modular Input*
• Can chew up memory.
• /opt/splunk/etc/apps/kinesis_ta/bin
java_args = [ JAVA_EXECUTABLE, "-classpath",CLASSPATH,"-
Xms512m","-Xmx512m",
"-
Dsplunk.securetransport.protocol="+SECURE_TRANSPORT,JAVA_MAIN
_CLASS]
Config snapshots are jsonormous
• Use Lambda to split out the resources.
* You can now use the Splunk TA for Kinesis Inputs
S

33. AWS hints
ELB permission granularity restrictions
• ModifyAttributes
Keep an eye on capacity. Watch:
• DynamoDB read capacity
• Amazon Kinesis shard usage
AWS internal actions
• Auto Scaling
• EMR
S

34. Where we’re at right now
• 57 AWS accounts currently enrolled
• ~3 TB/day
• Haven’t broken any accounts yet!
• Finding more data sources
• Config Rules
• Amazon Inspector
• Automating our AWS security policy audit
• Written a handful of Splunk Enterprise correlation rules
• Actioned by SOC
• Automated Jira ticketing for remediation
11/30/201634
S

35. Make machine data accessible,
actionable and valuable to everyone.
35
R

36. Splunk and AWS – Customer value
36
“Customers love the agility of AWS together with the end-to-end
visibility of Splunk.” Andy Jassy, AWS CEO
R

37. Operational Intelligence Security Intelligence
- Etc.
AWS data leveraged across multiple use cases
Financial Intelligence
R

38. Operations Intelligence
- What is my EBS footprint and posture
across all my accounts and all my
regions?
- Who started/stopped/restarted what
instances and when?
- What EC2 instances are underutilized
and perhaps overprovisioned?
- What is the traffic volume into my VPC
and where is it originating from?
- Why are certain resources unreachable
from certain subnets/VPCs?
- List resources with missing or non-
conforming tags?
- Etc.
Security Intelligence
- Who added that rule in the security
group that protects our application
servers?
- Where is the blocked traffic into that
VPC coming from?
- What was the activity trail of a
particular user before and after that
incident?
- Alert me when a user imports key
pairs or when a security group
allows all ports
- What instances are provisioned
outside of a VPC, by whom and
when?
- What security groups are defined but
not attached to ay resource?
- Etc.
- Etc.
Sample use cases for AWS data
Financial Intelligence
- How many instances are you
running?
- What Reserved Instances have you
purchased in the past?
- What is your Reserved Instance
utilization?
- How much are you paying per
account?
- How much are you using per service
across all accounts?
- How many Reserved Instances
should I buy based on usage?
- Is this account within budget this
month, and how have they tracked in
the last year?
- Etc.
R

39. Now you have all this data… what do you do with it?
HR Director: Good afternoon…
You: (smile nervously)
HR Director: Joe was let go today. Can you close his
account. I want to get an email if his account does anything
strange this weekend.
You: (nod) And create an alert.
R

40. sourcetype=aws:cloudtrail userIdentity.userName=joe|table _time event*
user*
Save as alert > Email action
R

41. Now you have all this data… what do you do with it?
CFO: Good Afternoon…
You: (smile nervously)
CFO: Our production account’s spending is
on track, but I need YOU to cut our
development account spend by 1/3.
You: No problem!
R

42. AWS tag-based instance auto start/stop
43
Weekends
Non-Working Hours
1. Create IAM user ‘robot’
2. Install AWS CLI on splunk host
3. Define tag: PowerSave=LongRun/
RareRun/Normal on each instances
4. Create splunk alert
• CRON, run in morning/night
• SPL to search instances by tag
• Alert action to call AWS CLI to
batch start/stop instances
And save 40%
Development cost!
R

43. Now you have all this data… what do you do with it?
Developer: I am going to cut out early.
By the way, I ran a script and created a bunch of
untagged EC2 instances.
Can you help me find them?
Have a great weekend!
You: What the #*$%!
R

44. Tag AWS resource properly
Find untagged EC2 instances
• sourcetype=aws:description source="*:ec2_instances" NOT "tags.Name"=*| table
region id instance_type ip_address key_name
Define a naming conventions for EC2 instance and enforce it
• DLA_Jove_testEC2Cmd. D: Dev, L: Linux, A: AWS project
• <Role><OS><Project>_<Owner><Note>
• sourcetype=aws:description source="*:ec2_instances" (NOT "tags.Name"=*) OR
("tags.Name"=* tags.Name!=Q* tags.Name!=D* tags.Name!=P* tags.Name!=U*)
R

45. Just use the “Name” tag
4R

46. 48
Splunk app for AWS
demo
R

47. Splunk runs on and with AWS
SOC2 Type II Certified
Cloud Services Apps
Splunk Add-on for AWS
Splunk App for AWS
Specific
Integrations
Config, CloudTrail, CloudWatch,
VPC Flow Logs, Lambda: AWS IoT,
Amazon Kinesis: AWS
CloudFormation
Splunk Core + Enterprise
Security & ITSI available
Enterprise on AWS
For small IT teams, starts $3/day
Software
Apps and Integrations
As a Service on AWS
Delivery Models
For small IT teams, starts $75/mo
R

48. Launched: Splunk Light w/ app for AWS
Multiple use cases across one platform
Splunk Light AMI on AWS Marketplace
Free 20GB License
6 Month Term = $6,000 Value
Bundled with App for AWS
Go To: https://aws.amazon.com/marketplace/ & Search “Splunk Light”
Demos available at AWS Re:Invent Booth #206

49. Thank you!
51
Contact:
scottjpack@gmail.com
github.com/scottjpack
Twitter: @scottjpack
Contact:
randall.young@gmail.com
Twitter: @drandallyoung

50. Remember to complete
your evaluations!