DevOops Redux Ken Johnson Chris Gates - AppSec USA 2016

DevOops, Redux
Chris Gates, Ken Johnson
AppSec USA 2016

Background: KJ
• I’m NOT Kevin Johnson

Background: KJ
•I’m NOT Ken Bone

Background: KJ
•I AM Ken Johnson
•CTO of nVisium - @cktricky
•Former US Navy
•Topics I’ve talked about:
– Rails Security (Railsgoat)
– Building an AppSec Program
– DevOops: Common Flaws in DevOps Tooling
– Exploitation of Web Applications

Background: KJ
• I run engineering (product)
• I work for a security company
• I have some concerns...same as you

Background: CG
• Chris Gates, Sr. Security Engineer - Uber
• Former Army
• Topics I’ve talked about:
– Breaking into Oracle, Windows, lots of stuff
– Phishing
– Low to Pwned
– Purple Teaming
– DevOops – Common Flaws in DevOps Tooling

Background: CG
• Was a full time breaker
• Now full-ish time fixer
• Currently doing Blue Team stuff
- <3 Python + REST APIs
- Astonished at # of ppl who can’t Internet

About This Talk
• Original talk DevOops was about breaking stuff
• We were asked about “Proactive” measures in
DevOps/Agile/CI-CD environments – Quick Story
• We made a solution focused model based on
“Common” architecture and needs

Before We Begin
• Buckle up, lots of info coming your way
• Q&A will be reserved for hallway discussions
• Slides will have all the resources you need and will be
available
• Sections are broken up between Human, Host, and
Infrastructure

Employee Intelligence
(Human)
Making it difficult (for employees) to allow
attackers to walk into our environment

Monitoring External Services
• Numerous ways for employees to accidently release
data
–Pastebin-like sites
–Github
•Gists
•Code
• Examples:
–Slack tokens in github
–AWS configs in .dotfiles back ups
–Tokens in logs/dumps/snippets

Monitoring GitHub
• How you could tackle the problem:
–Use GitLab (internal)
–Use gitolite (internal)
–Use GitHub Enterprise (internal)
–Use Phabricator (internal)

Monitoring GitHub
• But you won’t, you’ll set up a private GitHub for you org
like everyone else.
– Now you need to monitor when people post your
private stuff on their personal repo
– It happens. A lot.

Monitoring GitHub
• How you could tackle the problem:
–Have employees join the your GitHub organization
–Regularly crawl the list of members
–Check out all their repos
–Run regex against all files looking for known badness

Monitoring GitHub
• Gitrob
–https://github.com/michenriksen/gitrob

Monitoring Pastebin*
• Pastebin*
–https://github.com/jordan-wright/dumpmon
–https://github.com/xme/pastemon
–https://github.com/cvandeplas/pystemon

Monitoring Goals
• DumpMon https://github.com/jordan-wright/dumpmon

Monitoring Goals
• For Pay Services

Monitoring Goals
•For Pay Services - https://gitmonitor.com/

Monitoring Goals
GitMonitor - Some options they provide

Workstation Protection
(Host)
Protecting and monitoring employees on
their development workstations (and servers
too)

Host Protections
Developer Laptop Hardening
• osquery (OS X/Linux)
• Doorman
• BlockBlock
• Little Snitch
• CarbonBlack / Sysmon
• Splunk
• Simian

Host Protections
• osquery (https://osquery.io/)
• “osquery is an operating system instrumentation framework for OS X, Linux,
and FreeBSD. The tools make low-level operating system analytics and
monitoring both performant and intuitive.”
• “osquery exposes an operating system as a high-performance relational
database. This allows you to write SQL queries to explore operating system
data. With osquery, SQL tables represent abstract concepts such as
running processes, loaded kernel modules, open network connections,
browser plugins, hardware events or file hashes.”

Host Protections
osquery
• Adhoc
• Scheduled
• Schedule query
• Collect logs
• Review change
• File Integrity Monitoring
• Yara rules
• Query packs

Host Protections
• Doorman (https://github.com/mwielgoszewski/doorman)
• “Doorman is an osquery fleet manager that allows
administrators to remotely manage the osquery
configurations retrieved by nodes.”

Host Protections
• BlockBlock (https://objective-see.com/products/blockblock.html)
• Kernel hook to identify any time software wants to persist
• Prompt to allow or deny
• “The kernel extension tracks process creations, which are consumed by the
daemon, which also monitors various persistence locations to detect any
new items. Specifically the daemon (currently) watches for new kexts,
launch daemon & agents, and new login items via the fsevents device
(/dev/fsevents).”

Host Protections
• Little Snitch (https://www.obdev.at/products/littlesnitch/index.html)
• Host based firewall
• Prompt to allow or deny and for how long
• “Little Snitch intercepts these unwanted connection
attempts, and lets you decide how to proceed.”

Host Protections
• CarbonBlack (https://www.carbonblack.com/)
• Host based agent
• Monitor process create, writes, registry queries, net connections
• Create rules/watchlist for known bad behavior
–Mimikatz-->company_name:*gentilkiwi*
–FileVault Encryption Disabled -->process_name:fdesetup
cmdline:disable
–Unsigned JAR exec-->process_name:*.jar digsig_result:
(digsig_result:"Unsigned")
–OSX dump user hashes-->process_name:dscl
cmdline:ShadowHashData

Host Protections
• Sysmon
• https://www.root9b.com/sites/default/files/whitepapers/R9B_blog_005_white
paper_01.pdf
• https://jon.glass/tag/sysinternals/
• http://www.darkoperator.com/blog/2014/8/8/sysinternals-sysmon
• https://www.bsk-consulting.de/2015/03/21/detect-system-file-manipulations-
with-sysinternals-sysmon/
• https://www.firemon.com/enhance-windows-anomaly-detection-sysmon/

Host Protections
OSX Patch Management - Simian
• “Simian is an enterprise-class Mac OS X software
deployment solution.”
• Allows you to push munki updates
• Free / OSS
• Runs on google cloud
• Project: https://github.com/google/simian

Host Protections
Why do we bring this up?
• Some people aren’t aware you can perform free OSX
patch management
• There are a lot of OSX developer shops without an
“enterprise budget”
• Patch management is a no-brainer and security 101

Host Protections
https://github.com/google/simian

Host Protections
Simian Consists of 2 parts:
• Client
– Private and Public SSL Keys used to authenticate
– Configuration unique per OSX client
• Web Application/Server
– Runs on Google Cloud
– Keep in mind its free but… not for long (eventually costs a little for
storage)
Takes about a week to learn and get setup

Host Protections
Web Application used to Manage Updates

Host Protections
Client - DMG File

Host Protections
Simian Recap:
• Learning curve is moderately difficult IMO
• Free-ish (eventually storage costs but still very minimal)
• Useful for patch updates and monitoring clients systems
for low disk space, uptime, etc.

Production Protection
(Infra)
Protecting and monitoring production
environments (AWS)

My AWS Goals
• Harden – Make it difficult to reach your AWS
environment
• Monitor – If your AWS environment is breached, you
need to know and alert yourselves
• Restore – Have the ability to reconstruct data/configs
after a “hack”

AWS’s Plan
• Took the AWS Security Fundamentals Course and…
– Fortunately, our strategy lines up with AWS recommendations
– You are responsible for leveraging the tools AWS provides
(financially)
– Your configuration… that is on you
– https://aws.amazon.com/training/course-descriptions/security-
fundamentals/

AWS Hardening Basics
Making it difficult (for attackers) to
reach our environment

Hardening Checklist
1. Don’t Use The Root Account!
2. Disable Access Keys for Root Account
3. Multi-Factor Authentication
4. API + MFA
5. Strong Password Policy

Don’t Use Root Account
• Every AWS env has a root account, only necessary to
use for very specific circumstances
• When these circumstances arise, notify your team that
the account will be used
• We will discuss why this is important when we talk about
CloudWatch metrics

Disable/Delete Root Account Access Keys
• Just delete them if they exist
– Disable the access keys in the event you are unable
to delete them completely for some reason
• Make sure your admins have a (verbal/written) policy
that states “we don’t create access keys for the root
account”

MFA
• If credentials are stolen or guessed, we want a second
layer of protection
• You can use apps or hardware to do this
– Google Authenticator (Apps)
– Gemalto (Hardware)
• Find the full list of MFA devices here:
https://aws.amazon.com/iam/details/mfa/
• This is so ridiculously easy to do, everyone should do it

MFA
See the published slide deck for step by step
instructions

MFA
• At this point, it's worth mentioning that non-
administrators or those without IAM privileges cannot
enable MFA on their own account
• Why is this a problem? Well, they need to be able to
enable MFA on their own device… not the
administrator’s
• Fortunately, we have a solution!

MFA
• Okay so that wasn’t the easiest to read, so here is the
link:
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_c
redentials_delegate-permissions_examples.html#creds-
policies-mfa-console
• Basically this IAM policy allows a user to manage their
*OWN* MFA device

MFA (for Root Account)
• Need a shared MFA for root? TOTP!
• Recommend using something like 1password for teams,
can share the TOTP code:
https://support.1password.com/guides/mac/totp.html
https://www.youtube.com/watch?v=eZyb-ArMK9g

API + MFA
• You have the ability to place a restriction where
resources can only be interacted with if the user has
authenticated with MFA
• This helps prevent (ab)use should someone steal access
keys or credentials

API + MFA
• This entry enforces MFA for Web/API
• Do this for Admin & Power-User groups at a minimum

API + MFA
• Truth be told, doing this can be painful at first
• Things that used to work, might not (via the API)
• Fortunately, we have some answers for you
• Firstly, let’s discuss STS or SecurityToken Service

API + MFA
• Leverage STS in order to interact with the AWS API
should this MFA restriction be placed on resources (and
it should ☺ )
• Example of using STS:
https://gist.github.com/cktricky/127be4e431563a986f0f

API + MFA
Use the creds to leverage tools like ec2-api-tools
(-O <access key id>–W <secret> and –T <session token>)

API + MFA
And in case you don’t like Ruby…
https://github.com/jimbrowne/aws-sts-helpers

API + MFA
• ElasticBeanstalk does not work with STS. Le Terrible.
• However, there is a workaround, use CodePipeline
• Very simple process to setup but only works with:
– GitHub
– AWS CodeCommit
– Amazon S3

Password Policy
• Password policies are important because historically
people do not choose complex passwords
• MFA should help, but we’re talking about a layered
approach
• Again, making our AWS environment harder to reach

Hardening Recap
• Make credentials hard to guess
• If guessed or stolen, we still have MFA
• Remember MFA only protects against the web and NOT
the API… unless you change your policies and use STS
• Root account is King, protect your King

Hardening Recap
• Things we did not (and won’t discuss)
– S3 bucket policies
– Security Group configurations
– SSH Key Management
– Encrypting Data (Volumes, S3 buckets)
• Trusted Advisor – Use it, because it catches a lot of “low
hanging fruit” style issues

Hardening Recap
• Links to resources that discuss the items we’re not
covering:
– https://d0.awsstatic.com/whitepapers/compliance/AWS_Auditing_Securi
ty_Checklist.pdf
– http://aws-de-media.s3.amazonaws.com/images/Produktblaetter/AWS-
Security-Check-List_eng.pdf
– http://www.slideshare.net/AmazonWebServices/masterclass-advanced-
security-best-practices
• Frankly you can’t throw a rock without hitting some basic
info regarding AWS Security Checklists

AWS Monitoring
Detecting malicious activity

AWS Monitoring
• Assuming hardening (prevention) has failed, how would
we know?
• Luckily, AWS provides several services which alert to
anomalies
• We will walk through examples of using these services,
but ultimately decide what is right for you
• Fair warning, some of these services will provide a lot of
noise

AWS Monitoring
4 important services:
• CloudTrail – Logs
• SNS – Notifications
• Config – Alerts for modifications & noncompliance
• CloudWatch – Alerts for specific types of behavior

AWS Monitoring (CloudTrail)
• CloudTrail is primarily used for log collection
• Other services like CloudWatch, for example, use those
logs to filter relevant data

Pretty easy, first turn it on..

Configure the log group

Allow the creation of an IAM role by CloudTrail

• At this point you have cloudtrail enabled
• Next step, BEFORE moving to CloudWatch or Config, is
configuring SNS topics

AWS Monitoring (SNS)
• Fantastic offering, <3 it
– Examples of ways to be notified by SNS
• SMS
• Email
• JSON Post to your Application’s API endpoint

• Receive SMS/Email/Slack notifications for important
events
• ^ This is so you get immediate notifications
• You can have multiple subscribers, I’d suggest you use
that functionality
• Basic gist? Receive immediate updates for things you
want to see… immediately ☺

Create a topic

Create Subscription

Create SMS (or whatever, but in this case, SMS)

Example of creating email subscription… bottom line you
can have multiple ways of notifying people

AWS Monitoring (Config)
• Config:
– AWS resource inventory, configuration history, and
configuration change notifications
– Can either design custom Config rules or use
managed (pre-packaged) AWS Config rules
– Discovery -Change Management
– Compliance -Incident Response

• Pre-packaged “Managed” AWS Rules
– CLOUD_TRAIL_ENABLED
– EIP_ATTACHED
– ENCRYPTED_VOLUMES
– INCOMING_SSH_DISABLED
– INSTANCES_IN_VPC
– REQUIRED_TAGS
– RESTRICTED_INCOMING_TRAFFIC
https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_use-managed-rules.html

• Examples of things you can have alerts set for:
– Change in Firewall (Security Group) ports
– Changes in VPC
– Any change… at all

Go to the Config service and choose resources to track

Or choose to track everything

Create a bucket, create an SNS topic (…we’ll discuss next)

Allow the role to be created and you’re all set!

AWS Monitoring (CloudWatch)
• We can be very particular here about what it is we want
to see
• Some very interesting things you can monitor
• Some examples:
– Billing Alerts (Important for detection of abuse or
mistakes)
– Track Root Account Usage
– Failed login attempts

AWS Monitoring (CloudWatch - Billing)
• Used to prevent abuse or mistakes from costing your
organization money
• Analyze and approximate your monthly spend
• Configure via CloudWatch
• Use SNS for instantaneous alerting

Navigate to billing & cost management; enable billing alerts

Create an SNS topic

Subscribe to Topic

Navigate to CloudWatch -> Metrics -> Billing

Choose USD/Estimate Charges -> Create Alarm

Set price point, SNS topic, and create alarm

Exact steps to enable can be found here:
http://docs.aws.amazon.com/awsaccountbilling/latest/about
v2/free-tier-alarms.html

AWS Monitoring (CloudWatch – Root Login)
• Remember how I said don’t use the Root account
routinely?
• BUT… if this account is used, you should know about it
• This is the reason you’ll want to notify others (who
receive SNS alerts) of the fact you are about to use the
account

Choose log group, create metric

Define Logs Metric Filter

Assign/Create Filter

Click “Create Alarm”

Define Alarm and you’re good…

Exact steps (with pics) exist here:
https://blogs.aws.amazon.com/security/post/Tx3PSPQSN8
374D/How-to-Receive-Notifications-When-Your-AWS-
Account-s-Root-Access-Keys-Are-Used

AWS Monitoring (CloudWatch – Failed Logins)
• In the event someone is trying to break in, let’s alert
ourselves to this!
• Failed logins typically suggest either someone forgot
their password or… someone is trying to guess yours

• In the interest of time… the steps are pretty much the
same as the root login alarm
• The Regex Filter however, is different

Relevant filter pattern

• Exact steps exist here:
http://docs.aws.amazon.com/awscloudtrail/la
test/userguide/cloudwatch-alarms-for-
cloudtrail.html#cloudwatch-alarms-for-
cloudtrail-signin

AWS Monitoring (Unauthorized Activity)
• Remember the aws-interrogate tool?
• This alarm is the antidote
• Alerts us when someone is trying to access something
in AWS, and does not have permissions

• Again, in the interest of time, steps are same as root
login
• Regex is of course, different

Set up regular expression

What happens when we run interrogate

The result of doing that is a nice nifty email to the
engineering & security team

AWS Monitoring (CloudWatch) – Filter Patterns
• If you’d like to create your own custom filter patterns,
here is a resource for that:
http://docs.aws.amazon.com/AmazonCloudWatch/latest/De
veloperGuide/FilterAndPatternSyntax.html

AWS + Splunk
• Splunk is a pretty great resource for monitoring activity
– Two separate plugins:
• Splunk App for AWS
– https://splunkbase.splunk.com/app/1274/
• Splunk Add-On
– https://splunkbase.splunk.com/app/1876/

AWS + Splunk
• Examples of things you can view:
– Billing
– Topology
– Usage
– IAM Activity
– SSH Key Pair Activity
– User Activity
– Network ACL(s)
– VPC Activity
– and a lot more…

AWS + Splunk
• Splunk will need an AWS account in order to retrieve
data
• Create account(s) for Splunk, grab the necessary
permission policy from here:
http://docs.splunk.com/Documentation/AddOns/released/A
WS/ConfigureAWSpermissions

AWS + Splunk
Configure AWS App for Splunk, add account(s), configure
each input accordingly:

AWS + Splunk
•To view things like IAM Activity…
–Subscribe to a cloudtrail log via SNS
–Utilize SQS and subscribe SQS to an SNS Topic

Monitoring Recap
• Alert yourself when things change
• This will get noisy, find a way to filter that which is important
– If it’s a high risk event, send an SMS/Slack/Email blast
• At a minimum, alert yourself when odd things occur… like:
– Billing increases past your normal spend
– When somebody authenticates as Root
– When someone has a login failure

Monitoring Recap
• Interesting Quora thread:
– https://www.quora.com/My-AWS-account-was-hacked-and-I-have-a-50-000-bill-how-can-I-
reduce-the-amount-I-need-to-pay
• Highlights from the article:
– AWS has “a review board of sorts” to determine if you should
be refunded
– Bots are scouring GitHub searching for exposed access keys
– One of the more AWS-seasoned responders mentioned doing
part of what we discussed here today to avoid it
– A decent number of the people posting on this thread said
“Yes, happened to me too”

AWS Restoration & Recovery
Plan to fail, just don’t fail to plan

• Do not USE AWS TO BACKUP YOUR AWS
• Offsite backups (meaning, off AWS site)
• Common things to back-up:
– Databases/ Snapshots
– S3 Buckets
– EBS Volumes
– CloudFormation Templates

• Resources:
– http://stackoverflow.com/questions/17087542/backup-
solutions-for-aws-ec2-instances
– https://github.com/Scalr/installer-ng
– http://www.n2ws.com/blog/3-ways-ec2-windows-
backup-and-recovery.html

AWS Incident Response
Plan to fail, just don’t fail to plan

AWS Incident Response
• Could be its own talk
• Scout 2 -- https://github.com/nccgroup/Scout2
•Andrew Krug & Alex McCormack – Hardening AWS
Environments and Automating Incident Response
– https://www.youtube.com/watch?v=cmEUxxYFjK8

Contact
Chris Gates
Twitter: @carnal0wnage
Blog:
https://carnal0wnage.attackresearch.com
Ken Johnson
Twitter: @cktricky

DevOops Redux Ken Johnson Chris Gates - AppSec USA 2016

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (14)

Ähnlich wie DevOops Redux Ken Johnson Chris Gates - AppSec USA 2016

Ähnlich wie DevOops Redux Ken Johnson Chris Gates - AppSec USA 2016 (20)

Mehr von Chris Gates

Mehr von Chris Gates (12)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

DevOops Redux Ken Johnson Chris Gates - AppSec USA 2016