Wrangling Security Events in the Cloud

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Don “Beetle” Bailey, AWS Security
Josh Du Lac, AWS Professional Services
October 2015
SEC308
Wrangling Security Events
in The Cloud

What to expect from this session
• Tactical follow-on to previous talks
• Concrete examples of potential events and how you can
handle them
• Ideas for increasing security agility
• Specific AWS mechanisms to leverage
• More than 1 way to catch a cat burglar, so reinvent as
needed
• Relevant resources, including docs, code, and partners

“Intrusion Detection in the Cloud” redux
• AWS-specific areas to monitor for security-concerning
events
• Prerequisites
• Key concepts, such as security role, write-once storage
• Key services to leverage, events and behaviors to look
for
• Example detection of key configuration changes,
resource usage anomalies
• YouTube search “Intrusion Detection in the Cloud”

“Incident Response (IR) in the Cloud” redux
• Ensuring your existing IR process considers AWS
• More prerequisites
• Mechanisms for mitigation and investigation
• Tactics specific to AWS IR, such as constraining exposed
AWS credentials
• Tactics analogous to traditional IR, modified for AWS, such as
Amazon EC2 instance memory dumping, analysis
• YouTube search “Incident Response in the Cloud”

Security event wrangling = Response in depth
• Types of security events
• Detect -> Recover
• Investigate -> Protect
• Leveraging AWS mechanisms for increased security
agility

Example events of concern, signatures
• Configuration changes that impact ability to detect or
understand events
• Activities that are inconsistent with expectations
• Activities that violate policy
• Resources no longer available
• Resources more available than desired
• Event detection signatures != commercial product, and
may require careful thought vs. operations to develop

Protect, detect, react, recover, etc.
Protect
Detect
Recover
Investigate

AWS = Agility for security geeks
• Ability to programmatically inventory environment—
knowing what you need to protect is key
• Awareness of what’s happening, what’s changing, from
AWS API activity to application behavior
• Detection and alerting mechanisms, freedom to create
and flexibility to configure and tune what’s appropriate
for YOU
• Analysis and response, via the same platform, natively
or with AWS partner solutions

AWS CloudTrail
• Records AWS API calls for your account and delivers log
files to you.
• Turn it ON!
http://docs.aws.amazon.com/awscloudtrail/latest/usergui
de/cloudtrail-user-guide.html

CloudTrail events
• A record in JSON format that contains information about
requests for resources in your account.
• Describes which service was accessed, what action was
performed, and any parameters for the action.
• Helps you determine who made the request.
• The event data is enclosed in a Records array.
http://docs.aws.amazon.com/awscloudtrail/latest/usergui
de/send-cloudtrail-events-to-cloudwatch-logs.html

Example CloudTrail event
"Records": [{
"eventVersion": "1.0",
"userIdentity": {
"type": "IAMUser",
"principalId": "EX_PRINCIPAL_ID",
"arn": "arn:aws:iam::123456789012:user/Alice",
"accountId": "123456789012",
"accessKeyId": "EXAMPLE_KEY_ID",
"userName": "Alice"
},
"eventTime": "2015-03-24T21:11:59Z",
"eventSource": "iam.amazonaws.com",
"eventName": "CreateUser",
"awsRegion": "us-east-1",
"sourceIPAddress": ”55.55.55.55",
"userAgent": "aws-cli/1.3.2 Python/2.7.5 Windows/7",
"requestParameters": {
"userName": "Bob"
},
"responseElements": {
"user": {
"createDate": "Mar 24, 2015 9:11:59 PM",
"userName": "Bob",
"arn": "arn:aws:iam::123456789012:user/Bob",
"path": "/",
"userId": "EXAMPLEUSERID"
}
....

CloudTrail OFF
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAI5WIMUDR2UZUI62VO",
"arn": "arn:aws:iam::000123456789:user/reinvent-sec308",
"accountId": "000123456789",
"accessKeyId": "AKIAIRAHHRD3PHLUFJLQ",
"userName": "reinvent-sec308"
},
"eventTime": "2015-09-23T00:41:45Z",
"eventSource": "cloudtrail.amazonaws.com",
"eventName": "StopLogging",
"awsRegion": "us-west-2",
"sourceIPAddress": “55.55.55.55",
"userAgent": "aws-cli/1.7.25 Python/2.7.5 Darwin/13.4.0",
"name": "CloudTrail-Default"
},
"responseElements": null,
....

Amazon CloudWatch Logs
• Monitor, store, and access your log files from Amazon
EC2 instances, AWS CloudTrail, or other sources.
• Enable in the AWS Management Console, CLI, or via
AWS CloudFormation.
• Monitor and alarm for specific phrases, values, or
patterns.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/
DeveloperGuide/WhatIsCloudWatchLogs.html

CloudFormation -> CloudWatch alarms
• Downloadable and editable example CloudFormation template from
AWS
• Contains predefined CloudWatch metric filters and alarms that
enable you to receive email notifications when certain security-
related API calls are made in your AWS account
• Amazon S3 bucket events, network events, Amazon EC2 events,
AWS CloudTrail, and AWS Identity and Access Management (IAM)
events
http://docs.aws.amazon.com/awscloudtrail/latest/userguide/use-
cloudformation-template-to-create-cloudwatch-alarms.html

CloudTrail OFF event – Detect
"CloudTrailStopMetricFilter": {
"Type": "AWS::Logs::MetricFilter",
"Properties": {
"LogGroupName": { "Ref" : "LogGroupName" },
"FilterPattern": ”{ ($.eventName = StopLogging) }",
"MetricTransformations": [
{
"MetricNamespace": "CloudTrailMetrics",
"MetricName": "CloudTrailEventCount",
"MetricValue": "1"
}
]
}
},

CloudTrail OFF event – Detect
"CloudTrailStoppedAlarm": {
"Type": "AWS::CloudWatch::Alarm",
"Properties": {
"AlarmName" : ”CloudTrailStoppedAlarm",
"AlarmDescription" : "Alarms when StopLogging API call is made",
"AlarmActions" : [{ "Ref" : "AlarmNotificationTopic" }],
"MetricName" : "CloudTrailEventCount",
"Namespace" : "CloudTrailMetrics",
"ComparisonOperator" : "GreaterThanOrEqualToThreshold",
"EvaluationPeriods" : "1",
"Period" : "300",
"Statistic" : "Sum",
"Threshold" : "1"
}
},

CloudTrail OFF event – Recover

CloudTrail OFF event – Investigate
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAI5WIMUDR2UZUI62VO",
"arn": "arn:aws:iam::000123456789:user/reinvent-sec308",
"accountId": "000123456789",
"accessKeyId": "AKIAIRAHHRD3PHLUFJLQ",
"userName": "reinvent-sec308"
},
"eventTime": "2015-09-23T00:41:45Z",
"eventSource": "cloudtrail.amazonaws.com",
"eventName": "StopLogging",
"awsRegion": "us-west-2",
"sourceIPAddress": "55.55.55.55",
"userAgent": "aws-cli/1.7.25 Python/2.7.5 Darwin/13.4.0",
"name": "CloudTrail-Default"
},
....

CloudTrail OFF event – Protect
Deny permissions for CloudTrail in IAM groups or roles
{
"Sid": "Stmt0001",
"Effect": "Deny",
"Action": [
"cloudtrail:DeleteTrail",
"cloudtrail:StopLogging"
],
"Resource": [
"*"
]
}

Multi-Factor Authentication (MFA)
• Require unique authentication codes to access AWS
websites or services
• Hardware or virtual authentication device generates
codes
• Enter codes manually via AWS Management Console or
accompany API requests
• Configure via IAM
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_cr
edentials_mfa.html

MFA Deactivate Event
.....
"eventTime": "2015-09-20T18:53:02Z",
"eventName": "DeactivateMFADevice",
"userAgent": "signin.amazonaws.com",
"userName": ”bob",
"serialNumber": "arn:aws:iam::000019241430:mfa/bob"
},
"requestID": "d1a9ebf8-5fc8-11e5-9d8f-1bc7c6757e61",
.....

MFA Deactivate Event – Detect
"MFADeactivateMetricFilter": {
"Properties": {
"FilterPattern": "{ ($.eventName=DeactivateMFADevice) }”,
{
"MetricName": "MFADeactivateEventCount",
"MetricValue": "1"
}
]
}
},

MFA Deactivate Event – Recover
Reconfigure the MFA device

MFA Deactivate Event – Investigate
.....
"eventTime": "2015-09-20T18:53:02Z",
"eventName": "DeactivateMFADevice",
"userAgent": "signin.amazonaws.com",
"userName": ”bob",
"serialNumber": "arn:aws:iam::000019241430:mfa/bob"
},
"requestID": "d1a9ebf8-5fc8-11e5-9d8f-1bc7c6757e61",
.....

MFA Deactivate Event – Protect
Use AWS Identity & Access Management to require MFA
http://blogs.aws.amazon.com/security/post/Tx2SJJYE082KBUK/How-to-Delegate-
Management-of-Multi-Factor-Authentication-to-AWS-IAM-Users

S3 object deletion event – Detect
• Bucket logging? Check.
• Bucket versioning? Check.
• Continuously reviewing logs …? NO
• We can enable push notifications for S3 events that
might concern us (for example, deletions)
• Configure S3 to detect events like ObjectRemoved
• S3 sends alert to the Amazon SNS topic of your choosing
• SNS topic sends message to subscribers, such as an email
to your security_team@yourcompany.com

S3 object deletion event – Recover
• Restore deleted file from previous version.
• Via AWS Management Console, just a couple clicks to
download/upload deleted version.
• Via CLI/API, just an S3 copy object request, specifying
version ID with copy source.
• If you enabled versioning AFTER initial object put,
version ID will be “NULL”. OK, you can still specify
“NULL” as a version to restore from.

Recover deleted S3 object – AWS CLI
aws s3api list-object-versions --bucket
reinvent2015-sec308 --prefix prod
aws s3api copy-object --bucket reinvent2015-sec308 -
-copy-source reinvent2015-
sec308/prod/important.txt?versionId=null --key
prod/important.txt

Recover deleted S3 object (from backup) – AWS CLI
aws s3api copy-object --bucket reinvent2015-sec308 -
-copy-source reinvent2015-
sec308/backup/important.txt?versionId=null --key
prod/important.txt

S3 object deletion event – Investigate

S3 object deletion event – Protect
• Bucket versioning protects against inadvertent delete or
overwrite of objects.
• Consider more restrictive policies for credentials, such
as specifically disallow S3 object removal.
• Additional layer of protection; enable MFA Delete on a
versioned S3 bucket.
http://docs.aws.amazon.com/AmazonS3/latest/dev/Versi
oning.html#MultiFactorAuthenticationDelete

Log-in anomaly event – Detect
"ConsoleSignInAnomalyMetricFilter": {
"Properties": {
"FilterPattern": "{ ($.eventName = ConsoleLogin) &&
($.sourceIPAddress != 55.55.*) }",
{
"MetricName": "ConsoleSignInAnomalyCount",
"MetricValue": "1"
}
]
}
},

Log-in anomaly event – Recover
Add null IAM policy to the user (Deny all permissions):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": [
"*"
],
"Resource": [
"*"
]
}
]
}

Log-in anomaly event – Investigate
Look in CloudTrail – Determine what events happened
after the ConsoleLogin.

Log-in anomaly event – Protect
Add Condition statements to IAM
"Condition" : {
"IpAddress" : {
"aws:SourceIp" : [”55.55.0.0/16”]
}
}

Open security group
• 0.0.0.0/0 ingress has limited validity, but commonly used.
• Web server = Likely OK for the Internet to access 80/443.
• All of the web server’s OTHER ports? Likely NOT OK to
access the Internet.
• Policies can vary. No admin ports open to the world? OK.
• Creation and change velocity among security groups
should be LOW.

AWS Config
• AWS resource inventory, configuration history, and
configuration change notifications
• Discover existing AWS resources
• Export inventory of your AWS resources with all configuration
details
• Determine how a resource was configured at any point in
time
• Security geeks should LOVE it!
http://aws.amazon.com/documentation/config/

Open security group event – Detect
• Subscribe to AWS Config notification topic.
• Filter notifications for creation of security groups that
might be concerning. You could look for the following,
individually or combined:
• “SecurityGroup” and “Created” within subject
• changeType : “CREATE” within body
• resourceType: "AWS::EC2::SecurityGroup” within body

Open security group event – Detect
"groupId": "sg-7dc0d21a",
...
"ipPermissions": [
{
"ipProtocol": "-1",
"fromPort": null,
"toPort": null,
"userIdGroupPairs": [],
"ipRanges": [
"0.0.0.0/0"
],
"prefixListIds": []
}
],
...

Open security group event – Recover
• If responding soon enough to the creation of a new
security group and no instances, simply delete the
security group.
• Otherwise, assign running instances to another security
group, and then delete the offending security group.
• You can’t delete a default security group, but you can
change its rules back to something sane, including no
rules.

Delete open security group – AWS CLI
aws ec2 delete-security-group --no-dry-run --group-
id sg-d3bda2b4

Open security group event – Investigate
• Revisit the AWS Config change notification.
• Note time, action, and security group ID to correlate to
principal and source IP of EC2 API call via AWS
CloudTrail.
• If possible, engage principal to understand intent or
determine if unexplained, such as by external actor and
potentially malicious.

Open security group event – Protect
• Appropriately constrain or deactivate associated
credentials as warranted.
• Security group changes, particularly within production,
should not be a frequent event, so maintain high
vigilance.

Unapproved AMIs
Amazon Machine Images
• Public AMI
• Marketplace AMI
• Private AMI
• Approved AMIs/“Golden” AMIs

Unapproved AMI event – Detect
• Compare launched EC2 instances against a whitelist.
• What is a good method to compare against a whitelist?

Let’s use AWS Lambda!
• Runs your code in response to events.
• Automatically manages compute resources for you.
• Create new back-end services where compute
resources are automatically triggered based on custom
requests.
• You can read CloudTrail events with AWS Lambda.
http://docs.aws.amazon.com/lambda/latest/dg/welcome.html

Unapproved AMI event - Recover
matchingRecords,
function(record, complete) {
var params = {
InstanceIds: []
};
// List each instance ID
for (var i = 0; i < record.responseElements.instancesSet.items.length; i++){
params.InstanceIds.push(record.responseElements.instancesSet.items[i].instanceId);
}
// Terminate the enumerated instances
ec2.terminateInstances(params, complete);

Unapproved AMI event – Investigate
Interrogate CloudTrail logs as before
• Who launched it?
• Where did the request come from?
• Which subnet was it being launched into?

Unapproved AMI event – Protect
Restrict access in IAM to specific AMIs IDs

Automate IR?
• Most, if not all, of the pieces to automate IR exist in AWS
• Automated IR = Even greater security agility
• Detect -> Protect programmatically
• Lambda-fy your IR!

Detecting events in Lambda
…
var EVENT_SOURCE_TO_TRACK = /cloudtrail.amazonaws.com/;
var EVENT_NAME_TO_TRACK = /StopLogging/;
var matchingRecords = records
.Records
.filter(function(record) {
return record.eventSource.match(EVENT_SOURCE_TO_TRACK)
&& record.eventName.match(EVENT_NAME_TO_TRACK);
});
…
Source: http://docs.aws.amazon.com/lambda/latest/dg/wt-cloudtrail-events-
adminuser.html

Responding to events in Lambda
…
if (matchingRecords.length >= 1) {
console.log(’StopLogging detected! Reverting...');
cloudtrail.startLogging(cloudtrailParams, function(err, data) {
….

Responding to events in Lambda

Building a “Lambda Responder”
CloudTrail S3
Lambda
Lambda
SNS

1. Turn on AWS CloudTrail – Choose an S3 bucket.
2. Create an SNS topic.
3. Update the topic policy to allow event notifications from your
S3 bucket.
4. Configure your S3 bucket to send event notifications to the
SNS topic.
5. Create an IAM role for the Lambda functions.
6. Create the Lambda functions and process SNS messages.
https://aws.amazon.com/blogs/compute/fanout-s3-event-
notifications-to-multiple-endpoints/ by John Stamper

• What could you automatically respond to?

Lambda – Automated S3 object recovery
...
var bucket = event.Records[0].s3.bucket.name;
var key = event.Records[0].s3.object.key;
var backup = ’your-backup-bucket/' + key;
var params = {
Bucket: bucket,
CopySource: backup,
Key: key,
};
s3.copyObject(params, function(err, data) {
// removed for brevity
});
...

Lambda – Automated open security group delete
var snsMsgString = JSON.stringify(event.Records[0].Sns.Message);
var snsMsgObject = getSNSMessageObject(snsMsgString);
if (snsMsgObject.configurationItemDiff.changeType == 'CREATE' &&
snsMsgObject.configurationItem.resourceType == 'AWS::EC2::SecurityGroup' &&
snsMsgObject.configurationItem.configuration.ipPermissions[0].ipProtocol == '-1' &&
snsMsgObject.configurationItem.configuration.ipPermissions[0].ipRanges == '0.0.0.0/0')
{
var params = {
DryRun: false,
GroupId: snsMsgObject.configurationItem.resourceId,
};
ec2.deleteSecurityGroup(params, function(err, data) {
context.succeed(snsMsgObject);
});
}

AWS Config -> Lambda … IR aaS? AWS Config
Rules!
• Extends AWS Config with a powerful new rule system
• Use existing rules from AWS and from partners
• You can also define your own custom rules
• SEC314 - NEW LAUNCH! AWS Config/Config Rules:
Use AWS Config Rules to Improve Governance over
Configuration Changes to Your Resources

Practice makes perfect
• IR game day…YAY!
• Tabletop first…yay?
• See SEC316 – Harden Your Architecture with Security
Incident Response Simulations (SIRS), Jon Miller and
Armando Leite

AWS Partner, Dell SecureWorks, IR Support
• Customer IR case example
• Our IR preparedness “Wish List” for AWS customers
• How to contact us

IR Case Example – Background, Event
• Dell SecureWorks contacted by an AWS customer, a provider of cloud-
based collaboration software
• Customer investigated abnormally high CPU usage on Internet-facing
servers hosting their customers’ applications
• Customer’s review of system logs identified unauthorized logins from a wide
array of IP addresses using compromised credentials
• Threat actors leveraged the Customer’s compromised web app credentials
to gain unauthorized entry and propagate to a multitude of connected
resources within the Customer’s AWS environment
• Dell SecureWorks performed digital forensics on the Customer’s web
applications, AWS instances and snapshots, AWS CloudTrail logs, and
suspected on-premise systems

IR Case Example - Response
• Dell SecureWorks prepared forensic analysis environment:
• Launched forensic EC2 instances within Dell SecureWorks’ VPC
• Created S3 bucket for event data storage and transfer of forensic artifacts
• Using IAM, Customer provided appropriate access for Dell
SecureWorks to:
• Acquire snapshots of the affected Customer’s EC2 instances
• Transfer snapshots to Dell SecureWorks’ S3 bucket for forensic analysis
• Receive access to Customer’s CloudTrail logs for forensic analysis
• Using rapidly-deployed forensic toolsets, Dell SecureWorks
conducted forensic exam of:
• File systems of the Customer’s Internet-facing EC2 instances
• Customer’s AMIs
• Customer’s AWS CloudTrail logs
• Dell SecureWorks provided comprehensive analysis of the incident
and affected AWS resources

IR Case Example - Takeaways
• AWS enables shorter response times for security events vs. on-premise
• Time between engagement kickoff and commencing analysis was drastically reduced
• Security event data can be rapidly acquired, staged, and analyzed all within AWS
• Appropriate access can be quickly granted to security event responders via AWS IAM
• The ability to collaborate on configuration activities directly within AWS minimized time
taken for troubleshooting
• Creating effective environments for sharing incident response resources and
data within AWS is straight-forward
• Versus traditional IR, cost savings are also realized via IR within AWS
through reduction of the investigation timeline (minimized time to data
acquisition, resource setup, and initial analysis)

Our IR Prep “Wish List” for AWS Customers
• Take snapshots of all affected or suspected instances
• Collect network and instance metadata
• Create a restricted-access VPC, Security Group, and/or
separate AWS account
• Be ready to create temporary users / credentials via IAM
• Enable and centralize CloudTrail and CloudWatch logs
• Create a dedicated S3 bucket for sharing incident
response artifacts

How to Contact Dell SecureWorks
• Incident Response Hotline (24x7x365)
1-877-884-1110
• Website
http://www.secureworks.com/incident-response/
• Booth: #446 (next to Docker)
Flag me down and/or visit our booth to learn more about Dell
SecureWorks’ experience and capabilities and how we are partnered
with AWS to provide Incident Response for AWS customers!

AWS Security Best Practices whitepaper
• Help for designing security infrastructure and
configuration of your AWS environment
• High-level guidance for:
• Managing accounts, users, groups, and roles
• Managing OS-level access to instances
• Securing your data, OS, apps, and infrastructure
• Managing security monitoring, auditing, alerting, and incident
response
https://media.amazonwebservices.com/AWS_Security_Best_Practices.pdf

External resources – Reading, training
• SANS Reading Room, Incident Response
http://www.sans.org/reading-room/whitepapers/incident
• FIRST
http://www.first.org/resources/guides
• CERT, Incident Management
http://www.cert.org/incident-management/publications/

External resources – IR tools, frameworks
• Mozilla Investigator (MIG)
http://mig.mozilla.org/
• Netflix Fully Integrated Defense Operations (FIDO)
http://techblog.netflix.com/2015/05/introducing-fido-
automated-security.html

Other relevant talks this week
• SEC403 - Timely Security Alerts and Analytics: Diving
into AWS CloudTrail Events by Using Apache Spark on
Amazon EMR, Will Kruse
• SEC303 – Architecting for End-to-End Security in the
Enterprise, Hart Rossman and Bill Shinn
• If you miss(ed) any of them live, they will be on
YouTube, just like this talk.
• Don’t forget last year’s “Intrusion Detection in the Cloud”
and “Incident Response in the Cloud” that are already on
YouTube!

AWS Support for security concerns
• AWS Support is the one-stop shop for AWS customers,
for any concerns, including security related.
• If AWS Support cannot immediately address your
concerns, they will escalate internally to the appropriate
technical team, AWS Security included.
https://aws.amazon.com/support

AWS security resources
• AWS Security Blog
http://blogs.aws.amazon.com/security/
• AWS Security Center
https://aws.amazon.com/security
• Contact the AWS security team
aws-security@amazon.com

Summary
• Security agility with AWS
• Threat vs. policy-driven concerns, enumerate, create
signatures, detection mechanisms
• Automate IR where you can
• Two ways to get more practice: you only get to choose
one
• We (AWS and our technology partners) are here to help!

Remember to complete
your evaluations!

Wrangling Security Events in the Cloud

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (9)

Ähnlich wie Wrangling Security Events in the Cloud

Ähnlich wie Wrangling Security Events in the Cloud (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Wrangling Security Events in the Cloud