DevSecOps is the premise that everyone in the software development lifecycle is responsible for security. DevSecOps aims to embed security in every part of the development process. In this *workshop*, participants explore taking a standard CI/CD pipeline and adding security stages to improve security posture. Learn how to use AWS CodeCommit and AWS CodePipeline to build and publish golden AMI images. Also, learn how to modify pipeline flow to add security test cases. You also have to opportunity to perform CVE analysis and code analysis using Amazon Inspector and perform observational container analysis using Amazon GuardDuty.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Nate Case & Dave Walker
Security Geeks
Workshop Guide: https://tinyurl.com/yakft2rq
DevSecOps on AWS
Introduction to DevSecOps

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Setup the initial Account
- Run the Cloudformation
(This can take a bit, so while we wait..)
• Introduction to DevSecOps
• Introduction to Pipelines
Review of the Pipeline
- Correct the code! Or the rule?
- Test the Test environment
- Amazon Systems Manager
- To manual or not to manual?
- Finished!

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introduction to DevSecOps

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is DevOps?
Cultural
Philosophy
Practices Tools

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is DevOps?
Break down cultural barriers
Work as one team
Support business and IT agility
Collaborate and communicate
Treat infrastructure as code
Automate
Test, measure, and monitor
Culture
Process

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why do organizations adopt DevOps?
Faster time to value
Agility
Quality
Speed

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Competing Forces
Business
Development Operations
Build it faster Keep it stable
Security
Make it
secure

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is DevSecOps?
DevSecOps is the combination of cultural
philosophies, practices, and tools that exploits
the advances made in IT automation to achieve
a state of production immutability, frequent
delivery of business value, and automated
enforcement of security policy.
DevSecOps is achieved by integrating and
automating the enforcement of preventive,
detective, and responsive security controls into
the pipeline.
Security
OperationsDevelopment

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tenets of DevSecOps
1. Test security as early as possible to accelerate
feedback.
2. Prioritize preventive security controls to stop bad
things from happening.
3. When deploying a detective security control, ensure it
has a complementary responsive security control to
do something about it.
4. Automate, automate, automate.

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Three Major Components to DevSecOps
1. Security OF the pipeline
2. Security IN the pipeline
3. Enforcement of the pipeline

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security OF the pipeline
Use the Core 5 Security Epics from the CAF
• Identity and Access Management
• Detective Controls
• Infrastructure Security
• Data Protection
• Incident Response
Security OF the Pipeline module will provide guidance

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security IN the pipeline
Static analysis
• Infrastructure-as-code
• Security-as-code
Dynamic analysis
• Unit tests
• Integration tests
• System tests

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enforce the pipeline
Establish environments (e.g. Sandbox, Dev, Test, Prod)
Humans should have increasingly fewer rights as you
progress through environments
Only the pipeline should be able to “make changes” to
Prod

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is a Pipeline?
Build automation
Continuous Integration
• Deployment automation
Test automation
Service Orchestration

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CI vs CD
Continuous Integration
Techniques and tools to
implement the continuous process
of applying quality control; in
general, small pieces of effort,
applied frequently, to improve
the quality of software, and to
reduce the time taken to deliver
it.
Continuous Deployment
Techniques and tools to improve
the process of software delivery,
resulting in the ability to rapidly,
reliably, and repeatedly push out
enhancements and bug fixes to
customers at low risk and with
minimal manual overhead.

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Promotion Process in Continuous Deployment

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pipelines as Workloads

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pipeline as a workload
Securing the application starts with securing the pipeline
The CI/CD pipeline is a workload
Its purpose is to integrate and deliver other workloads
It has users, supporting infrastructure, application, and
data components, etc.
Those components are typically managed as code …

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Infrastructure as Code is a practice
where by traditional infrastructure
management techniques are
supplemented and often replaced by
using code based tools and software
development techniques.

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Resources
Operating System and
Host Configuration
Application
Configuration
Amazon Virtual Private
Cloud (VPC)
Amazon Elastic
Compute Cloud (EC2)
AWS Identity and Access
Management (IAM)
Amazon Relational
Database Service (RDS)
Amazon Simple Storage
Service (S3)
AWS CodePipeline
…
Windows Registry
Linux Networking
OpenSSH
LDAP
Centralized logging
System Metrics
Deployment agents
Host monitoring
…
Application dependencies
Application configuration
Service registration
Management scripts
Database credentials
…
AWS CloudFormation
AWS SSM
AWS CodeDeploy

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
allOfThis == $Code
https://secure.flickr.com/photos/wscullin/3770015991

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enforcing Least Privilege between pipelines
Pipeline can perform a specific job
E.g. Jenkins/Spinnaker/CodePipeline is a pipeline factory
Pipelines can be limited to blast radius-based functions
• Pipeline Factory
• AMI Factory
• Artifact Factory

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Decompose security requirements for CI/CD
Remember that CI/CD is not a “thing” unto itself
CI/CD is a collection of microservices
In the same way we use the cloud to protect the cloud,
use Agile to deploy Agile
Start with Security epics and create user stories for
security features to include in the pipeline
Make the pipeline another reference architecture
Don’t forget that the pipeline is a workload!

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security Epics
Identity & Access Management
Logging & Monitoring
Infrastructure Security
Data Protection
Incident Response
Resilience
Compliance Validation
Secure CI/CD (DevSecOps)
Configuration & Vulnerability Analysis
Big Data Analytics
Bearded security guy
asks, “Who can list the
10 Security Epics of the
CAF in order?”

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Top 10 11 IAM Best Practices
0. Users
1. Permissions
2. Groups
3. Conditions
4. Auditing
5. Passwords
6. Rotation
7. MFA
8. Sharing
9. Roles
10.Root
Oops, looks
like a 0-based
code error
J

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Some IAM risks for pipelines
Anyone can run build jobs
Consistent user management across build servers
Pipeline role is too permissive
Slave node adverse effects on Masters

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Exercise: IAM for pipelines wrap-up
Could you write a user story for the DevOps team
managing the pipeline to implement?
• If not, what is missing?
What are the acceptance criteria for your user story?
How would you validate your user story?

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Top Detective Controls Best Practices
1. Coordinate time sources (EC2 Time Sync!)
2. Capture all logs
3. Determine auditable events and audit record content
4. Protect the confidentiality and integrity of audit logs
(WORM)
5. Detect audit processing failures
6. Determine thresholds for warnings and alerts
7. Respond to warnings and alerts

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Exercise: Detective Controls
• What produces logs?
• How are logs produced?
• Where do logs go?
• How do I protect my logs?
• What are the items of interest in my logs?
• At what threshold are those items interesting?
• What should I do when thresholds are exceeded?

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Detective Controls for pipelines
Who logged in?
What code was committed and by whom?
What jobs did they run?
Did the jobs succeed/fail?
Was static/dynamic analysis enforced?
What were the results of the static/dynamic analysis?

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Detective Controls for pipelines wrap-up
There are multiple consumers of logs produced by the
pipeline.
Fast feedback to the log consumers is critical.
Results of static/dynamic tests are as important as any
other audit trail.

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Top Infrastructure Security Best Practices
Provision AWS accounts in accordance with team organization
Separate environments by their access levels
Create good architectures for pipelines (e.g. dedicated, shared,
team)
Whitelist the environments, API’s, and services the pipeline is
allowed to interact with
Use EC2 roles combined with IAM policies (and CloudTrail!) to
your advantage
Limit the blast radius!

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hopefully you've seen this already…
Developer
Sandbox
Dev Pre-Prod
Team/Group accounts
Security
Core accounts
AWS Organizations
Shared
Services
Network
Log
Archive
Prod
Team
Shared
Services
Network Path
Developer accounts Data Centre
Orgs: Account management
Log Archive: Security logs
Security: Security tools, AWS Config rules
Shared services: Directory, limit monitoring
Network: Direct Connect
Dev Sandbox: Experiments, learning
Dev: Development
Pre-Prod: Staging
Prod: Production
Team SS: Team Shared Services, Data
Lake

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services
Security
Core accounts
AWS Organizations
Shared
Services
Network
Log
Archive
Network Path
Data Centre
Orgs: Account management
Log Archive: Security logs
Security: Security tools, AWS Config rules
Shared services: Directory, limit monitoring
Network: Direct Connect
Dev Sandbox: Experiments, learning
Dev: Development
Pre-Prod: Staging
Prod: Production
Team SS: Team Shared Services, Data
Lake

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Developers
Developer
Sandbox
Team/Group accounts
AWS Organizations
Network Path
Developer accounts Data Centre
Orgs: Account management
Log Archive: Security logs
Security: Security tools, AWS Config rules
Shared services: Directory, limit monitoring
Network: Direct Connect
Dev Sandbox: Experiments, learning
Dev: Development
Pre-Prod: Staging
Prod: Production
Team SS: Team Shared Services, Data
Lake

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Connections to Team Accounts
Developer
Sandbox
Team/Group accounts
AWS Organizations
Network Path
Developer accounts Data Centre
Orgs: Account management
Log Archive: Security logs
Security: Security tools, AWS Config rules
Shared services: Directory, limit monitoring
Network: Direct Connect
Dev Sandbox: Experiments, learning
Dev: Development
Pre-Prod: Staging
Prod: Production
Team SS: Team Shared Services, Data
Lake
Core accounts
Dev Pre-Prod
Team/Group accounts
Prod
Team
Shared
Services

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Team verses Production Accounts
Dev
Prod Green
Team/Group accounts
Prod Blue
Dev Pipeline account
Network Path
Data Centre
Orgs: Account management
Log Archive: Security logs
Security: Security tools, AWS Config rules
Shared services: Directory, limit monitoring
Network: Direct Connect
Dev Sandbox: Experiments, learning
Dev: Development
Pre-Prod: Staging
Prod: Production
Team SS: Team Shared Services, Data
Lake
Testing Account
Prod Pipeline account
Network
Prod/Group accounts

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Top Infrastructure Security Best Practices
Provision AWS accounts in accordance with team organization
Separate environments by their access levels
Create good architectures for pipelines (e.g. dedicated, shared,
team)
Whitelist the environments, API’s, and services the pipeline is
allowed to interact with
Use EC2 roles combined with IAM policies (and CloudTrail!) to
your advantage
Limit the blast radius!

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Infrastructure Security risks to pipelines
Who has access to underlying infrastructure resources?
How are pipelines patched and updated?
How is least privilege between pipelines enforced?
Are my pipelines deploying into approved AWS accounts?
Does the pipeline align with organizational responsibility?

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Infrastructure Security for pipelines wrap-up
The pipeline is a workload and needs to be treated with
the same rigor as other critical infrastructure.
Build a pipeline factory to build pipelines from known
good configurations.
Deploy workloads into known good environments.

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use IAM roles for Amazon EC2 instances
Benefits
Easy to manage access keys
on EC2 instances
Automatic key rotation
Assign least privilege to the
application
AWS SDKs fully integrated
AWS CLI fully integrated
How to get started
Create an IAM role
Assign permissions to role
Launch instances w / role
If not using SDKs, sign all
requests to AWS services
with the role’s temporary
credentials

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Top Data Protection Best Practices
Control access and permissions to the code repository
Trigger builds automatically (time-based or event-based)
Use tokenization or dummy data in non-production
environments
Categorize data and enforce restrictions through pipeline
• For example, pipeline is configured to build Dev environment
is not allowed to pull Production data from repo

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
No more humans in production.
Dev
Prod Green
Team/Group accounts
Prod Blue
Dev Pipeline account
Network Path
Data Centre
Orgs: Account management
Log Archive: Security logs
Security: Security tools, AWS Config rules
Shared services: Directory, limit monitoring
Network: Direct Connect
Dev Sandbox: Experiments, learning
Dev: Development
Pre-Prod: Staging
Prod: Production
Team SS: Team Shared Services, Data
Lake
Testing Account
Prod Pipeline account
Network
Prod/Group accounts

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Protection risks for pipelines
Who can change/commit code?
How is production data prevented from being introduced
into non-prod environments?
How is artifact integrity maintained?

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
No more humans in production.
Dev
Prod Green
Team/Group accounts
Prod Blue
Dev Pipeline account
Network Path
Data Centre
Orgs: Account management
Log Archive: Security logs
Security: Security tools, AWS Config rules
Shared services: Directory, limit monitoring
Network: Direct Connect
Dev Sandbox: Experiments, learning
Dev: Development
Pre-Prod: Staging
Prod: Production
Team SS: Team Shared Services, Data
Lake
Testing Account
Prod Pipeline account
Network
Prod/Group accounts

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Protection for pipelines wrap-up
Control access and permissions to source repository:
artifacts are critical data for your pipeline.
Build pipelines that are environment-aware (e.g. prod vs.
non-prod).
Build artifact handlers to validate integrity across
pipelines and environments.

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DevSecOps Benefits
• Confidence that workloads and changes are validated
against corporate security policies.
• Consistency and repeatability of security validation.
• Match the business’ pace of innovation.
• Security at scale!

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Helpful Links
https://aws.amazon.com/devops/
https://aws.amazon.com/devops/what-is-devops/
https://stelligent.com/2015/01/20/deployment-
pipeline-aws/
https://aws.amazon.com/getting-started/projects/set-
up-ci-cd-pipeline/
https://aws.amazon.com/certification/certified-devops-
engineer-professional/
https://github.com/awslabs/git-secrets

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions?

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Core accounts
Core Accounts
AWS Organizations Master
Network Path
Data Center
Foundational
Building Blocks
Once per organization
Have their own development
life cycle (dev/qa/prod)

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Log archive account
Core Accounts
AWS Organizations Master
Log
Archive
Network Path
Data Center
Versioned Amazon S3 bucket
Restricted
MFA delete
CloudTrail logs
Security logs
Single source of truth
Alarm on user login
Limited access

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security account
Core Accounts
AWS Organizations Master
Log
Archive
Network Path
Data Center
Optional data center
connectivity
Security tools and audit
GuardDuty Master
Cross-account read/write
Automated Tooling
Limited access
Security

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Shared services account
Security
Core Accounts
AWS Organizations Master
Log
Archive
Network Path
Data Center
Connected to DC
DNS
LDAP
Shared Services VPC
Deployment tools
Golden AMI
Pipeline
Scanning infrastructure
Inactive instances
Improper tags
Snapshot lifecycle
Monitoring
Limited access
Shared
Services

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Network account
Security
Core Accounts
AWS Organizations Master
Shared
Services
Log
Archive
Network Path
Data Center
Managed by
network team
Networking services
AWS Direct
Connect
Limited access
Network

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Developer sandbox
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Network Path
No connection to
DC
Innovation space
Fixed spending limit
Autonomous
Experimentation
Developer
Sandbox
Developer Accounts

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Team/group accounts
Developer
Sandbox
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Network Path
Developer Accounts Data Center
Based on level of needed
isolation
Match your development
lifecycle
Think Small
Team/Group Accounts

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dev
Developer
Sandbox
Team/Group Accounts
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Network Path
Developer Accounts Data Center
Develop and iterate
quickly
Collaboration space
Stage of SDLCDev

58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pre-production
Developer
Sandbox
Dev
Team/Group Accounts
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Network Path
Developer Accounts Data Center
Connected to DC
Production-like
Staging
Testing
Automated Deployment
Pre-Prod

59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Production
Developer
Sandbox
Dev Pre-Prod
Team/Group Accounts
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Network Path
Developer Accounts Data Center
Connected to DC
Production applications
Promoted from Pre-Prod
Limited access
Automated Deployments
Prod

60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Team shared services
Developer
Sandbox
Dev Pre-Prod
Team/Group Accounts
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Prod
Network Path
Developer Accounts Data Center
Grows organically
Shared to the team
Product-specific common
services
Data lake
Common tooling
Common services
Team
Shared
Services

61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Innovation pipeline
Developer
Accounts
Developer Accounts
PoC
Developer
Accounts
Developer Accounts
Dev
Pre-Prod
Team/Group Accounts
Prod
Shared
Services
PoC
New initiatives
Experimentation
Innovation

62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multi-account approach
Developer
Sandbox
Dev Pre-Prod
Team/Group Accounts
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Prod
Team
Shared
Services
Network Path
Developer Accounts Data Center
Orgs: Account management
Log Archive: Security logs
Security: Security tools, AWS Config rules
Shared services: Directory, limit
monitoring
Network: Direct Connect
Dev Sandbox: Experiments, Learning
Dev: Development
Pre-Prod: Staging
Prod: Production
Team SS: Team Shared Services, Data
Lake

63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Team: Billing tools
Developer
Sandbox
Dev Pre-Prod
Billing Tools Team Accounts
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Prod
Network Path
Developer Accounts Data Center
Reduces access to
Organizations account
Billing reports
Usage metrics and
reporting
Usage optimizations and
RI management

64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Team: Internal audit
Developer
Sandbox
Dev Pre-Prod
Internal Audit Team Accounts
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Prod
Network Path
Developer Accounts Data Center
Regulatory compliance
Read-only access to
needed logs
Limited access
ENT315: Automate and
Audit Cloud Governance
and Compliance in your
Landing Zone

65. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Team: Amazing new product
Developer
Sandbox
Dev Pre-Prod
Amazing New Product Team Accounts
Security
Core Accounts
AWS Organizations Master
Shared
Services
Network
Log
Archive
Prod
Network Path
Developer Accounts Data Center
Match your development
lifecycle
Think Small