CON302_Building a CICD Pipeline for Containers on Amazon ECS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Building a CI/CD Pipeline for Containers on
Amazon ECS
J e r e m y C o w a n , A W S S o l u t i o n A r c h i t e c t
A j i t Z a d g a o n k a r , E x e c u t i v e D i r e c t o r o f O p s
N o v e m b e r 2 7 , 2 0 1 7

What to Expect from This Session
Review of continuous integration, delivery, and deployment
Examine pipelines stages
Overview of CI/CD reference architectures
Next steps and call to action
How Edmunds is using Amazon Elastic Container Service
(Amazon ECS) and CI/CD to accelerate innovation

What is CI/CD?
Continuous integration
• The practice of merging all developer working copies to a
shared mainline multiple times a day
Continuous delivery
• An approach in which teams produce software in short
cycles, ensuring that the software can be reliably released
at any time

Deployment versus Delivery
Continuous delivery
• Creates a deployable asset
• Often requires manual approval before changes pushed to
production
Continuous deployment
• Changes pushed to production automatically

What problems does CI/CD address?
Automates the build, test, and deployment phases
Accelerates the software release cycle
Reduces the cost and risk of delivering changes
Gives developers an opportunity to get fast feedback
Facilitates an iterative approach to software development

Why Use Containers for Continuous Delivery?
Containers allow you to deploy updates quickly
• Layered file system
• Shared OS kernel
• Packaged dependencies
Revision control through tagging
Predictable and reproducible across different environments

A Common Docker Workflow
Build
• Creates standardized packaged applications
Test
• Offers an easy way to smoke test your whole stack
• Runs consistently between environments
Deployment
• Abstracts how services work or are deployed
• Deploys easily via an orchestration service

Source Build Test Production
• Version Control
• Branching
• Code Review
• Compilation
• Unit Tests
• Static Analysis
• Packaging
• Integration Tests
• Load Tests
• Security Tests
• Acceptance
Tests
• Deployment
• Monitoring
• Measuring
• Validation

Source Stage
Tools
• AWS CodeCommit
• Git compatible source code repo
• Integrates with other AWS services, e.g. IAM, KMS
• Audit changes
• Pull requests (released 11/20) and CWE
• GitHub Enterprise
• Bitbucket, etc.

Source Best Practices
• Do frequent check-ins multiple times a day
• Store source code in same repository as
dockerfile
• Store compiled assets in separate versioned
repository, e.g. Amazon S3 or third-party solution
• Copy assets into container at build-time
• Keep code for building container infrastructure in
separate repository

Build Stage
Tools
• AWS CodeBuild
• A managed build service that compiles source code, runs
tests, and creates assets that are ready to deploy
• VPC access (released 11/21)
• Jenkins
• Others

Build Specification—Phases
Phase Description Examples
install Installation of packages into the
environment
Install testing frameworks
e.g. RSpec, Mocha
pre_build Commands to run before the build
such as login steps or installation of
dependencies
Log in to Amazon ECR.
run Ruby bundler or npm
build Sequence to run the build such as
compilation and/or running tests
Run go build, sbt, Mocha,
RSpec
post_build Commands to run after a build on
success or failure
Build a JAR via Maven or
push a Docker image to
Amazon ECR

Build Specification—Docker
version: 0.1
phases:
pre_build:
commands:
- $(aws ecr get-login --no-include-email)
build:
commands:
- docker build –t ʺ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}ʺ .
post_build:
commands:
- docker push ʺ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}ʺ

Docker Build Best Practices
• Tag output artifacts to source control revisions
(e.g. git SHA, semantic version)
• Avoid using a “latest” or “production” tag
• Optimize for build speed
• Co-locate build process with its artifact repository
• Scan images for vulnerabilities before pushing to
a repository
• Rebuild images when base image changes

Testing Phase
Verify updates across different dimensions
• UI, load, integration, API
Simplified because app dependencies are packaged along with
the application
Validate updates and uncover bugs

Testing In the Container
Build the container and invoke the test suite inside the container
Override cmds in dockerfile from the command line docker post
build
docker run –e ENVIRONMENT=testing -it my_app:v1 ~/harness/test.sh
Capture output to a file to see if it has resulted in success

Detached Mode Testing
Deploy to a test environment and run load and other tests
outside the container
Testing tools
• Selenium, Runscope (APIs), JMeter, etc.
Handling environment variables

Production/Deployment Phase
Repeatable process
Configuration captured in dockerfile
Orchestration services: ECS, Kubernetes, Mesos
Deployment types:
• Rolling
• Blue/green
• Canary

Amazon ECS—Task & Service
EC2 INSTANCES
LOAD
BALANCER
Internet
ECS
AGENT
TASK
Container
TASK
Container
ECS
AGENT
TASK
Container
TASK
Container
AGENT COMMUNICATION
SERVICE
Amazon
ECS
API
CLUSTER MANAGEMENT
ENGINE
KEY/VALUE STORE
ECS
AGENT
TASK
Container
TASK
Container
LOAD
BALANCER
ECS Service

Ref Architecture CI/CD ECS
AWS
CodeCommit
AWS CodeBuild
Amazon ECR
AWS CodePipeline
Amazon ECS
Source
Repository
Build
Deploy

Integration with CodePipeline
AWS CloudFormation Action
• Call CloudFormation script from CodePipeline that updates
the task and service definitions
ECS Action for CodePipeline (coming soon)
• Calls the ECS update service API to trigger a deployment
• Eliminates the need to learn CloudFormation

Controlling ECS Deployments
Minimum and maximum healthy instances
• Controls how container instances are replaced
• Connections are drained before the task is stopped
Update process
• Update task definition with new container version
• Update service definition with new task definition
• Automate using API, CFN, ECS Action, or the CLI

Deployment—Rolling
Availability Zone Availability Zone
Scenario
Service’s task definition is
updated to a new revision with
parameters:
Desired Count = 2
Minimum Healthy Percent = 50%
Maximum Percent = 100%
These settings constrain the
service to not exceed its desired
size but allow it to halve the
number of tasks during
deployment
EXISTING EXISTING

First, an existing task is
stopped which brings the
healthy percentage of the
service to 50% and makes
room on the cluster for new
tasks
EXISTING
Desired Count = 2

A task using the new task
definition is started,
bringing the service back to
100% healthy
EXISTING
Desired Count = 2
NEW

After the new task is
verified to be healthy by the
Elastic Load Balancer health
check, the next existing task
with the older task
definition is drained and
stopped
Desired Count = 2
NEW

The second new task is
started on the cluster.
bringing the service back to
100% healthy
NEW NEW
Desired Count = 2

Blue/Green Deployments
Involves deploying new version of container/task alongside the
old version
Only one version serves production traffic
Allows for fast rollback
Can test new version without impacting production
• Example: new version could be listening on a different port
or only accessible by internal resources

Deployment – Blue/Green – Target Group Swap
Availability Zone
EXISTING EXISTING
Scenario
Two services are defined each
with their own target group
registered in the same
Application Load Balancer
using host-based routing
Deployment is completed by
swapping the listener rules
between the two target
groups
Availability Zone

Availability Zone
EXISTING EXISTING
The second service is deployed
with a new target group and
registered to the same
Using host-based routing,
requests to www.example.com
are directed to our blue service
while requests to
next.example.com are directed
to our green service
NEW NEW
Availability Zone

Availability Zone
After automated or manual
testing, the deployment can
be completed by swapping
the listener rules on the
and sending traffic to the
green service
NEW NEW
Availability Zone
EXISTING EXISTING

Availability Zone
The previous service and
its target group can then
be destroyed
NEW NEW
Availability Zone

Canary-Style Deployments
Diverts a percentage of production traffic to the new version of a
container/task
Enables A/B feature testing
Routing could be random or based on user profile
Based on results, canary version could replace production version
or stopped

Ref Architecture Blue/Green Canary

Amazon ECS Continuous Delivery Partners

Call to Action
Clone the CI/CD reference architectures
Provide feedback by filing issues or submitting pull requests
https://github.com/awslabs/ecs-refarch-continuous-deployment
https://github.com/awslabs/ecs-canary-blue-green-deployment
https://github.com/awslabs/ecs-blue-green-deployment
Join the Slack workspace, amazon-ecs.slack.com

How Edmunds Uses ECS

We love helping
people find their
perfect car

Agenda
Who are we? Edmunds introduction
Edmunds migration into AWS
Containers, architecture at Edmunds
Continuous Integration and Continuous Delivery at Edmunds

Interesting Numbers
200M+ monthly page views
Over 13K+ dealer partners
Over 18K+ franchise
Over 5M active inventory on website
Quarter billion unique visitors visit edmunds.com every year
59% car shoppers are influenced by Edmunds

Journey into Clouds
2011
Zen
2012 2013 2014 2016
Citrix CloudStack
AWS
100%
AWS
Site
100%
AWS
ECS
Docker
2017
Ops
Works

Why Clouds
Business growth
Need for speed
Desire for rapid experimentation
Dynamic infrastructure changes
Ongoing high cost of infrastructure

So, why containers?
• Immutable, consistent artifact bundle across benches
• Faster deployments
• Better HA, less upkeep in production
• ~30% lower operating cost

Container Management & Orchestration
• Transitioned our services from Amazon EC2 to Docker and ECS
• Amazon ECS for managing Docker deployments
• Amazon ECR for our Docker repositories
• CloudFormation for provisioning ECS nodes and E/ALBs
• AWS OpsWorks for configuring ECS nodes {logging, SSM}
• With a few exceptions, our entire stack is powered by Docker and ECS

Why Amazon ECS?
• Native integration with other AWS services (ELB, CloudWatch, EC2, etc.)
• Automatic scaling of services deployed in ECS
• Using ELB metrics, CPU, memory to scale
• AZ and resource aware container placement
• Task placement strategies
• Faster scaling
• Better utilization of our AWS resources
• Zero downtime deployments
• Easier AMI upgrades

Need for CI/CD
• Faster feedback cycle
• Lower risk
• Exposes bottlenecks and inefficiencies
• Many more

Platform
App
Tier
Service
Tier
Persistence
Tier
S3 MongoDB
Elasticsearch
{ Vehicle }
{ Inventory }
{ Dealer }
Commit
... in Sandbox
... in Pre-production
... in Production
Write
Git
Better Team code
collaboration
Control over Artifact
progression
Canary Deployments
Safe (isolated),
Easy Feature Testing
Deploy Manage
View into Real Users,
EVERY Transactions
CI/CD at Edmunds

Edmunds Site
• Multi-AZ and multi-region deployments
• Nginx for traffic routing, Canary, and A/B testing
• Nginx hundreds of rules, regex, custom header
• Multi CDN for caching
• Latency or availability based routing
Multi-CDN
(InstartLogic, Fastly, AWS CloudFront)

ECS—Infrastructure Architecture
CloudFormation provisions ECS instances and ELBs
ECS instances configured via OpsWorks
•Easier to manage host level config, e.g. Docker agents, ECS, logging, APM config, Amazon
Inspector, etc.
Internal tooling to manage creation of ECS services, task def and Auto Scaling config
•Standardized templates—sets up service with required properties and defined container
standards

Container Standardization—Do It !
ECS Host
|-------------------|
Reserved Memory
(host)
|-------------------|
Reserved Memory
(container)
|---------------------------------------------------------
|Container Memory Allocation
|--------------------------------|
Heap Allocation
Standard Container Sizing - Container CPU Units + Container Memory Units
Standard EC2 Instance Type for ECS Hosts

Configuration by Default
All applications running on ECS platform are environment-aware
Internal library automatically points application to correct VIP
• e.g. inventory-elastic-prod-x.route.edmunds.com

Auto Scaling
• ECS service Auto Scaling
• Request-based (ALB metrics - #RequestCount*)
• CPU and memory-based
• Schedule based
• ECS (EC2) instance Auto Scaling
• Scale based on reserved capacity
• Downscale and remove instances with no
containers

EC2 Downscaling Nuances
Avoid thrashing
• Check free nodes
• Enable termination protection on occupied EC2 nodes
• Enable downscaling

Custom Docker Build Framework
Edmunds EcoSystem Framework
- Templated dockerfiles
- Templates are extensible
- Easy to manage global configs
- e.g. APM config, DNS related
properties, etc.
- Routing rules
- Few types of artifacts = few dockerfiles
to manage
- Developer flexibility
- Modifications are easy
- Generates dockerfile at build time

• Application routing framework to
route requests to multiple
applications
• Nginx for routing requests
• Developers define request
paths/patterns for their
applications
• Rules are sorted
Edmunds Traffic Routing (aka Service Registry)

Ability to validate and/or release code changes
with live traffic
• Unique services for canary and stable
• Nginx for traffic distribution between canary
and stable
• Real-time analytics and processing of Nginx
logs
• Real-time canary feedback loop
• Feedback-based auto traffic increment/
roll-back
Edmunds Canary

mainline branch
(project repo. master/origin)
< branch >
feature branch
(project repo. branch) azadgaonkar-uct-
3221
myapp
< clone >
local feature branch
(local repo.)
- developer -
- acting technical lead
-
< commit >
< rebase >
< push > [ merge request ]+
< merge >
Continuous Integ.
(Jenkins builds)
Continuous Delivery
(Deployment pipeline)
v1.
3
Dev QA Prod
< build >
< deploy >
[ resolve ]
v1.2v1.1v1.0
Sandbox Deployment
(Docker images)
azadgaonkar-
myapp.sand.edmundsd
ev.com
< deploy > < deploy >
- sandbox -
Commit Merge request workflow example
Prod
C C
Tests Tests Tests
Canary Feedback
loop

Our CI/CD Tool Set
• Git
• Jenkins, CodeBuild
• ECR
• OpsWorks
• Tools on ECS APIs
• A lot of ƛ and step functions
• Jenkins Pipeline, CodeDeploy
• Metric store (Wavefront), CloudWatch—events, rules, and alerts
• AWS Config
• Alerts (VictorOps)

End-to-End CI/CD Visual Representation
If Canary Version is an Improvement
● Time to First Byte is equal or less
● HTTP Error Rates are equal or
reduced
● Logged Exception Rates are equal or
reduced
If Canary Version is NOT an Improvement
● Time to First Byte is greater
● HTTP Error Rates, Logged Exception
Rates are increased
● Difference for ttfb (75th percentile)
(107.89%) - Canary: `1037.05 ms`,
Stable: `310.25 ms`
● QA canary version recovered at traffic
ratio *20%*
● Canary version reset to 0%

• ECS related metrics sent to
metrics datastore for trending,
analytics, and alerting
• AWS Config to monitor
differences between same ECS
config across different cluster
• AWS Config to ensure ECS
config follows container
standards and application
requirements (e.g. min heap)
Metrics and Monitoring—ML
Time Series – data store

Canary versus Stable Version Real-Time Dashboard

AI for Decision Making
• Gather data for everything—CPU, memory, #requests, test failures
• Build simplistic time series trending data
• Find, track, and remove anomalies
• Address anomalies, understand root causes
• Build simple ML models to remove white noise
• Let DL guide your deployment decisions—roll forward, roll back, autoscale,
etc.

The Mantra for CI/CD Success
• Invest in better engineering practices
• Respect failures, invest in well-written automated tests
• 360O Monitoring—monitor everything
• Measure, baseline and benchmark
• Use data science (AI/ DL/ML) to guide decisions
• Have a commitment from ALL stakeholders

In Closing
• Better developer experience
• Smoother feature and product rollout
• Rock solid HA ~99+% and a fully automated recovery
• No efforts are wasted on container management
• $$$ {33%}
• It simply works!

Surveys
Don’t forget to complete your survey for CON302 in the mobile app

Thank you!

CON302_Building a CICD Pipeline for Containers on Amazon ECS

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie CON302_Building a CICD Pipeline for Containers on Amazon ECS

Ähnlich wie CON302_Building a CICD Pipeline for Containers on Amazon ECS (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

CON302_Building a CICD Pipeline for Containers on Amazon ECS