You know that adopting Continuous Delivery and DevOps is key to a high-performance company. You’ve read the books and are ready to build microservices in the cloud. Great! Let’s go back to the principles and see how to apply them in a cloud native environment. What used to about shipping code to static servers, is now about quickly creating decoupled pipelines for new services that are readily wired up into the platform and everything is driven by code.
This talk will give concrete guidance in a world where autonomous teams continuously deploy many independent services and containers into an infrastructure that is dynamically created via APIs. Learn how to establish a culture that fosters fast local decisions and is driven by fast feedback. Remove friction by removing the staging environment and still release with confidence.
Listen to stories from the trenches on true DevOps on AWS with “You build it, you run it” teams from AutoScout24, the largest online car marketplace Europe-wide.
https://devopsconference.de/continuous-delivery/cloud-native-continuous-delivery/
3. Cloud native Continuous Delivery
• Bring changes into production
• Fast
• Reliable
• Repeatable
• Traceable
• In order to
• Get fast feedback
• Lower risk
18. Science and Continuous Delivery
Forsgren, Nicole and Humble, Jez, The Role of Continuous Delivery in IT and Organizational Performance (October 27, 2015). Forsgren, N., J. Humble
(2016). "The Role of Continuous Delivery in IT and Organizational Performance." In the Proceedings of the Western Decision Sciences Institute (WDSI)
2016, Las Vegas, NV. . Available at SSRN: http://ssrn.com/abstract=2681909 or http://dx.doi.org/10.2139/ssrn.2681909
27. • Travis CI, CircleCI, GitLab CI, …
• Deploy agent needs access to production
• Use separate tools for CI and CD
• AWS Code*
• Definitely for OSS
• Not an option for AS24
Managed deployment pipelines?
28. • CD infrastructure should be the first task in a new project
• CD should not become a snowflake itself
• For disaster recovery you will need your CD infrastructure
• Aim for “CD as a service”
Automate CD infrastructure
29. • Containerized
• Isolated builds – bring your own agent
• Elastic agents
• Container as artifact
• Pipeline as code
• Declarative in service repository
• Fast and simple bootstrapping of new pipelines
• Avoid single, shared CI instance
New CI practices
30. • Everything that used to be good practices
• No CI theatre
• Embrace deployment pipelines
• No smarts in the CI tool
Old CI practices – Recap
35. Cloud native deployment pipeline
Application code and
infrastructure
specification in one
repository per service.
36. Cloud native deployment pipeline
Application code and
infrastructure
specification in one
repository per service.
CI
Deployment package
and infrastructure
declaration as
artifact.
37. Cloud native deployment pipeline
Application code and
infrastructure
specification in one
repository per service.
CI
Deployment package
and infrastructure
declaration as
artifact.
CD
1. Create or update
service infrastructure.
38. Cloud native deployment pipeline
Application code and
infrastructure
specification in one
repository per service.
CI
Deployment package
and infrastructure
declaration as
artifact.
CD
1. Create or update
service infrastructure.
2. New instances pull
down package and
start application.
40. • Follow microservices boundaries
• At least one stack per microservice
Decompose into Micro-Infrastructures
41. • Macro stack(s)
• Outputs parameters exported
• Keep it small, only things that don’t
change often
• No services
Macro-Infrastructure
• Network
• Security
• Bastion Host
42. • Services share macro stack
• Service stacks import parameters
• Service teams own service stack
• All services are in service stacks
Shared stack and service stacks
43. • Services have dependencies
• CD infrastructure
• Macro stack
• Base images (AMI, container)
• …
• But avoid explicit pipeline dependencies
• Try to reference pinned dependencies
Isolate deployment pipelines
55. Lifecycle of immutable servers/containers
Created
V3
Healthcheck
ok
V3
Traffic from
load balancer
V3
56. Lifecycle of immutable servers/containers
V3
Created
V3
Healthcheck
ok
V3
Traffic from
load balancer
V3
Connections
drained
57. Lifecycle of immutable servers/containers
V3
Created
V3
Healthcheck
ok
V3
Traffic from
load balancer
V3
Terminated
V3
Connections
drained
58. Lifecycle of immutable servers/containers
V3
Created
V3
Healthcheck
ok
V3
Traffic from
load balancer
V3
Terminated
V3
Connections
drained
• No need for configuration management tools: Chef, Puppet, Ansible
• Patches/Security? Alert on base image age
• Simpler with stateless services
82. • New service to be launched
Dark launches
Old New
Client
83. • Fork real traffic to new service and
discard response
• Monitor new service under real load
• Compare responses
• Fork on server or client side
Dark launches
Old New
Client
Bring changes into production, or into the hands of users, safely and quickly in a sustainable way.
Speed -> Fast cycle time, Fast creation of new pipelines
Being fast with a small team is one thing, but we want to not slow down, when the organization is larger.
No committees, no waiting
Autonomous teams -> Pipeline owned by teams
Independent deployable -> Many pipelines
They became silos, separated by a virtual wall.
Tickets, Waiting, Blaming
If you still have those silos: Get rid of them!
True DevOps culture
Jezz Humble: What We Learned from Three Years Sciencing the Crap out of Devops
Now lets walk through the pipeline steps and see what happens when we put on the cloud native hat.
Every engineer knows GitHub
SSO: Depending on plan, additional tooling required.
Backup: Scheduled clone/pull of all repos.
Secure: Story of the committed GitHub key.
Orchestration in one tool
Better overview, Simpler
CI: All the known tools
CD: For example Spinnaker or Bosh, often platform specific
We didn’t want to handover the golden key to a managed CI tool.
If you cannot use a managed CD service…
Isolated builds: Bring your own build environment, multi tenant services, like Travis, already work like this
Also need to solve the problem of dependency caches. For example local Nexus.
Elastic agents: Cost efficient, quickly spin up agents on demand. No job queue.
Container as artifact: DinD or DooD
Pipeline as code: Service templates includes pipeline declaration
Single CI: Autonomous teams, CI patterns and guidelines are shared, not the infrastructure.
https://www.thoughtworks.com/radar/techniques/pipelines-as-code
CI theatre: Infrequent commit, long red, feature branches
Embrace deployment pipelines: Beyond CI, eg. Jenkins, TC
Smarts in the tool: Use the tool for coordination, smarts are in your automation,
make like build scripts
should run from shell
https://www.thoughtworks.com/radar/techniques/ci-theatre
Now we are done with CI, let’s talk about infrastructure.
https://www.thoughtworks.com/talks/infrastructure-design-patterns-xconf-eu-2017
No slowflakes
Attribution: CC0, https://pxhere.com/en/photo/813531
Which versions on staging?
Prod differs anyway: Load, data, patterns
Still we want to release with confidence: Be bold, but not stupid.
Mean time to detection
Who is using feature branches and does CI?
You are not doing CI, when you are branching.
Short lived branches < 1 day are ok.
Branch by abstraction
https://trunkbaseddevelopment.com/
https://launchdarkly.com/
Optional:
Toggle forwarding to downstream services
Toggle dependencies.
Many of those patterns are viable without going cloud native, but are feasible then.
Attribution: CC0, not attribution required, https://pixabay.com/en/ibiza-rock-sea-water-821719/
Serverless frees you from this deployment details
Let’s step back to where we actually deal with servers or containers.
Verify health <> canary
Attribution heartbeat CC0, no attribution required
http://maxpixel.freegreatpicture.com/Heartbeat-Heart-Medical-Ecg-Electrocardiogram-2270728
Instead of load balancer also batch or queue processing possible
Or added to cluster.
State should be pushed to client or down to managed state services form cloud provider.
But this pattern also works with modern distributed data stores.
Create
Healthcheck
Traffic
Drain
Terminate
Repeat for all nodes
You bring your canary to the coal mine and observe its behavior.
Feature toggle canary: Staged rollout.
Attribution:
Canary: Public domain, https://commons.wikimedia.org/wiki/File:Carduelis_chloris_m.jpg
Coal Mine: CC0, no attribution required, https://pixabay.com/en/coal-black-mineral-underground-1626368/
Trade offs for client and server side forking:
Server: Risk, timeout protection, additional load
Client: Disclosure, bandwidth and load on client, less control
Attribution: CC0, no attribution required, https://www.pexels.com/photo/close-up-of-christmas-decorations-257855/
Build isolation: Decoupling, no coordination for a new Java SDK.
Independent pipelines: Decoupling
Elasticity: Do not waste resource or make teams wait for an agent
Everything as code: Pipeline definition, deployment scripts, infrastructure
Attribution: CC0, https://pixabay.com/en/thumbs-up-thumb-hand-positive-1006176/
Improve over time, what is included in a newly bootstrapped service: Logging, Monitoring, Alerting, Dashboards
Attribution: CC0, http://maxpixel.freegreatpicture.com/Measurement-Stopwatch-Timer-Clock-Symbol-Icon-2624277
Autonomous, empowered teams. No silos
Do not deal with DC constraints and mutate existing servers.
Do not build an infrastructure monolith, decoupling
Attribution: Raised hands, CC0, no attribution required, https://pixabay.com/en/hands-hand-raised-hands-raised-220163/