Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Loft Your Web PlatformIntothe Clouds withImmutable Infrastructure
Steven Merrill
© 2016 Phase2
OVERVIEW
4© 2016 Phase2
TOPICS COVERED
Immutability in software development
Immutable Infrastructure
Building immutable artifacts
U...
© 2016 Phase2
MUTABILITY
6© 2016 Phase2
im·mu·ta·ble
adjective
unable to be changed
7© 2016 Phase2
IMMUTABILITY IN SOFTWARE DEVELOPMENT
First-class immutable variables in language runtimes
No need to synchr...
© 2016 Phase2
IMMUTABLE INFRASTRUCTURE
9© 2016 Phase2
IMMUTABLE INFRASTRUCTURE? WHAT?
Continuation of infrastructure as code
Core tenet: build immutable artifact...
10© 2016 Phase2
AN EXAMPLE
A client had autoscaling troubles
Provisioned from scratch from a plain Ubuntu AMI
Problems:
Dr...
11© 2016 Phase2
COUNTERPOINTS
Most servers may not have read-only disks
Container systems can allow read-only rootfs
Will ...
12© 2016 Phase2
WHY?
Immutable infra practices enable autoscaling
Rallying cry: have cattle, not pets
It can allow for eas...
© 2016 Phase2
FOUNDATIONS:
BUILD PROCESSES
14© 2016 Phase2
BUILD THEORY
A modern web app likely has several build steps
Pull dependencies down
Source compilation
CSS...
15© 2016 Phase2
VENDORING
Consider vendoring your dependencies
Great discussion about this in “Who Owns Your Availability?...
16© 2016 Phase2
BUILD TARGETS
Deployment repository
Tarball
Machine image
Amazon AMI / Google Private images
Rkt / Docker ...
JENKINS: FORCE FOR GOOD OR EVIL
18© 2016 Phase2
BUILD PROCESS EVOLUTION
Awful
Build in-place on each of n web heads
Better
Build once in workspace, rsync ...
19© 2016 Phase2
BUILDING IMAGES
Autoscaling-ready image builds have 2 components:
Tagged code releases
A combination of Li...
© 2016 Phase2
AUTOSCALING
21© 2016 Phase2
WHY AUTOSCALE?
Reduces stress
Automatic healing
Cost benefits
Performance benefits
Automatic scaling based...
© 2016 Phase2
AUTOSCALING DRUPAL
IN AMAZON WEB SERVICES
23© 2016 Phase2
DRUPAL SITE COMPONENTS
Big 3 components of a running Drupal site
Codebase on an app server
Relational data...
24© 2016 Phase2
AUTOSCALING GROUPS IN AWS
An autoscaling group is attached to one or more ELBs
ELB health checks and timeo...
25© 2016 Phase2
AWS RECIPE FOR DRUPAL
Jenkins instance building tarballs + AMIs
Autoscaling group with public Drupal insta...
26© 2016 Phase2
EXAMPLE AMI AUTOSCALING SETUP
Have one build process to create a tarball / build artifact
Use Packer to st...
27© 2016 Phase2
ELIDED PACKER EXAMPLE
{
"variables": {
"aws_access_key" : "xxxxxxxxxxxxxxxxxxxx",
"aws_secret_key" ...
28© 2016 Phase2
ELIDED PACKER EXAMPLE
"provisioners": [{
"type": "file",
"source": "/path/to/my/build.tgz",
"destinati...
29© 2016 Phase2
ELIDED PACKER EXAMPLE
"builders": [{
"type": "amazon-ebs",
"name": "east-ami",
"access_key": "{{u...
30© 2016 Phase2
ELIDED PACKER EXAMPLE
{
"type": "amazon-ebs",
"name": "west-ami",
"access_key": "{{user `aws_acc...
31© 2016 Phase2
S3 SHARP EDGES
S3 files with amazons3/s3fs.module use memory_limit
You may be tempted to try the s3fs daem...
32© 2016 Phase2
AUTOSCALING QUIRKS
Schema updates
Whitescreens on new/old as schema changes flip
Add double capacity, let ...
© 2016 Phase2
CLIENT EXAMPLE
34© 2016 Phase2
MLS DIGITAL - MLSSOCCER.COM AND CLUBS
21-site Drupal 7 platform
League and all club sites
Game day scaling...
35© 2016 Phase2
MOVING MLS TO AUTOSCALING
Old setup provisioned from a base Ubuntu with Salt
Timeout problems
Moved to Pac...
36© 2016 Phase2
SUCCESS
From 4 pets to 2-5 cattle
Major hosting cost reduction
Later Drupal optimizations lowered instance...
A TYPICAL MONTH
DRUPAL OPTIMIZATIONS
AFTER TUNING
© 2016 Phase2
CONTAINERS
41© 2016 Phase2
POTENTIAL DOCKER BENEFITS
Increased density
Ability to run heterogenous workloads
Native CPU and disk perf...
42© 2016 Phase2
DOCKER IN PRODUCTION
Docker on single hosts is easy
Docker in production requires a lot of coordination
Ov...
43© 2016 Phase2
DOCKER BUILDS
Build considerations
Dockerfile vs configuration management
Dynamic templating
Build tools
d...
44© 2016 Phase2
DOCKER
Docker daemon allows for tagging
Ideally, use tags from canonical repo and as branches
p2/site:v1.0...
45© 2016 Phase2
KUBERNETES
Open-source container scheduler from Google
“Manage a cluster of Linux containers as a single s...
46© 2016 Phase2
KUBERNETES
Requires you to bring overlay networking
Features
Service discovery, load balancing, cluster DN...
47© 2016 Phase2
KUBERNETES OBJECTS
Nodes
Labels
Pods
DaemonSets
Replication controllers
Services
Deployments
48© 2016 Phase2
KUBERNETES POD YAML
apiVersion: v1
kind: Pod
metadata:
name: lamp
app: lamp
spec:
containers:
- n...
49© 2016 Phase2
KUBERNETES RC YAML
apiVersion: v1
kind: ReplicationController
metadata:
name: lamp
spec:
replicas:...
50© 2016 Phase2
KUBERNETES SERVICE YAML
apiVersion: v1
kind: Service
metadata:
name: lamp
spec:
selector:
app: lam...
© 2016 Phase2
SUMMARY
52© 2016 Phase2
IMMUTABLE INFRASTRUCTURE
Build immutable artifacts for deployment
Limit configuration drift in your infras...
© 2016 Phase2
QUESTIONS?
SoHow Was It?- Tell Us What YouThink
Evaluate this session - https://events.drupal.org/neworleans2016/schedule
Thanks!
Loft Your Web Platform into the Clouds with Immutable Infrastructure
Nächste SlideShare
Wird geladen in …5
×

Loft Your Web Platform into the Clouds with Immutable Infrastructure

400 Aufrufe

Veröffentlicht am

Watch the full video here: https://vimeo.com/166817956
DrupalCon New Orleans - May 11, 2016
Steven Merrill, Phase2 Director of DevOps

Noticing that you automatically scaled up to double your normal footprint and back down after a wave of high traffic passed is a great feeling, especially if you only notice it on Monday morning when you look back at the weekend’s traffic in your monitoring system. Being able to use autoscaling systems to horizontally scale the web tier of your Drupal web platform is a great outcome, but getting to that point requires some planning and rigor in your build and automation processes.

The concept of immutable infrastructure revolves around building immutable deployment artifacts and then promoting them through your environments so that you can horizontally scale your web tier. These immutable deployment artifacts can be machine images or Docker images.

Phase2 has helped major media customers and sports leagues set up and maintain autoscaling infrastructures for their digital web platforms that include Drupal, and in this talk we will talk about the requirements for implementing an autoscaling environment in a public cloud like Amazon’s EC2, as well as discussing how this translates into the use of Docker and an application scheduler.

In this talk, we’ll go over the following topics:
- How to set up a build process to produce machine images (e.g. AMIs) or Docker images
- How to use autoscaling in practice to save money and be able to automatically handle traffic spikes
- Hear about our experiences implementing these solutions for major media and sports leagues
- How to handle horizontal autoscaling of Docker containers in production using Kubernetes

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Loft Your Web Platform into the Clouds with Immutable Infrastructure

  1. 1. Loft Your Web PlatformIntothe Clouds withImmutable Infrastructure Steven Merrill
  2. 2. © 2016 Phase2 OVERVIEW
  3. 3. 4© 2016 Phase2 TOPICS COVERED Immutability in software development Immutable Infrastructure Building immutable artifacts Using immutable artifacts with AWS autoscaling Using immutable artifacts with containers Real-world examples
  4. 4. © 2016 Phase2 MUTABILITY
  5. 5. 6© 2016 Phase2 im·mu·ta·ble adjective unable to be changed
  6. 6. 7© 2016 Phase2 IMMUTABILITY IN SOFTWARE DEVELOPMENT First-class immutable variables in language runtimes No need to synchronize read access in parallel systems Immutable data stores Elm’s time-travel debugging Mercurial’s immutable history / write-ahead log Immutable data structures in React.js
  7. 7. © 2016 Phase2 IMMUTABLE INFRASTRUCTURE
  8. 8. 9© 2016 Phase2 IMMUTABLE INFRASTRUCTURE? WHAT? Continuation of infrastructure as code Core tenet: build immutable artifacts and promote them As in development, try to limit mutation of deployed infra Unchanging artifacts (AMIs, containers) eliminate drift
  9. 9. 10© 2016 Phase2 AN EXAMPLE A client had autoscaling troubles Provisioned from scratch from a plain Ubuntu AMI Problems: Drift Time to provision / autoscaling timeouts Hard dependency on configuration management server
  10. 10. 11© 2016 Phase2 COUNTERPOINTS Most servers may not have read-only disks Container systems can allow read-only rootfs Will you really disable SSH? One theory: contaminate instances if you do
  11. 11. 12© 2016 Phase2 WHY? Immutable infra practices enable autoscaling Rallying cry: have cattle, not pets It can allow for easier rollback Possibility for canary or blue/green deployment From here on out, we’ll discuss autoscaling readiness
  12. 12. © 2016 Phase2 FOUNDATIONS: BUILD PROCESSES
  13. 13. 14© 2016 Phase2 BUILD THEORY A modern web app likely has several build steps Pull dependencies down Source compilation CSS/JS compilation and assembly Phase2 has grunt-drupal-tasks to automate this Build from drush make, npm theme tasks Has a package command to make a folder or tarball
  14. 14. 15© 2016 Phase2 VENDORING Consider vendoring your dependencies Great discussion about this in “Who Owns Your Availability?” episode of Arrested DevOps
  15. 15. 16© 2016 Phase2 BUILD TARGETS Deployment repository Tarball Machine image Amazon AMI / Google Private images Rkt / Docker tagged container image
  16. 16. JENKINS: FORCE FOR GOOD OR EVIL
  17. 17. 18© 2016 Phase2 BUILD PROCESS EVOLUTION Awful Build in-place on each of n web heads Better Build once in workspace, rsync to docroots Good Build once, store artifact, capistrano-style deploy Goal Build artifact, build image around artifact
  18. 18. 19© 2016 Phase2 BUILDING IMAGES Autoscaling-ready image builds have 2 components: Tagged code releases A combination of Linux packages on a filesystem Linux vulnerabilities happen fairly frequently Be ready to build for new code releases and package CVEs
  19. 19. © 2016 Phase2 AUTOSCALING
  20. 20. 21© 2016 Phase2 WHY AUTOSCALE? Reduces stress Automatic healing Cost benefits Performance benefits Automatic scaling based on metrics Manual adjustment up and down
  21. 21. © 2016 Phase2 AUTOSCALING DRUPAL IN AMAZON WEB SERVICES
  22. 22. 23© 2016 Phase2 DRUPAL SITE COMPONENTS Big 3 components of a running Drupal site Codebase on an app server Relational database Files directory Optional Object cache (?) Varnish / CDN
  23. 23. 24© 2016 Phase2 AUTOSCALING GROUPS IN AWS An autoscaling group is attached to one or more ELBs ELB health checks and timeouts control ASG actions You use CloudWatch alarms to trigger autoscaling Avg CPU > 70% for 5 min, avg CPU < 40% for 30 min A LaunchConfig ties together an AMI and instance size e.g. ami-beefbeef on c4.xlarge w/ 100 GB disks One ASG is set to one LaunchConfig at a time
  24. 24. 25© 2016 Phase2 AWS RECIPE FOR DRUPAL Jenkins instance building tarballs + AMIs Autoscaling group with public Drupal instances RDS for MySQL S3 bucket for files directory via amazons3.module ElastiCache for memcache via memcache.module CloudFront for page caching / distribution SES for email via smtp.module
  25. 25. 26© 2016 Phase2 EXAMPLE AMI AUTOSCALING SETUP Have one build process to create a tarball / build artifact Use Packer to start from a base AMI File provisioner copies code up Run Ansible to install packages Create new LaunchConfig Flip instances over one at a time
  26. 26. 27© 2016 Phase2 ELIDED PACKER EXAMPLE { "variables": { "aws_access_key" : "xxxxxxxxxxxxxxxxxxxx", "aws_secret_key" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "aws_instance_type" : "t2.micro", "east_aws_ami" : "ami-00000000", "west_aws_ami" : "ami-11111111", "aws_ssh_username" : "ec2-user", “east_aws_subnet" : "subnet-00000000”, “west_aws_subnet" : "subnet-11111111", "east_aws_vpc": "vpc-00000000", "west_aws_vpc": "vpc-11111111" },
  27. 27. 28© 2016 Phase2 ELIDED PACKER EXAMPLE "provisioners": [{ "type": "file", "source": "/path/to/my/build.tgz", "destination": "/opt/build.tgz" }, { "type" : "shell", "inline": [ "/usr/bin/ansible-playbook -ilocal /opt/playbook.yml” ]}], ],
  28. 28. 29© 2016 Phase2 ELIDED PACKER EXAMPLE "builders": [{ "type": "amazon-ebs", "name": "east-ami", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "{{user `east_aws_ami`}}", "instance_type": "{{user `aws_instance_type`}}", "ssh_username": "{{user `aws_ssh_username`}}", "security_group_ids" : ["sg-00000000"], "subnet_id" : "{{ user `east_aws_subnet` }}", "vpc_id": "{{user `east_aws_vpc`}}", "ami_name": "autoscale_{{user `build_tag`}}_east" },
  29. 29. 30© 2016 Phase2 ELIDED PACKER EXAMPLE { "type": "amazon-ebs", "name": "west-ami", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-west-2", "source_ami": "{{user `west_aws_ami`}}", "instance_type": "{{user `aws_instance_type`}}", "ssh_username": "{{user `aws_ssh_username`}}", "security_group_ids" : ["sg-11111111"], "subnet_id" : "{{ user `west_aws_subnet` }}", "vpc_id": "{{user `west_aws_vpc`}}", "ami_name": "autoscale_{{user `build_tag`}}_west" }]}
  30. 30. 31© 2016 Phase2 S3 SHARP EDGES S3 files with amazons3/s3fs.module use memory_limit You may be tempted to try the s3fs daemon Instead, s3fs_cors/amazons3_cors modules
  31. 31. 32© 2016 Phase2 AUTOSCALING QUIRKS Schema updates Whitescreens on new/old as schema changes flip Add double capacity, let ELB health checks do the work Maintenance mode Separate editorial instance Build one image and switch behavior on ENV
  32. 32. © 2016 Phase2 CLIENT EXAMPLE
  33. 33. 34© 2016 Phase2 MLS DIGITAL - MLSSOCCER.COM AND CLUBS 21-site Drupal 7 platform League and all club sites Game day scaling patterns - huge traffic spikes No autoscaling, separate docroots Moved to multisite in prep Build tarballs from 500 MB to 40 MB Total opcache / APC savings of 1.6 GB
  34. 34. 35© 2016 Phase2 MOVING MLS TO AUTOSCALING Old setup provisioned from a base Ubuntu with Salt Timeout problems Moved to Packer AMI builds with all software & code An AMI can come up if everything else is down Systems of record for Drupal state are elsewhere MySQL database - Amazon RDS Memcache - Amazon ElastiCache Files - Amazon S3 via s3fs.module
  35. 35. 36© 2016 Phase2 SUCCESS From 4 pets to 2-5 cattle Major hosting cost reduction Later Drupal optimizations lowered instance size
  36. 36. A TYPICAL MONTH
  37. 37. DRUPAL OPTIMIZATIONS
  38. 38. AFTER TUNING
  39. 39. © 2016 Phase2 CONTAINERS
  40. 40. 41© 2016 Phase2 POTENTIAL DOCKER BENEFITS Increased density Ability to run heterogenous workloads Native CPU and disk performance Resource limits
  41. 41. 42© 2016 Phase2 DOCKER IN PRODUCTION Docker on single hosts is easy Docker in production requires a lot of coordination Overlay networking / network security Service discovery Intelligent scheduling and capacity management Routing / load balancing Secrets management Monitoring and performance management
  42. 42. 43© 2016 Phase2 DOCKER BUILDS Build considerations Dockerfile vs configuration management Dynamic templating Build tools docker build Packer Configuration management system
  43. 43. 44© 2016 Phase2 DOCKER Docker daemon allows for tagging Ideally, use tags from canonical repo and as branches p2/site:v1.0.0, p2/site:v1.1.0 p2/site:dev, p2/site:stage, p2/site:prod Build script docker build -t p2/site:v1.1.0 . docker tag p2/site:v1.1.0 p2/site:dev docker push p2/site:v1.1.0; docker push p2/site:dev
  44. 44. 45© 2016 Phase2 KUBERNETES Open-source container scheduler from Google “Manage a cluster of Linux containers as a single system to accelerate Dev and simplify Ops.” Based on Borg design principles Large pool of contributors and upstream/downstream users Red Hat (OpenShift 3) Deis (Helm) CoreOS (Tectonic)
  45. 45. 46© 2016 Phase2 KUBERNETES Requires you to bring overlay networking Features Service discovery, load balancing, cluster DNS Pod scaling and autoscaling, node draining Resource-aware scheduling and capacity limits REST API and CLI clients Secrets management YAML or JSON definition of k8s API objects
  46. 46. 47© 2016 Phase2 KUBERNETES OBJECTS Nodes Labels Pods DaemonSets Replication controllers Services Deployments
  47. 47. 48© 2016 Phase2 KUBERNETES POD YAML apiVersion: v1 kind: Pod metadata: name: lamp app: lamp spec: containers: - name: web image: phase2/apache24php55 resources: requests: memory: "512Mi" cpu: "1000m" limits: memory: "1024Mi" cpu: "2000m”
  48. 48. 49© 2016 Phase2 KUBERNETES RC YAML apiVersion: v1 kind: ReplicationController metadata: name: lamp spec: replicas: 5 selector: app: lamp template: metadata: labels: app: lamp spec: containers: - name: lamp image: phase2/apache24php55 resources: requests: memory: "512Mi" cpu: "1000m" limits: memory: "1024Mi" cpu: “2000m” ports: - containerPort: 8080 name: lamp protocol: TCP
  49. 49. 50© 2016 Phase2 KUBERNETES SERVICE YAML apiVersion: v1 kind: Service metadata: name: lamp spec: selector: app: lamp ports: - name: lamp port: 8080
  50. 50. © 2016 Phase2 SUMMARY
  51. 51. 52© 2016 Phase2 IMMUTABLE INFRASTRUCTURE Build immutable artifacts for deployment Limit configuration drift in your infrastructure Be ready to build new images for code releases or CVEs Autoscaling can help with cost, stress, and self-healing
  52. 52. © 2016 Phase2 QUESTIONS?
  53. 53. SoHow Was It?- Tell Us What YouThink Evaluate this session - https://events.drupal.org/neworleans2016/schedule Thanks!

×