Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Case Study: Using Terraform and Packer to deploy go applications to AWS

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 92 Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Case Study: Using Terraform and Packer to deploy go applications to AWS (20)

Anzeige

Aktuellste (20)

Case Study: Using Terraform and Packer to deploy go applications to AWS

  1. 1. Case Study: Using Terraform and Packer to deploy go applications to AWS Patrick Bolduan April 2017
  2. 2. Presenter CV Patrick Bolduan Digital Technology Department, Global Digital Division ASICS Corporation 20 years building web applications (product/app dev/ops) 12 years involved platform operations (MTV/ASICS) 16 years in Japan よろしく
  3. 3. ASICS Corporation Global Headquarters: Kobe Global presence: Over 240 countries Digital Presence: Over 50 countries Global Brands ASICS, Onitsuka Tiger, ASICS Tiger, Haglöfs, Runkeeper
  4. 4. talk { start = true
  5. 5. section { name = “Business context for this talk”
  6. 6. Digital platform operations for ASICS Corporation
  7. 7. Story time
  8. 8. 2015
  9. 9. ASICS Corporation Global Headquarters: Kobe Global presence: Over 240 countries Digital Presence: Over 50 countries Global Brands ASICS, Onitsuka Tiger, ASICS Tiger, Haglöfs, Runkeeper
  10. 10. ASICS Corporation global footprint Global presence: Over 240 countries Digital Presence: Over 50 countries Over 120 websites Around 8 global web application platforms More than 5 data centers with a mix of bare metal and cloud
  11. 11. ASICS Corporation ops team (all outsourced) (to multiple vendors)
  12. 12. Outsourced ops challenges ● Ops practices inconsistent across vendors ● Missed opportunities for leveraging scale ● Performance varied from poor to just “OK”
  13. 13. Ops improvement plan proposed... Build in-house operations team to manage strategic platforms mostly...
  14. 14. Modern ops team building research PaaS PaaS Search internet Attend conferences
  15. 15. What is a PaaS? Platform as a service (PaaS) or application platform as a service (aPaaS) is a category of cloud computing services that provides a platform allowing customers to develop, run, and manage applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app. https://en.wikipedia.org/wiki/Platform_as_a_service IaaS PaaS SaaS Software as a Service Platform as a Service Infrastructure as a ServiceXaaS Pyramid
  16. 16. Ops is automated to the point where all engineers are focused on delivering value to your product PaaS goals =
  17. 17. What is a full blown PaaS good for? Environments with many different apps with variable stack requirements
  18. 18. Platform management evolution Bare metal Cloud Cloud automation PaaS
  19. 19. Example providers and tools Cloud Cloud automation PaaS
  20. 20. Management automation offload matrix Bare metal Cloud Cloud automation Cloud + deployment automation PaaS Hardware ✕ ◯ ◯ ◯ ◯ Network ✕ ✕ △ △ ◯ Server ✕ ✕ △ △ ◯ Application ✕ ✕ ✕ △ ◯
  21. 21. PaaS focused team building... ● Review and select PaaS vendor options ● Write job descriptions for FTE team members ● Start vendor contract negotiations and FTE hiring process...
  22. 22. 2016
  23. 23. Already has in-house ops team acquires
  24. 24. This is good news for ASICS ops (but team planning needs to be reset)
  25. 25. We need a PaaS RIGHT NOW!!! Hold on, how about we try some automation first? Platform strategy reset negotiations... ok
  26. 26. } section { name = “Case study: automation with Terraform and Packer”
  27. 27. Walk before you run
  28. 28. Initial project: automation for a single web app
  29. 29. Initial project: app stack ● Cloud infrastructure: AWS ● Infrastructure automation: Terraform ● Deploy automation: Packer ● The app: QOR – golang CMS Project: https://getqor.com Github: https://github.com/qor/qor
  30. 30. Initial project: target end state Project: https://getqor.com Github: https://github.com/qor/qor ● Automated infrastructure management ● Manually triggered consistent app deploys ● Ability to create ad hoc full copies of production
  31. 31. Functional walkthrough
  32. 32. ASICS selected AWS as our public cloud provider
  33. 33. Baseline app infrastructure diagram
  34. 34. The repo
  35. 35. Terraform repo tree structure |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo Based on https://charity.wtf/2016/03/30/terraform-vpc-and-why-you-want-a-tfstate-file-per-env/
  36. 36. Terraform repo tree structure |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo Shared infrastructure configurations
  37. 37. Terraform repo tree structure |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo Environment specific variations
  38. 38. Terraform repo tree structure |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo Application build tools for all envs
  39. 39. Terraform repo tree structure |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo Application submodule
  40. 40. What is Terraform? Terraform is a cloud management automation tool Why was Terraform used in this case? Terraform has robust cloud feature automation capabilities and simple code structure
  41. 41. Terraform infrastructure diagram This is an autoscaling group now And we added a CloudFront
  42. 42. Terraform defined infrastructure
  43. 43. Terraform config file basics .tf files: Terraform configuration files .tfvars files: variable for vars referenced in .tf files
  44. 44. We will be looking at .tf files today
  45. 45. Network tier
  46. 46. |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo Network tier
  47. 47. module "vpc" { ... } resource "aws_subnet" "db" { ... } resource "aws_route_table_association" "db" { ... } resource "aws_security_group" "bastion_security_group" { ... } resource "aws_security_group_rule" "bastion_vpc_ingress" { ... } resource "aws_security_group_rule" "bastion_ssh_ingress" { ... } resource "aws_security_group_rule" "bastion_all_egress" { type = "egress" Terraform network example: VPC + SG
  48. 48. Terraform network example: ssh bastion resource "aws_security_group_rule" "bastion_ssh_ingress" { type = "ingress" from_port = "22" to_port = "22" protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] security_group_id = "${aws_security_group.bastion_security_group.id}" } data "aws_ami" "bastion_ami" { most_recent = true filter { name = "name" values = ["${var.bastion_ami_name}"] } owners = ["amazon"] } resource "aws_instance" "bastion_instance" { ami = "${data.aws_ami.bastion_ami.id}" instance_type = "${var.bastion_instance_type}" instance_initiated_shutdown_behavior = "terminate"
  49. 49. Edge tier
  50. 50. |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo Edge tier
  51. 51. Terraform edge example: ALB resource "aws_alb" "application" { name = "tf-lb-app-${var.stack}-${var.env}" internal = false security_groups = ["${aws_security_group.lb_app_security_group.id}"] subnets = ["${var.lb_subnets}"] idle_timeout = "${var.idle_timeout}" access_logs { bucket = "${var.log_bucket}" prefix = "logs/alb" } tags { Name = "tf-lb-app-${var.stack}-${var.env}" } depends_on = ["aws_s3_bucket_policy.log_bucket_policy"] }
  52. 52. Terraform edge example: CloudFront resource "aws_cloudfront_distribution" "static_cloudfront" { origin { domain_name = "${var.static_bucket}.s3.amazonaws.com" origin_id = "${var.static_bucket}" s3_origin_config { origin_access_identity = "${aws_identity.static_cloudfront.cloudfront_path}" } } enabled = true comment = "Terraform-managed distribution for ${var.stack}-${var.env}" default_root_object = "index.html" aliases = ["${compact(concat(list(domain),var.static_dns_aliases))}"] http_version = "http2" price_class = "${var.cloudfront_price_class}" default_cache_behavior { allowed_methods = ["DELETE", "GET", "HEAD", "POST", "PUT"] cached_methods = ["GET", "HEAD"] target_origin_id = "${var.static_bucket}"
  53. 53. App tier
  54. 54. |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo App tier
  55. 55. Terraform app example: ASG data "aws_ami" "application_ami" { most_recent = true filter { name = "image-id" values = ["${var.application_ami_id}"] } owners = ["self","amazon"] } resource "aws_autoscaling_group" "application_asg" { name = "tf-asg-app-${var.stack}-${var.env}-${aws_launch_configuration.application_lc.name}" vpc_zone_identifier = ["${var.application_subnets}"] min_size = "${var.app_min_size}" max_size = "${var.app_max_size}" desired_capacity = "${var.app_desired_capacity}" wait_for_elb_capacity = "${var.app_desired_capacity}" health_check_grace_period = "${var.health_check_grace_period}" health_check_type = "${var.health_check_type}" termination_policies = ["OldestLaunchConfiguration", "ClosestToNextInstanceHour", "Default"]
  56. 56. Data tier
  57. 57. |____base |____env-dev |____env-production |____env-[other envs...] |____modules | |____app_tier | |____data_tier | |____edge_tier | |____network |____packer | |____application-sub-repo Data tier
  58. 58. Terraform data example: Aurora cluster resource "aws_rds_cluster" "aurora" { cluster_identifier = "tf-rds-${var.stack}-${var.env}" availability_zones = "${var.azs}" database_name = "${var.database_name}" master_username = "${var.master_username}" master_password = "${var.master_password}" backup_retention_period = "${var.backup_retention_period}" preferred_backup_window = "${var.preferred_backup_window}" vpc_security_group_ids = ["${aws_security_group.aurora_sg.id}"] storage_encrypted = "${var.storage_encrypted}" kms_key_id = "${aws_kms_key.asics_digital_rds_key.arn}" apply_immediately = "${var.apply_immediately}" db_subnet_group_name = "${aws_db_subnet_group.aurora_subnet_group.id}" db_cluster_parameter_group_name = "${aws_rds_cpg.aurora_cpg.id}" lifecycle { prevent_destroy = "true" } } resource "aws_rds_cluster_instance" "aurora_instance" {
  59. 59. Terraform process write { code } $ terraform plan $ terraform apply = updated
  60. 60. What is Packer? Packer is a tool for creating machine images (AMIs) Why was Packer used in this case? Outputs AMIs as artifacts which can easily be used to populate Terraform managed ASGs
  61. 61. Packer machine image build process 1. platform image build a. image with bootstrap dependencies to app builds b. reusable – update as required by dependencies 2. application image build a. uses platform image a starting point b. app and dependencies installed from git repo c. image used to populate asgs
  62. 62. Packer folder tree structure |____application.json |____bootstrap.sh |____build.sh |____deploy.pp |____dev.yaml |____[other dev].yaml |____platform.json |____prod.yaml |____provision.pp |____staging.yaml |____application-sub-repo
  63. 63. Packer config file basics .json files: Packer configuration files with build rules .yaml files: variable used in the image build process .sh files: build process control scripts .pp files: puppet manifests
  64. 64. |____application.json |____bootstrap.sh |____build.sh |____deploy.pp |____dev.yaml |____[other dev].yaml |____platform.json |____prod.yaml |____provision.pp |____staging.yaml |____application-sub-repo Packer configs
  65. 65. Packer json example: app build config { "variables": { "aws_default_region": "{{env `AWS_DEFAULT_REGION`}}", "source_ami": "{{env `PLATFORM_AMI`}}", "tier": "{{env `APP_TIER`}}" }, "builders": [{ "type": "amazon-ebs", "region": "{{user `aws_default_region`}}", "source_ami": "{{user `source_ami`}}", "instance_type": "m3.medium", "ssh_username": "ec2-user", "ami_name": "asics-cms-application-{{timestamp}}", "tags": { "Name": "asics-cms-{{user `tier`}}-application" } }], "provisioners": [ { "type": "file",
  66. 66. |____application.json |____bootstrap.sh |____build.sh |____deploy.pp |____dev.yaml |____[other dev].yaml |____platform.json |____prod.yaml |____provision.pp |____staging.yaml |____application-sub-repo Packer var files
  67. 67. Packer yaml example: app build vars app_config: ASICS3_DB_NAME: "asics3" ASICS3_DB_USER: "******" ASICS3_DB_PASSWORD: "******" ASICS3_DB_HOST: "[RDS cluster details].rds.amazonaws.com" ASICS3_ENV: "dev" QOR_AWS_REGION: "us-west-2" QOR_AWS_ACCESS_KEY_ID: "" QOR_AWS_SECRET_ACCESS_KEY: "" S3Bucket: "[S3 bucket name]" ASICS3_HTTP_ADDR: ":8000" GOPATH: "/go" QOR_AWS_CLOUD_FRONT_DOMAIN: "https://[cloudfront host].asics.digital"
  68. 68. |____application.json |____bootstrap.sh |____build.sh |____deploy.pp |____dev.yaml |____[other dev].yaml |____platform.json |____prod.yaml |____provision.pp |____staging.yaml |____application-sub-repo Build scripts
  69. 69. Packer sh example: app build script #!/bin/bash set +e echo "*** Configuring golang..." source /etc/profile.d/golang.sh echo "*** Sourcing env vars..." source /go/.envrc echo "*** Changing to source directory..." cd /go/src/github.com/[repo root]/asics3 echo "*** Installing glide..." go get github.com/Masterminds/glide echo "*** Workaround for go4.org TLS issue..." mkdir -p "/root/.glide/cache/src" && git clone https://github.com/camlistore/go4 /root/.glide/cache/src/https-go4.org
  70. 70. |____application.json |____bootstrap.sh |____build.sh |____deploy.pp |____dev.yaml |____[other dev].yaml |____platform.json |____prod.yaml |____provision.pp |____staging.yaml |____application-sub-repo puppet manifests
  71. 71. Packer pp example: puppet manifest # install golang # $workspace = '/go' $workdirs = [ "${workspace}/bin", "${workspace}/pkg", "${workspace}/src", ] $app_user = 'app' $hiera_datadir = '/etc/puppet/hieradata' # we need augeas for later on yumrepo { 'epel': enabled => 1, } -> package { 'ruby-augeas': ensure => 'present',
  72. 72. Packer platform build
  73. 73. Packer and Terraform deploy process update { app asg ami } $ terraform plan $ terraform apply $ packer build Code deployed! get ami
  74. 74. 2017
  75. 75. Operational metrics
  76. 76. Ops team manual task offload ● >90% ops tasks automated with Terraform ● Some manual task remain: SSL cert, production DNS ● Ongoing tasks: perf monitoring, architectures changes
  77. 77. App dev team impact ● Long* deploy process: packer build → terraform apply ● Easy to provision number of identical dev environments ● Reliable cross environment testing *For this case Packer builds take between 15 and 20 minutes.
  78. 78. General impact ● 3 more apps have been automated after initial project ● Overall manual ops team tasks going down ● App teams have reliable access to test environments ● All environments are consistent across dev process ● General impact: very positive
  79. 79. Other critical resources
  80. 80. Steve Huff Runkeeper SRE github.com/hakamadare
  81. 81. Frank Yue SA and Devops at The Plant github.com/frankyue
  82. 82. } section { name = “Next steps and takeaways”
  83. 83. Next steps for ASICS ops ● Automate deploy process with Jenkins and Ansible ● Trigger app deploys with git commits ● Apply learnings to other app environments ● Iterate and improve capabilities towards PaaS
  84. 84. Takeaways ● PaaS still relatively new as a concept ● Lots of opportunities to reduce burden of ops tasks ● “Ideal ops state” will depend on your needs
  85. 85. Honorable mention ● Have a plan for managing keys and secrets
  86. 86. }
  87. 87. Thank you
  88. 88. Questions? Twitter: @ahq_p Keybase: asicspatrick Github: asicspatrick

×