Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Large Scale Cloud Infrastructure Using Shared Components

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 19 Anzeige

Large Scale Cloud Infrastructure Using Shared Components

Rasmus Møller Selsmark
DevOps team lead – Unity Technologies
Rasmus Selsmark has developed software professionally for 20 years, last 5 of these at Unity Technologies in various roles from Quality and Test automation to now being Lead for the DevOps team in Unity Ads. Believes that quality is measured as value to the user, both for internal and external facing software projects. Moved from Denmark to Finland 4 years ago, and can often be found biking in the forests around Helsinki during weekends.

Rasmus Møller Selsmark
DevOps team lead – Unity Technologies
Rasmus Selsmark has developed software professionally for 20 years, last 5 of these at Unity Technologies in various roles from Quality and Test automation to now being Lead for the DevOps team in Unity Ads. Believes that quality is measured as value to the user, both for internal and external facing software projects. Moved from Denmark to Finland 4 years ago, and can often be found biking in the forests around Helsinki during weekends.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Large Scale Cloud Infrastructure Using Shared Components (20)

Anzeige

Weitere von Eficode (20)

Aktuellste (20)

Anzeige

Large Scale Cloud Infrastructure Using Shared Components

  1. 1. G en er ati ve Ar t – M ad e wi th U nit y Large Scale Cloud Infrastructure Using Shared Components 1
  2. 2. Practical examples of building a large scale Kubernetes infrastructure, handling 50K requests/sec, by distributing development of shared components, increasing ownership and reducing bottlenecks in the development process About me: Rasmus Selsmark DevOps Team Lead, Unity Ads 2
  3. 3. 3
  4. 4. 4 Publishers Advertisers Monetizing their games Acquiring users
  5. 5. 5
  6. 6. >1.2B mobile devices ~50K requests/sec 16K CPUs on cloud infra 6
  7. 7. High-level architecture 7 NATS (nats.io) Events EventsAd delivery Databases Game config, ads (HTTP) Configuration Tools qMongoDBDatabases Data Pipeline Kafka S3/GCS etc
  8. 8. DevOps Handbook describes three primary types of organizational structures Scaling Engineering Teams 8 Matrix-oriented: combination of functional and market oriented Functional-oriented: centralizing/optimizing expertise Market-oriented: optimize for fast response to customer needs, each team is responsible for feature delivery and deployment Model at Unity
  9. 9. Unity Ads dev teams 9 Unity Ads: 140 developers in Helsinki and San Francisco offices 20 dev teams ~200 repositories ~90 microservices Unity: 2400 employees worldwide SRE Seattle SRE Helsinki SRE Shanghai
  10. 10. Component ownership and development 11 Shared: Cloud infra, network, Prometheus, Terraform Enterprise Shared Unity infrastructure Common CI/Build and Deployment Pipeline DevOps team Shared development Terraform infrastructure Microservices Databases Terraform infrastructure Microservices Databases DevOps Handbook deployment pipeline requirements: ● Automated, repeatable and predictable ● Consistent environments, by using same deployment tools for staging and production ● Enabling easy automated self-service deploymentsTerraform modules Monitoring and alerts framework Messaging + monitoring libraries q Dev teams q “Internal Open Source Model”: ● Typically one team is maintainer, not necessarily only developer of a shared module ● Focus on consistency, while allowing others to contribute ● Most development is done by dev teams, not DevOps team
  11. 11. The tools we’re using to support model with teams owning and deploying services ● Terraform ● Kubernetes / Helm ● Jenkins / GitLab CI 12 Scaling Services
  12. 12. Keeping build and deployment relevant configuration in service repo, makes it visible to team, allowing independent ownership and simplifies workflows for the team All configuration in service repo client (...) helm prd.yaml stg.yaml values.yaml scripts build.sh test.sh server (...) Dockerfile Jenkinsfile monitoring.yml sonar-project.properties13
  13. 13. ● All services deployed using same common build pipeline ● Using Jenkins shared libraries / GitLab CI include files: ○ https://jenkins.io/doc/book/pipeline/shared-libraries ○ https://docs.gitlab.com/ee/ci/yaml/#include Common build pipeline if (config.deploy_prod == "true") { stage('Deploy to production') { print "Check if we can auto-deploy for ..." if (config.skipConfirmForUsers != null && ...) { print "Skip deployment confirmation" deployToProduction() } else { timeout(time: 1, unit: 'HOURS') { userInput = input(message: 'Deploy?', ...) if (userInput) { deployToProduction() Common build pipeline logic example 14 Script { deploy = "helm" deploy_prod = "true" staging_env = ["ads-gke-stg"] production_env = ["ads-gke-prd"] } Jenkinsfile for a service repo, only containing configuration relevant to build/deploy, no code
  14. 14. Infrastructure as Code - Terraform 15 resource "google_compute_instance" "mongodb" { name = "mongodb-${var.cluster_name}-(...)" machine_type = "${var.machine_type}" (...) } resource "google_compute_disk" "mongodb-disk" { size = "${var.data_disk_size}" type = "${var.data_disk_type}" (...) } Shared Terraform module module "mongodb_ads_selfserve_stg" { source = "tfe/unity-ads/mongodb/google" version = "1.1.6" machine_type = "n1-standard-2" data_disk_type = "pd-ssd" data_disk_size = "40" cluster_size = "3" (...) } Module usage/implementation in team repo
  15. 15. ● https://helm.sh - “The package manager for Kubernetes” ● Abstracts the complexity of Kubernetes manifests for dev teams, only specifying parameters relevant for deploying the service ● Helm templates stored in central repository, maintained by DevOps team ● Helm config is stored in service repo, i.e. with application code ● Shared “unity-common-chart” chart, hosted on internal https://github.com/helm/chartmuseum repository Helm and Kubernetes 16
  16. 16. {{- if .Values.enableDeployment -}} apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: {{ .Values.namespace }} name: {{ .Chart.Name }} labels: chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}" app: {{ .Chart.Name }} environment: {{ .Values.environment }} productgroup: {{ .Values.productgroup }} spec: progressDeadlineSeconds: {{ .Values.deployment.progressDeadlineSeconds }} (...) unity-common-chart/templates/deployment.yaml Helm chart example and usage 17 Using Helm template ensures consistent naming across services, which would be harder to achieve if teams maintained their individual Kubernetes manifest files Service helm usage environment: stg deployment: replicas: 2 resources: requests: cpu: 0.1 memory: 256Mi limits: cpu: 0.1 memory: 512Mi
  17. 17. Jenkins build + deployment 18
  18. 18. ● We have found a model, which works well for us in terms of organizational and technical implementation of our service ownership model ● Don’t underestimate the task for dev teams to own their infrastructure. Teams needs support from organization 19 Learnings
  19. 19. Generative Art – Made with Unity Thank you! 20 Rasmus Selsmark rasmus@unity3d.com https://careers.unity.com/location/helsinki

×