Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Paradigm shift in CI at PayPal with Docker and Mesos

251 Aufrufe

Veröffentlicht am

This deck covers how PayPal adopted open source tools Docker, Mesos, Jenkins and Aurora to build a scalable, highly available CI solution for PayPal which resulted in a Paradigm shift compared to the conventional VM based model

Veröffentlicht in: Software
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Paradigm shift in CI at PayPal with Docker and Mesos

  1. 1. Running CI at scale with containers at PayPal Sathiya Narayanan
  2. 2. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 2 At PayPal, we take CI seriously
  3. 3. ©2015 PayPal Inc. Confidential and proprietary. 3 o One Ginormous Jenkins Instance, customized for PayPal o 40,000 runnable jobs o 24,000 builds run each day on Single Instance o Stack: Java, Node, Python, C++ o CD : UnitTest, Static analysis and code coverage, & deployments o One Jenkins master with a pool of a hundred Jenkins slaves o Custom build system for 50 MLOC in <1 Min Single Jenkins instance One Giant Jenkins Butler!
  4. 4. ©2015 PayPal Inc. Confidential and proprietary. 4 o SPOF o Scalability o Change Management o No freedom for users o Inefficient resource management o Plugins don’t scale Limitations with single Jenkins instance Too much load!
  5. 5. ©2015 PayPal Inc. Confidential and proprietary. 5 o 1:1 model o Individual VMs running Jenkins for each Application o 2,500+VMs in use o Each Jenkins instance would have 2 executor on master to perform builds o Freedom for users Users loved it! Dedicated VM for each Jenkins Instance
  6. 6. Jenkins on Mesos o Light weight o Long running task o Marathon scheduler ©2015 PayPal Inc. Confidential and proprietary. 6 Jenkins Master o Provisioned on demand with Jenkins Mesos plugin o Flavors of slaves based on the application stack Jenkins slave
  7. 7. ©2015 PayPal Inc. Confidential and proprietary. 7 Optimized Resource Utilization with Mesos
  8. 8. ©2015 PayPal Inc. Confidential and proprietary. 8 Operating cost reduced 10X times After Mesos o ~2400 CPU’s, o 18 TB, o 7.2 TB RAM Before Mesos o ~ 24,000 CPU’s, o 180 TB Disk, o 54 TB RAM
  9. 9. ©2015 PayPal Inc. Confidential and proprietary. 9 After: ~2400 CPU’s, 18 TB, 7.2 TB RAM ~300K 2.7 million saving could buy: - A personal Jet for GPI 10X Savings Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million Recovery was slow without Mesos
  10. 10. ©2015 PayPal Inc. Confidential and proprietary. 10 1 0 After: ~2400 CPU’s, 18 TB, 7.2 TB RAM ~300K 2.7 million saving could buy: - A personal Jet for GPI 10X Savings Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million Instant Recovery with Mesos
  11. 11. ©2015 PayPal Inc. Confidential and proprietary. 11 After: ~2400 CPU’s, 18 TB, 7.2 TB RAM ~300K 2.7 million saving could buy: - A personal Jet for GPI 10X Savings Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million Our workload was too much for Marathon
  12. 12. ©2015 PayPal Inc. Confidential and proprietary. 12 o Zero outages and no customer impact. o Aurora was more stable o Aurora was able to scale to more than 5,000 jobs Replaced Marathon with Aurora
  13. 13. ©2015 PayPal Inc. Confidential and proprietary. 13 After: ~2400 CPU’s, 18 TB, 7.2 TB RAM ~300K 2.7 million saving could buy: - A personal Jet for GPI 10X Savings Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million 13 CIAPI Aurora Mesos Master Mesos Cluster Swift / Object Storage Zookeeper Jenkins Master Jenkins Slave Nginx Highly available setup Devxdash
  14. 14. Challenges ©2015 PayPal Inc. Confidential and proprietary. 14 o Inconsistent build environments between user workstation and CI build system o Tasks stepping onto each other o Providing conflicting tool set for different technology stacks o Different DevOps teams/applications had their own Mesos cluster o Slower rollout of new tools to the build environment o Inconsistent build environments within the CI build system
  15. 15. Why Docker? o Task isolation o Cloud Native o Eliminates the dependency on the host o Reproducibility o Portability o Immutability o Homogenous cluster ©2015 PayPal Inc. Confidential and proprietary. 15
  16. 16. ©2015 PayPal Inc. Confidential and proprietary. 16 10X Savings Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million Dockerized Jenkins Masters o Immutable containers for consistency o Eliminates the need to have tools installed on slave o Provides task isolation o Other work loads can co-exist in Mesos cluster o Easy to cater to conflicting OS / toolset requirements in a polyglot environment o Easy to roll out updates o Other work loads can co-exist in Mesos cluster o Users can bake their own build environment Dockerized Jenkins Slaves Jenkins with Docker
  17. 17. ©2015 PayPal Inc. Confidential and proprietary. 17 10X Savings Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million Adoption to Docker containers BC [ Before Containers ] AD [ After Docker ] SSH key / Tokens / Maven credentials Stored in local disk Injected only during build time and rotating them Toolsets like JDK, Maven etc. Installed in host Installed within Docker container
  18. 18. ©2015 PayPal Inc. Confidential and proprietary. 18 Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million What is state for Jenkins master ? o Jenkins war o Plugins o Global configuration o Job configuration o Build history o Console output o Reports like code coverage, test result trend etc. o Archived artifacts
  19. 19. ©2015 PayPal Inc. Confidential and proprietary. 19 Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million Backing up Jenkins state Superviso rd Archive the directory Encrypt the data Store in cloud storage
  20. 20. ©2015 PayPal Inc. Confidential and proprietary. 20 Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million When Jenkins Master container moves to another VM Untar the files into Jenkins home Decrypt the backup Check if a backup exists in cloud storage Provisioning script [Entry point for Docker]
  21. 21. ©2015 PayPal Inc. Confidential and proprietary. 21 10X Savings Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million Use case - Maven build Total build time - 4 minutes Persisting workspace for subsequent builds Decreased build time Source code [Clone] Artifacts Time taken 0m27.618s 2m56.000s Number of files 10570 4456 Total Size 158 M 173 M
  22. 22. ©2015 PayPal Inc. Confidential and proprietary. 22 10X Savings Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million CIAAS with Docker Architecture Public cloud Private cloud
  23. 23. ©2015 PayPal Inc. Confidential and proprietary. 23 After: ~2400 CPU’s, 18 TB, 7.2 TB RAM ~300K 2.7 million saving could buy: - A personal Jet for GPI Before: ~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM ~3 Million

×