Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

PayPal's Cloud Journey From Folsom to Kilo

1.069 Aufrufe

Veröffentlicht am

At Paypal, we spent more than one year to upgrade all our Openstack deployments from folsom/grizzly to havana. In this talk, we would like to share the lessons we learned the hard way from the previous upgrade, and how we plan to change the process starting from kilo upgrade. Our final target is to bring Paypal's openstack cloud up to date with upstream and to continuously integrate with upstream.

Veröffentlicht in: Ingenieurwesen

PayPal's Cloud Journey From Folsom to Kilo

  1. 1. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. PayPal's Cloud Journey From Folsom to Kilo Wei Tian -- Cloud Performance Lead at Paypal 10/ 28 / 2015 What We Learned in the Upgrade Progress
  2. 2. © 2014-15 PayPal Inc. All rights reserved. Confidential and proprietary. Agenda 2 • About paypal Cloud • Past Upgrade before Kilo • Kilo Upgrade • What next ?
  3. 3. © 2014-15 PayPal Inc. All rights reserved. Confidential and proprietary. About PayPal Cloud 3 • Background – Started in July 2012 with 1 engineer and 16 decommissioned servers – Today, one of the world’s Largest OpenStack Private Cloud – Number of Physical Servers: 8064 – Number of Racks: 84 – Total Cores: 386,000 – Block Storage: 2 peta bytes – Largest AZ with 2500+ hypervisors • Business Goals – Hosting ~100% of PayPal’s production traffic (except Databases and Messaging) – Powers 100% of PaaS, Dev/QA and M&As – First production workload on SDN in 2013
  4. 4. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 4 Upgrade History 4 • Early 2014, Multiple versions of Openstack in 10 AZ’s. • Grizzly and Folsom. • Time to upgrade to Havana – 1 YEAR • Decision – Skip Icehouse and Juno • Don’t want to be in constant catch-up mode • Upgrade directly to Kilo. Current Status • One AZ in Kilo • Rest In-flight
  5. 5. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 5 Upgrade is Difficult! 5 • One of the largest Openstack private cloud. • 100% of the Paypal production • In Service Upgrade – No Availability Impact Allowed. • Mixed Folsom and Grizzly environments! • Nova-network AND Neutron networking. • Custom Code in-line with Upstream • Two months to code-ready.
  6. 6. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 6 Complicated Code Base – Need Manual Merge 6
  7. 7. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 7 Prepare the Code for the Upgrade – it Takes Time 7
  8. 8. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 8 Custom Database Migration 8 • Custom tables (besides Nova tables) • Custom DB migration script for Nova to migrate content from the custom tables. • Custom DB migration script for keystone
  9. 9. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 9 Seamless Migration from Nova-network to Neutron 9 • At Paypal, several data centers were using Folsom nova-networking in production with thousands of VMs in service. We successfully upgraded to OpenStack Havana, adopted Neutron as a Network service instead of nova-networking, and replaced Linux bridge by Open vSwitch. • The upgrade covered both control plane and data plane migration. The control plane includes neutron network/subnet/port management and SDN controller integration, while the data plane includes tap device migration from Linux bridge to openvswitch bridge, DHCP service, security group, and Libvirt/KVM configuration • For more details, Check out the presentation from my colleagues at the Paris Summit: https://www.openstack.org/summit/openstack-paris- summit-2014/session-videos/presentation/seamless- migration-from-nova-network-to-neutron-in-ebay- production
  10. 10. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 10 Openstack Services Deployment with Virtualenv 10 • Each controller node runs multiple services (keystone, nova, glance, neutron, cinder, etc.) and each service runs in its own virtual environments. • The best way to have painless and reproducible deployments is to package whole virtual environments of the application you want to deploy including all dependencies but without configuration. • The benefit for upgrade with virtualenv packages: • Speed. Deploying a new version is as simple as unzipping a tar ball. • Predictability. Each service running with its own virtualenv and can be upgrade independently. • Easy rollback. All versions of a service exist under different folders, and to rollback is simply changing the startup script with different working directory. • Simple Puppet Script. Only deal with startup script and config file for Openstack services.
  11. 11. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 11 Kilo Upgrade from Havana 11 Principles • No down time on data plane • A few hours of down time on control plane. • Upgrade on Saturday while the usage is low. • Prepare hypervisors as much as we can before the upgrade day. Process • Code readiness for kilo. • Build a shadow control plane. • Prepare run books for overall process and for each components. • Dry-run I • QA cycle I with Functional testing, performance and load testing. • Dry-run II • QA cycle II with Functional testing, performance and load testing. • Pre-upgrade. Prepare all hypervisors. • The official upgrade.
  12. 12. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 12 Code Readiness for Kilo – Branch Strategy 12
  13. 13. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 13 Code Readiness for Kilo – Coding Guidelines 13 Guidelines • No change to upstream code. • No directly back port or merge from havana changes. • Completely re-write the Paypal extensions align with kilo code. • Apply standard Openstack extension method.. • Actively anticipant in upstream. • Report bug to upstream. • Fix bug in upstream. Ways to customize Openstack • WSGI middleware. • Openstack API extensions (resource, controller, and child resource). • Extending manager classes for services. • Custom filters and weighers. • Custom RPC methods. • Nova hooks. • Monkey patch as a last resort.
  14. 14. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 14 Code Readiness for Kilo – Project Structure 14
  15. 15. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 15 Build a Shadow Control Plane 15
  16. 16. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 16 Dry-Run 16
  17. 17. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 17 The Benefit of Using Latest Release from Upstream 17 • Latest features and bug fixes from upstream. • We can also GIVE BACK to the community. • No More cherry picking bug fixes. • Older branches DELETED from Github !!! • NO CODE DUPLICATION between Upstream and PayPal: • In Folsom, we implemented ‘compute zone’. • Similar to host aggregate. • In Grizzly, we implemented a list of ‘aggregate_XXXX_filter’. • Similar to the ‘aggregate_XXXX_filter’ in havana • In Havana, we implemented nova extension to show host status and instance faults in nova list and nova show. • Was Added Upstreamin Juno
  18. 18. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 18 Prepare for the Liberty Upgrade 18 • After the overhaul refactoring in both code base and upgrade process, the preparation time for Liberty upgrade will be weeks instead of months. • From kilo to liberty, we could apply the recommended upgrade process from upstream and do a live upgrade without down time in either control plane or data plane. • The schedule for Liberty upgrade will be early next year.
  19. 19. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 19 Questions ?

×