Presentation given in Seoul, South Korea at the Cloud and Data Center Conference in March 2014. Introduces the concept of the Rackspace Hybric Cloud Experience, the product platforms that are being used to make that happen, and then focuses on the operation and deployment of the Public Cloud.
Operating OpenStack - Case Study in the Rackspace Cloud
1. Rainya Mosher, Software Dev Manager
Infrastructure Deployment Systems
Twitter: @rainyamosher | Email: rainya.mosher@rackspace.com
Operating OpenStack at
Rackspace for a Hybrid Cloud
Experience
20 March 2014
2. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Name: Rainya Mosher
• Twitter: @rainyamosher
• IRC Freenode: rainya
• Email:
rainya.mosher@rackspace.com
Rainya is the software development manager for deployment
automation in the Product Infrastructure segment at
Rackspace Hosting, Inc. Rackspace is based in San
Antonio, Texas, United States.
2
Your Presenter
Operating OpenStack @ Rackspace
3. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
“If computers of the kind I have advocated become the computers of the
future, then computing may someday be organized as a public utility just
as the telephone system is a public utility … The computer utility could
become the basis of a new and important industry.” - John McCarthy,
American Computer Scientist, MIT Centennial, 1961
3
Brief History of Computing
Operating OpenStack @ Rackspace
Mainframe
1960s
Virtualization
1990s
Cloud
2000s
Hybrid
2010s
4. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
4
Hybrid Cloud Experience
Operating OpenStack @ Rackspace
The hybrid cloud is the notion that an individual can consume whatever
piece of infrastructure they need at the time that they need it with a
consistent experience so they can focus on their core competency, be it
coding or research or data analysis.
5. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Powered by OpenStack
• Open source software with global
community of over 1,000 contributors
and 80 companies
• OpenStack provides freedom from
proprietary software and vendor lock-
in
• Rackspace Infrastructure includes the
following OpenStack services
• OpenStack Compute
• OpenStack Image Service
• OpenStack Networks
• OpenStack Storage 5
Public Cloud at Rackspace
Operating OpenStack @ Rackspace
7. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Kickstart
• Install hypervisor
onto physical host
Bootstrap
• Prepare hypervisor
and create control
plane nodes to run
OpenStack
services
Maintain
• Update OpenStack
services in control
plane nodes
7
Building an Open Cloud
Operating OpenStack @ Rackspace
The hypervisor is
where the customer
instance / server is
created.
Control plane nodes are
virtual instances that run
the OpenStack services.
8. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Hypvisors (HVs) &
Compute Control Nodes
Hundreds of HVs &
Cell Control Plane Nodes
Thousands of HVs &
Region Control Plane Nodes
Tens of Thousands of HVs &
Control Plane Nodes
Global
Cloud
Region Region
Cell Cell Cell
HV HV HV HV HV HV
Cell Cell
Region
8
Maintain at Cloud Scale
Operating OpenStack @ Rackspace
9. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Code
Package
Deploy
Verify
9
Updating the Control Plane
Operating OpenStack @ Rackspace
During a control plane
update, customer
instances on the
hypervisor stay up and
running.
10. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
10
Code Iterations & Releases
Operating OpenStack @ Rackspace
6-Month OpenStack Release Cycle
60 Days 60 Days 60 Days
11. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Benefits and Challenges of Trunk Deploys
Operating OpenStack @ Rackspace
11
Why We Do It (Benefits)
• Issue Resolution
– Early detection of issues and conflicts
– Shorter feedback loop within the
community
– Faster resolution of issues
• Early Feature Delivery
– Smaller, incremental periodic releases
– More stable release candidates for the
community at end of cycle
Why It’s Hard (Challenges)
• Code Management
– Merge conflicts with local patches
– Disruptive DB migrations
– Service restarts
– Temporary version skew
• Testing
– Devstack-based testing vs testing at
scale
– Rework when issues found in RAX deploy
pipeline
• Process
– CI/CD vs 6-month Releases
– Time to merge upstream patches
15. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
15
Tools to Maintain the Cloud
Operating OpenStack @ Rackspace
• Puppet
– Configuration management for all control plane nodes
– YAML-defined fact files for config variables
– Masterless configuration to prevent bottlenecks at scale
• Ansible
– Playbooks for orchestration of OpenStack service bootstrap and updates
– Playbooks for orchestration of out-of-iteration hotpatches and changes
• Jenkins
– Scripts called to create virtual environment (venv) packages
– Jobs configured for automating the pull and merge of upstream code
– QE test automation triggered after deployments
– Ansible playbooks invoked from jenkins to reduce human error
16. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• The hybrid cloud concept will enable individuals to access the
infrastructure they need when they need it.
• One part of the hybrid cloud portfolio at Rackspace is the public cloud,
powered by OpenStack.
• OpenStack is an open source cloud operating system with distinct
services that work together to enable the creation of cloud servers.
• Rackspace uses a variety of tools and processes to maintain
OpenStack services throughout the OpenStack release cycle.
16
Final Thoughts
Operating OpenStack @ Rackspace
Twitter: @rainyamosher | Email: rainya.mosher@rackspace.com
17. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Backup Slides
Operating OpenStack at Rackspace
17
Many of these backup slides were first presented on 4/16/2013 during the
OpenStack Summit session “Deploying from OpenStack Trunk” and are
included here for reference.
18. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
18
OpenStack Release Cycle
Operating OpenStack @ Rackspace
6-Month OpenStack Release Cycle
new features accepted feature freeze
Planning
Community designs,
discusses, & targets
release cycle
Implementation
Community makes
changes to OpenStack
code & creates new
functionality Pre-Release
Community focuses
on bug fixes, docs,
& testing
Maintenance
Security vulnerability fixes are supported for the last three Releases
Final
Release
19. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
19
Rackspace Iteration Cycle
Operating OpenStack @ Rackspace
Choose Release
Branch
Pull upstream code,
merge with internal
patches, and deploy
to CI environment
for validation
Release Branch
Integration
Increase test coverage on
release branch candidate
to include full end-to-end
integration testing with
Rackspace services
Approve Release
Once testing
passes, release is
approved and
production deploy is
scheduled
60-Day Rackspace Iteration Cycle
code may change release branch frozen
20. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
20
Merge and Branch Strategy
Operating OpenStack @ Rackspace
• Rackspace Development branch is
about 40 patches on top of
OpenStack trunk for internal service
compatability
21. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
21
Scale of Deploy Pipeline
Learning to Scale OpenStack
1,000s of Nodes100s of Nodes10s of NodesDevStack
Dev
Integration
& QA
PreProd Production
Hello. I am Rainya Mosher, the software development manager at Rackspace Hosting, based in San Antonio, TX, USA. My team is in charge of deployment automation for the Product Infrastructure segment, which is the foundation of our public cloud offering, powered by OpenStack. You can reach out to me after today on twitter or if you are on the IRC freenode network, you can find me under the handle “rainya.” I am very excited to be here today to share with you information about OpenStack, how we are Rackspace use the software to run the largest open public cloud, and provide some insight into the hybrid cloud use case and story. When you leave today’s session, my intent is that you have a few key takeaway, namely: What is the hybrid cloud? Why choose OpenStack? How does Rackspace operate OpenStack for their public cloud.
In the early 1960s, computer scientists working on mainframes envisioned a future where computing would be available using a utility model. The mainframe model evolved in the 90s to the modern data center. Individual servers began filling the old mainframe rooms as the growth of internet bought a surge of demand for computing. In the late 90s and early 2000s, virtualization became more common and easier to accomplish. Businesses realized they could extend a single dedicated server to provided multiple purposes for different projects and control costs and the time to acquire new hardware. In the mid- to late-2000s, the first cloud offerings began to enter the market. Now, more than 50 years since those early scientist's vision for computing, OpenStack provides open source software that enables more computing options than ever before.REF: http://siliconangle.com/blog/2014/03/05/the-evolution-of-the-data-center-timeline-from-the-mainframe-to-the-cloud-tc0114/
When you look at the history of computing in the last 50 years, the evolution from mainframe to dedicated servers to virtualization to public cloud computing, the transitions between these different technologies are not the same for everyone. Each offering has strengths and every user of computing may have a slightly different use case to meet their needs. When we talk about the “hybrid cloud” at Rackspace, it goes beyond a single product that can be packaged up and delivered to everyone. The hybrid cloud is the idea that an individual can consume whatever piece of infrastructure they need at the time that they need it in an overall experience that is consistent, accessible, and seamless. This story enables the individual to focus on their core competency – be it to write code, analyze data, or conduct research – while letting the hybrid cloud experience handle their infrastructure needs. With a hybrid cloud experience, you have access to the performance of a single dedicated server, the option to virtualize a group of servers for your own private use, and the economies of public cloud servers all in a single, integrated portfolio offering. That is the promise and potential of a hybrid cloud solution, and Rackspace is defining the cloud as open.
Within the context of hybrid cloud story, the public cloud powered by OpenStack has proven to be a unique challenge for Rackspace. Offering infrastructure as a service is a challenge in and of itself, but OpenStack software is written by thousands of contributors located all over the world. As a vendor, Rackspace is a member of the community and does not directly own or control the software we are consuming and deploying to our public infrastructure. We work with the community and collaborate with dozens of other companies and hundreds of individuals on a regular basis.
TheOpenStack Cloud Operating System is a collection of distinct services that are accessed through APIs. When installed on top of hardware, it creates a flexible and powerful cloud offering in which virtual servers can be created, modified, and deleted through a few clicks on the dashboard or through a direct call to the API.REF: http://www.openstack.org/software/
The Rackspace cloud spans multiple regions, which are collections of physical hosts located in the same geographic area. A region is made of one or more cells, each of which is made of several hundred hypervisors installed on physical hosts. As of March 2014, Rackspace has 6 regions: 3 in North America, 1 in Australia, 1 in the United Kingdom, and 1 in Hong Kong. For Rackspace, cloud scale means maintaining OpenStack services on hundreds of thousands of instances (physical and virtual) in an expanding, multi-region global cloud. In addition to the physical hosts, each hypervisor is controlled by a compute service. At the cell and region levels, OpenStack services are installed on virtual instances to form the region and cell control plane.
When we talk about updating and deploying to the public cloud, we most frequently update the code that runs the OpenStack services. The hypervisor software needs patching from time-to-time, but this is a far less frequent event than the OpenStack changes. The basic strategy used to deploy updates to OpenStack onto our public cloud is simple. Rackspace pulls down OpenStack code from the upstream trunk branch, packages the code with configurations and integration points for internal systems, distribute the package to all the nodes, execute the code in the package, and then verify that it works through automated testing. The concept is simple, but the optimal execution is an ongoing learning opportunity.
OpenStack releases in 6 month cycles. The OpenStack release cycle is given a code name (for example, “Icehouse” or “Grizzly”) and the community targets a specific set of functionality to deliver by the end of the cycle. Rackspace pulls more frequently from OpenStack upstream code, about every 60 days, and will do one or more deploys to production from that iteration.
OpenStack cycles on a
Check out Wednesday’s session at 430p on how Rackspace is “Learning to Scale OpenStack” for the story behind the most recent internal release branch!