Open stackconference deployingopenstack-final-2010-11-10

Hinweis der Redaktion

2. <Photo of Unplugging Compute nodes> This is a photo/video of one of the swift clusters we’ve stood up. Just last week. Visiting Data Center. Half-jokingly asked a colleague, what would happen... So he reached over and I stopped him to take a video, just because I thought it was so cool. Storage is fragile. We’ve all had the pain. Loosing your company’s data, or loss of reputation as a service provider or vendor. Or the data becomes unavailable for some reason and it takes a long time for recovery, causing extensive downtime. The consequences of data loss or corruption is extremely high. That is what was so revolutionary in my eye about having the _confidence_ of being able to casually walk up and pull the plug. What I’m going to talk about is deploying the Open Stack Object Storage project Swift. - Why we think it’s going to be a huge force - I’m going to share our experiences with swift. Standing up the hardware and software. - Finally, what it takes to go live with Swift “as-a-service”
3. What we do help telcos and service providers build the most competitive clouds in the world; We build large scale, highly competitive Public Clouds for Global 300 Telcos and service providers
4. <Diagram objects over HTTP, Containers & Data> <Rings> <Swift Processes> Just to baseline. Swift is the project name for the OpenStack Object Storage. It’s a storage service that is accessed via an API. Via the api you can create containers and PUT objects (data) into them. ***That’s about it. It’s not a blocks. It’s not a filesystem. Needs an ecosystem
5. Through this talk I’ll be mostly address everything around Swift. This isn’t going to be a talk about the internals or externals of Swift. There is plentiful and ever-increasing documentation covering the core Swift mechanisms. And to those implementors and operators who are in the room today, thanks for open-sourcing your baby. We’ve enjoyed baby-sitting.
6. I first want to talk about the ecosystem. But to give context, let me go over a brief history.
7. This whole thing got started in 2006 when Amazon launched S3, Simple Storage Service. And if everyone can re-wind in their heads back to 2006 when S3 came out. It was a strange animal. It made sense, but it was kind-of a novelty. - No SLA - Paranoid about “outsourcing data” But we got used to it. It started with backup tools. When new applications were developed, application developers became really keen on using S3 - Didn’t need to go out and buy a storage array -- And no upfront cost -- They didn’t need to guess at how much they were going to use For these reasons, S3 became more and more baked into the tools that developers were using. - Ruby on Rails (paperclip) - Zend PHP - Hadoop (map-reduce) /* At the Ruby on Rails deployment company, Engine Yard (which is where I was before Cloudscaling). - In 2008, we developed an AWS-based deployment platform. - The old deployment system was on in-house hardware. - One of the features was a clustered, posix-compliant filesystem with GFS. You could have many virtual machines all connecting to the same volume in a relatively-sane way. - In the move to AWS, we couldn’t provide the same type of storage system. - But because S3 had permeated into the tools developers were using, it wasn’t an issue. */
8. In 2008, Mosso, a subsidiary of Rackspace, launched its own Object Storage system called CloudFS, now called Rackspace Cloud Files.
9. And, of course, over this summer the project was open-sourced as part of the OpenStack project. And that brings us to present day. ***So here we are: - Object Storage is a big market. With two big players in the space. - Rackspace has open-sourced their implementation - This sets the stage for more deployments going forward
10. *** Ecosystem of tools is emerging, but we’re very early in this cycle for Swift. (recycle-like logo) There is opportunity for a positive feedback loop to happen. - The better the tools, the more compelling it will be for more implementations of Swift to come online. - The more implementations of swift, the more attractive it becomes to build great tooling around it. Our customers need a cloud storage product to offer. They have a choice to make -- bring up something that is compatible with S3 APIs or use Swift and the OpenStack API. - While the volume of tooling out there favors S3, the stage isn’t quite set to deploy your own S3-compatible system. -- The S3-based solutions have to _emulate_ an implementation. - Swift offers a much higher degree of compatibility with an existing Rackspace Cloud Files tool ecosystem. -- with Swift you can use the _actual_ implementation. So speaking of tooling... Here’s a report:
11. - Cyberduck: Mac-based GUI. Needed patching. Author has pulled in changes in latest build. - Fog: Ruby, multi-cloud library. Needed patching. Wesley Berry has pulled in our changes. (But it still references Rackspace in its interface.) - Cloudfuse: Implements a FUSE-based filesystem. Needed patching. Still working on getting changes merged. - Rackspace’s libraries: Again, some needed patching to support a Swift cluster. Very quick response to folding in patches. So if you’re thinking of deploying an openstack object storage, there is a reasonable body of open-source tools and client libraries. So that brings us to getting this up and running with service providers. /* What’s missing from this list are cloud management services. I would love to talk with those who are providing OpenStack support. I know what it’s going to take to provide the real motivation is a large potential customer base, that’s going to show up when there are running, public, implementations of OpenStack. */
12. - Cyberduck: Mac-based GUI. Needed patching. Author has pulled in changes in latest build. - Fog: Ruby, multi-cloud library. Needed patching. Wesley Berry has pulled in our changes. (But it still references Rackspace in its interface.) - Cloudfuse: Implements a FUSE-based filesystem. Needed patching. Still working on getting changes merged. - Rackspace’s libraries: Again, some needed patching to support a Swift cluster. Very quick response to folding in patches. So if you’re thinking of deploying an openstack object storage, there is a reasonable body of open-source tools and client libraries. So that brings us to getting this up and running with service providers. /* What’s missing from this list are cloud management services. I would love to talk with those who are providing OpenStack support. I know what it’s going to take to provide the real motivation is a large potential customer base, that’s going to show up when there are running, public, implementations of OpenStack. */
13. - Cyberduck: Mac-based GUI. Needed patching. Author has pulled in changes in latest build. - Fog: Ruby, multi-cloud library. Needed patching. Wesley Berry has pulled in our changes. (But it still references Rackspace in its interface.) - Cloudfuse: Implements a FUSE-based filesystem. Needed patching. Still working on getting changes merged. - Rackspace’s libraries: Again, some needed patching to support a Swift cluster. Very quick response to folding in patches. So if you’re thinking of deploying an openstack object storage, there is a reasonable body of open-source tools and client libraries. So that brings us to getting this up and running with service providers. /* What’s missing from this list are cloud management services. I would love to talk with those who are providing OpenStack support. I know what it’s going to take to provide the real motivation is a large potential customer base, that’s going to show up when there are running, public, implementations of OpenStack. */
14. - Cyberduck: Mac-based GUI. Needed patching. Author has pulled in changes in latest build. - Fog: Ruby, multi-cloud library. Needed patching. Wesley Berry has pulled in our changes. (But it still references Rackspace in its interface.) - Cloudfuse: Implements a FUSE-based filesystem. Needed patching. Still working on getting changes merged. - Rackspace’s libraries: Again, some needed patching to support a Swift cluster. Very quick response to folding in patches. So if you’re thinking of deploying an openstack object storage, there is a reasonable body of open-source tools and client libraries. So that brings us to getting this up and running with service providers. /* What’s missing from this list are cloud management services. I would love to talk with those who are providing OpenStack support. I know what it’s going to take to provide the real motivation is a large potential customer base, that’s going to show up when there are running, public, implementations of OpenStack. */
16. We’ve been working with OpenStack Object Storage since it came out in late July. Because Object Storage deployments makes a lot of sense for our clients. Having a way to conveniently host data is a very good thing for a service provider. 1) It’s good to have data close to the resources they're using (with compute, or content distribution, for example) 2) It’s awfully convenient to have a place for data - whether that’s for backups - media assets. 3) Provide in-house tools to build an application around - anywhere you are using S3 / CloudFiles during application development Now, if you want, you can build and run your own cluster with the - performance characteristics that you choose - where you choose, - with the security model you want In less than 2 months, we had a reasonably-sized - 100TB deployment that allowed us to get a good idea of how this system would behave. - We’re in process of standing up a larger, multi-rack system.
17. Here is what it looked like.
18. What we learned was this: Swift alone doesn’t equal a whole service. - The Swift code provides the kernel of functionality - But there is much software to write, systems to integrate, hardware choices and design decisions to make. The components are - Data Center Concerns - Networking - Hardware Choices - Access Layer - Installing Let’s first dive into the core of what the choices we made to get Swift up and running.
19. Early on in the standup process. Swift has this concept of Zones. Zones designed to be physical segmentation of data. - Racks, power sources, fire sprinkler heads, physical buildings - Boundaries that can isolate physical failures The standard configuration is to have data replicated across three zones. So initially we were thinking, well, let’s just create three zones. But the problem is, if one of the zones catastrophically fails, there isn’t another zone for the data to move into. Now, we wanted to be able to tolerate a whole rack failure and still have a place for data to land. We choose to go with 5 zones. If we had two total zone failures, and there was enough capacity remaining in the system, we could run with. With our small deployment where we had one machine == one zone Our larger deployment will have one rack == one zone
20. The Data Center One of the 1st things to note is power density -or- space requirements of the system - mechanical things tend to draw a lot of power. - In our configuration, to utilize a full-rack in a data center, we had to live in a “high-density” neighborhood of the data center. - Our configuration runs with 10 4u object stores ran 7kw a rack - That’s 370 drives per rack. - Careful when powering up whole racks - Plan accordingly - The other option for us was to “go wide” and run 1/2 racks, where we would use more space.
21. The Networking We took a 2-tier approach. It starts out with a pair of redundant aggregation switches. A single switch would be a single point of failure. All requests go through the “Access Layer” that connect directly to the aggregation switches at 10GbE. - The access layer contains proxy servers, authentication, load balancers, etc. Each rack has a single switch that is connected via 10GbE to the aggregation layer. - We went with single as we plan on being able to handle single rack failures. And it tapers down to 1GbE to an individual object store from the top-of-rack switches.
22. The object stores: Beefy! - 48GB RAM - 36 2TB - Newish Xeon These are not just a bunch of disks! The system has a lot of work to do. Lots of metadata to keep in memory. ***Lots of processes needed to be ran to handle the parallelism. Commodity, but enterprise quality gear. Enterprise drives. /* SATA not SAS */
23. Access Layer AKA “Proxy Servers” - Needed to be beefier than we originally thought. - We settled on a newish Xeon w/ 24 GB RAM - 10 GbE Our original deployment bottlenecked here.
24. Huge caveat here. Different usage patterns will dramatically very Architecture, Hardware and Networking mix - Archive -- Occasionally tar-up a wad of data and park it there. -- Much lower burden on the entire system. - Trusted networks -- Don’t need SSL, but wants lots of bandwidth - Lots of little puts and gets -- Proxy servers will need to handle the SSL load of many requests Although I just outlined some hardware here. It’s not an exact recipe to follow.
25. How the system is used has a huge impact on what to build. The next uncharted territory we had to figure out was what we’re calling the ‘Access Layer’ You need to run the swift proxy-server process, which routes requests to the correct location in the object stores (using the ring) In addition: - But you’re also likely to want to handle SSL termination - And to load balance across the servers running the proxy processes - We’ve also heard about using a commercial load balancer here as well - HA Solution Many options here: - What we’re running is a round-robin DNS w/ Nginx terminating SSL directly in front of the proxy. - What we’re likely to end up with is running an HA Proxy configuration sharing an IP using VRRP for HA, dumping straight into the proxy processes Being pragmatic, there are other services that need a home as well - Authentication - Portal Sticking them in the access layer can make a whole lot of sense.
26. So what did we learn about this configuration? - High data rates typically require a lot of parallel accesses, there is often significant per-access latency (10s of ms, 1000x what a san or local storage device might show) - But the tradeoff is a ton of objects can be retrieved and written at the same time. Having a lot of RAM in aggregate helps. It makes the metadata costs for accessing lots of objects manageable.- We grossly undersized the proxy servers. We did a ton of testing: - Bitrot, bad disk, (happened to have) bad DIMM, failed node, failed zone, failed switch This is an embarrassing story, but it’s worth telling. We even had one of the zones down for two days right early on. Nobody noticed. We noticed when we re-ran some benchmarks and some of the peak performance numbers weren't what they were on a previous run.
27. When you buy a new car today. You open up the hood and there is a black plastic cover over the engine. To diagnose, you plug the car into a computer and there are sensors all over the place to tell you what is wrong and how to fix it. We need to be mindful to the fact that this is just one of many systems that they operate. As much as possible, we’d like to hand them a whole working product that tells them how to operate and maintain it. For there to be a large number of swift deployments, the system needs to have handles so that operators can maintain their system. I think this will be a key trait for swfit to go mainstream. Here is what we’ve assembled so far.
28. <virtual appliance> We built an installer. Installing: - To operate at any reasonable scale, consistency in configuration is hugely important. - So we automate the crap out of everything. - This installer (which runs as a virtual appliance) brings bare metal to fully-installed node It is running a PXE server, so machines can ask for a new OS Our configuration management tool of choice is Chef, so the installer is running chef-server The net effect is that an operator can use a cli tool and punch in the mac address & what role the machine should be. And this system will take it from factory fresh, to a fully-installed swift node.
29. A big web application Software System. Automating the install is not enough. A couple of years ago, I was going through the process of choosing a storage vendor. - Each sales team that I met with said the same thing: “We’re a software company” - Now, I think they said that because there is more margin in being a software company. - But in truth, they’re right. There is a heck of a lot in the software that drives these systems. We treat the entire system as if it were a big web application that is undergoing active development. Change happens. We are continuously adding features or needing to respond to operational issues We pull from our dev ops roots to build tooling that is capable of driving frequent change into the system. We must be able to perform upgrades in the field with confidence. Here is what we have put together for Swift:
30. The first thing we put together was to automate the Swift ‘all-in-one’ so that we could have a one button install of a local development environment. - This brought the system up from a fresh ubuntu install on a single VM somewhere. - Perfect to do local development on. - This we have up on github if you want to check it out. The next step was to model-out a simulated deployment of how we were to configure a production environment. - In a lab environment, we recreate the production environment with virtual machines. - And remember that ‘virtual appliance’ installer that does the provisioning? - We use that same tool so that we can have a self-contained, simulated build of the entire system. Next, we have our small-ish environment where we have one physical machine per zone. - We can use that environment as a pre-production for changes.
31. Finally, to vet the software changes as they are being made, we have a continuous integration setup with Hudson. When code is checked-in, we can rebuild in the lab. This is the system with a bunch of VMs. Tests can be ran against that system and report any errors that crop up. /* A full-rebuild takes us about 45 minutes. */ All these systems put together gives us the confidence that the system that we’ve put together will run well and upgrade smoothly. /* Time-to-live is a big deal for us. We aggressively pursue automation because it enables a repeatable deployment process at scale. Yes, a lot of time was spent automating the install of Swift. But we built these systems because we ended up doing a lot of software development to integrate Swift and to make it run, ‘as a service’. */
32. Finally, operators need to know when the system needs work. We’re baking in monitoring agents as part of the system install. Because we have lot cost install tools, we are favoring replacement over repair. - For example if a boot volume goes bad, or bad DIMMs. - Basically anything that doesn’t involve swapping out a drive. - It’s easier to replace the component, have the system rebalance itself and add new equpment. - Integrate w/ existing NOC systems -- nobody will act upon an alert that isn’t seen
33. OpenStack Object Storage: As a Service Okay Great. We had our swift cluster up and running, now we can sell some storage... right! sort or. What’s left? All those things that enable the system to be sold “as a service” Transition slide to next subjects: - Utilization & Billing integration - Authentication - Portal - Logging
34. Billing This is a huge topic! We could have an entire talk on this topic alone. Collect and store those metrics. Not baked into Swift -- yet. - We had to collaborate on some of the utilization code for swift - But we have to tune it for our customers billing plan Utilization - must compute storage ‘averages’ measuring on some interval you feel comfortable with - Decide if you are going to keep track of small obj / num requests Do you bill for requests? or make small files penal? charge for internal access? At one company I was a heavy user of Amazon S3 -- they called me up out of the blue and told me to re-architect our application because we were creating too many ‘Buckets’ (Folders). Apparently we were causing excessive load. He thought he was being helpful saying “you will save $700 a month!” I appreciated the sentiment to try to save me money, but the cost of re-architecting the app would be more than a reasonable payback period in that context. The excessive load was their problem. I was okay paying an extra $700 a month. The moral of the story is to be okay with the usage you price for.
35. Welcome to the world of consumption-pricing vs. capacity pricing. Making a dinner reservation at the hippest restaurant in town. They ask for your credit-card. Why? Because if you don’t show up, they still need to charge you for the meal you didn’t eat. They’re practicing capacity pricing. They have a fixed number of seats that they need to fill. This same practice is the standard in our industry. When you go buy a bit of storage equipment, you buy based on its capacity. You can’t say to a vendor... would you mind installing 100 TB of storage, but I’ll only pay for 50... because I only plan on using 50TB on _average_. They would laugh you out the door! When you go to buy bandwidth, you are charged 95-percentile. You pay for bandwidth that goes unused because you’re paying for the capacity to be available with some wiggle-room for extra-ordinary bursts. So service providers are having to figure out how to deal with this. It’s a bigger deal at a smaller scale. A single customer could but-in and consume a large amount of a cluster on a percentage basis. The averages even out at a larger scale.
36. - This is a multi-tenant system - The Authentication server that comes with Swift is not intended to be used by a service provider. It is a placeholder with significant performance issues, missing features and bugs. And that’s okay! Because if you are a service provider, you will certainly be integrating into an existing customer authentication system. - However, you will need to setup a few things in between the core storage cluster and your authentication system. - Clients (as in libraries/tools/services) expect to hit an auth server so that they can get 1) a token that can be used with subsequent requests and 2) a URL of the storage system they should hit. - This “Authentication Middleware” must also be capable of getting a token from the storage cluster to verify that it’s still good. - So here is how the process works: Client makes request to Auth server and gets a token + URL, Client makes API calls to the storage cluster with the Token, The storage cluster asks the Auth Server if the token is good.
37. On top of the authentication system, we’re building a system to create API keys. - API keys are useful because it allows clients (which are code libraries, desktop clients and other services) to authenticate without putting password. - What you don’t want to do is have your username / password checked-in to a source repository, or hand it over to a 3rd-party service, like a scalr, rightscale, etc. - So you’ll need to build an additional service that creates and manages API Keys.
38. - Even if all the users are we come across are programmers... it’s still nice to be able to login and poke around and do stuff with a mouse. - Look at a bill, manipulate data, etc. /* - But that’s not what we’re typically seeing with our ‘as a service’ deployments. They want the data to get there so something can be done with it! - So often-times the customer portal work _is_ the application itself. */
39. Internal Dashboards - Manage accounts
40. Be-all-end all Getting the mechanics of getting / storing and dishing-out data should be boring. - If we can provide a system with a strong ecosystem of tooling, that is manageable, easy to operate -- It can serve as a platform, a jumping-off point for the rest of the services. - Where the value is, is what is done with the data. - That’s where the high-margin products are.