![Capacity Management ,[object Object],John Allspaw Operations Engineering](https://image.slidesharecdn.com/cfakepathcapacitymanagementpresentation-090922010104-phpapp01/85/capacity-management-from-flickr-1-320.jpg)
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Ähnlich wie Capacity Management from Flickr
Ähnlich wie Capacity Management from Flickr (20)
Mehr von xlight
Mehr von xlight (17)
Capacity Management from Flickr
- 2. the book I’m writing
- 3. ???
- 4. Rules of Thumb Planning/Forecasting Stupid Capacity Tricks (with some Flickr statistics sprinkled in)
- 13. 2400 photos per minute being uploaded right NOW (Tuesday afternoon)
- 14. Ceilings the most amount of “work” your resources will allow before degradation or failure
- 16. Find your ceilings what you have left The End
- 17. Use real live production data to find ceilings Production: “it’s like a lab, but bigger!”
- 18. Like: database ceilings replication lag: bad!
- 19. Ceilings waiting on disk too much sustained disk I/O wait for >40% creates slave lag* *for us, YMMV
- 20. 35,000 photo requests per second on a Tuesday peak
- 21. Safety Factors
- 22. Safety Factors Ceiling * Factor of Safety = UR LIMITZ
- 24. “ safe” ceiling @85% CPU Safety Factors 85% total CPU = ~76 busy apache procs what you have left
- 25. Safety Factors Yahoo Front Page link to Chinese NewYear Photos (photo requests/second) (8% spike)
- 26. Forecasting
- 27. Forecasting Fictional Example: webservers
- 28. Forecasting Fictional example: 15 webservers. 1 week. peak of the week
- 29. ...bigger sample, 6 weeks....isolate the peaks... Forecasting
- 30. ...”Add a Trendline” with some decent correlation... Forecasting now not too shabby
- 31. Forecasting 15 servers @76 busy apache proc limit = 1140 total procs when is this? this will tell you when it is ceiling what you have left
- 32. Forecasting (week #10, duh) (1140-726) / 42.751 = 9.68
- 34. Forecasting Fictional Example: storage consumption
- 35. Forecasting Automation actual flickr storage consumption from early 2005, in GB (ceiling is fictional) this will tell you when this is
- 36. Forecasting Automation cmd line script output jallspaw:~]$cfityk ./fit-storage.fit 1> # Fityk script. Fityk version: 0.8.2 2> @0 < '/home/jallspaw/storage-consumption.xy' 15 points. No explicit std. dev. Set as sqrt(y) 3> guess Quadratic New function %_1 was created. 4> fit Initial values: lambda=0.001 WSSR=464.564 #1: WSSR=0.90162 lambda=0.0001 d(WSSR)=-463.663 (99.8059%) #2: WSSR=0.736787 lambda=1e-05 d(WSSR)=-0.164833 (18.2818%) #3: WSSR=0.736763 lambda=1e-06 d(WSSR)=-2.45151e-05 (0.00332729%) #4: WSSR=0.736763 lambda=1e-07 d(WSSR)=-3.84524e-11 (5.21909e-09%) Fit converged. Better fit found (WSSR = 0.736763, was 464.564, -99.8414%). 5> info formula in @0 # storage-consumption 14147.4+146.657*x+0.786854*x^2 6> quit bye...
- 37. Forecasting Automation (SAME) fityk gave: y = 0.786854x 2 + 146.657x + 14147.4 ( R 2 = 99.84) Excel gave: y = 0.7675x 2 + 146.96x + 14147.3 ( R 2 = 99.84)
- 39. High and Low Water Marks alert if higher alert if lower Per server, squid requests per second
- 40. A good dashboard looks something like... (yes, fictional numbers) type # limit/box ceiling units limit (total) current (peak) % peak Est days left www 20 80 busy procs 1600 1000 62.50% 36 shard db 20 40 I/O wait 800 220 27.50% 120 squid 18 950 req/sec 17,100 11,400 66.67% 48
- 42. Diagonal Scaling example: image processing 4 cores 8 cores (about the same CPU “usage” per box)
- 43. ~45 images/min @ peak ~140 images/min @ peak (same CPU usage, but ~3x more work) “ processing” means making 4 sizes from originals Diagonal Scaling example: image processing throughput
- 44. Diagonal Scaling example: image processing 3008.4 Watts 1036.8 Watts went from: 23 Dell PE860s 8 HP DL140 G3s to: 1035 photos/min 1120 photos/min ( 75% faster, even) 23U rack 8U rack !!!
- 45. 3.52 terabytes will be consumed today (on a Tuesday)
- 46. 2nd Order Effects (beware the wandering bottleneck) running hot, so add more
- 47. 2nd Order Effects (beware the wandering bottleneck) running great now, so more traffic! now these run hot
- 49. Stupid Capacity Tricks quick and dirty management DSH http://freshmeat.net/projects/dsh [root@netmon101 ~]# cat group.of.servers www100 www118 dbcontacts3 admin1 admin2
- 50. Stupid Capacity Tricks quick and dirty management [root@netmon101 ~]# dsh -N group.of.servers dsh> date executing 'date' www100: Mon Jun 23 14:14:53 UTC 2008 www118: Mon Jun 23 14:14:53 UTC 2008 dbcontacts3: Mon Jun 23 07:14:53 PDT 2008 admin1: Mon Jun 23 14:14:53 UTC 2008 admin2: Mon Jun 23 14:14:53 UTC 2008 dsh>
- 52. Stupid Capacity Tricks Turn Stuff OFF uploads (photo) uploads (video) uploads by email various API things various mobile things various search things etc., etc.
- 55. thanks http://flickr.com/photos/bondidwhat/402089763/ http://flickr.com/photos/74876632@N00/2394833962/ http://flickr.com/photos/42311564@N00/220394633/ http://flickr.com/photos/unloveable/2422483859/ http://flickr.com/photos/absolutwade/149702085/ http://flickr.com/photos/krawiec/521836276/ http://flickr.com/photos/eschipul/1560875648/ http://flickr.com/photos/library_of_congress/2179060841/ http://flickr.com/photos/jekkyl/511187885/ http://flickr.com/photos/ab8wn/368021672/ http://flickr.com/photos/jaxxon/165559708/ http://flickr.com/photos/sparktography/75499095/
- 56. We’re Hiring! flickr.com/jobs Come see me!
- 57. questions?
Hinweis der Redaktion
- Only two more chapters to go. :)
- How many of you manage servers for your site? How many of you know how many servers you have? (databases, webservers, etc.) How many of you collect metrics for all of your capacity resources?
- I’ll be repeating some concepts that I’ve talked about in other presentations on the same topic... 1. Planning 2. Manage 3. Stupid Catastrophe Tricks with some random statistics from Flickr sprinkled throughout
- By “capacity problems” I mean NORMAL capacity trends, not spiking ones. By edge cases, I mean usage patterns that exist outside the realm of normal operation. Examples: users with 60,000 tags on 20 photos (not possible anymore)...search API calls with 60 ORs (not possible anymore)
- The High Performance Computing industry has created a lot of tools and deployment philosophies that web operations can learn from.
- It *DOESN’T* matter which tool you use, as long as it can satisfy these criteria.
- Knowing what system resources mean in terms of application usage puts the whole capacity shebang into context. Another example would be: Max QPS for a MySQL server = X users, Y photos.
- (that’s about 44 per second.)
- Artificial stress testing is rarely good for testing real capacity ceilings. It’s great for comparing two different hardware platforms, tho.
- How many of you know how many QPS your MySQL machines can do without degrading or failing? (slave lag, anyone?)
- Find ceilings by measuring *real* data from production. WHY?? 1. Development “cycles” are TIGHT, so code changes, so load characteristics change all the time. (sometimes in big ways) 2. Edge cases get shown in production, not in my imagination. (60k tags on 100 photos?) 3. Too much time wasted on artificial test setups to get accuracy that doesn’t matter.
- Sometimes you don’t have to increase load artificially, you bump up against the limits naturally...
- So, our ceiling is disk I/O wait, and it’s around 30-40% that we want to stay under... WHY does this happen? I don’t know, and I don’t care, not right now....
- Squid requests per second, at peak, on a Tuesday.
- Structural and mechanical engineering use a Factor of Safety (FoS) when designing components that experience load, both stress and strain: bridges, airbags, buildings, seatbelts, toasters. So should we, as web operations.
- Whether you express it as a “reserve”, or “overhead”, or some fraction/percentage of your ultimate limit, you should know what these are for ALL of your resources. Civil, mechanical engineers use them when designing bridges, airbags, buildings, seatbelts, toasters. So should we.
- Being a multi-core box, I know I can get up to 95% user CPU without performance degradation, but I don’t want to get that far. We’ll plan to have our webservers reach 85% so we have wiggle room for sporadic spikes above that.
- Being a multi-core box, I know I can get up to 95% user CPU without performance degradation, but I don’t want to get that far. We’ll plan to have our webservers reach 85% so we have wiggle room for sporadic spikes above that. Why 85%? Because that is what history has told me I could see spikes of (15%)
- Being a multi-core box, I know I can get up to 95% user CPU without performance degradation, but I don’t want to get that far. We’ll plan to have our webservers reach 85% so we have wiggle room for sporadic spikes above that.
- Forecasting capacity means making educated guesses about the future by using data from the past. Throw in any knowledge you have about: - timing of feature launches - seasonal differences - new hardware deployment
- We’ll just go through a simple example of using extrapolation and curve-fitting to make a prediction on how data (capacity metrics) will change in the future. THERE IS NO SUCH THING AS PREDICTING THE FUTURE.
- RRDtool data, put into Excel. Need a bigger sample than 1 week, not enough peaks....let’s try 6 weeks...
- ...try 6 weeks of data...
- “ Add a Trendline” is a feature in Excel. Note the # of weeks at the bottom. The R-squared number is the “coefficient of determination” which indicates how good of a “fit” the equation is to the data.
- This is a linear equation given for the curve-fitting function.
- Using Excel is time-consuming, you should be able to automate this so you can keep tabs on it easier. fityk has a command-line version, cfityk.
- Same drill with Excel.
- The same! Yay!
- High and low-water marks
- ADD THE ESTIMATED TIME LEFT ON HERE...
- Yay! Savings all around!
- Terabytes will be consumed today, not including video.
- Watch out for 2nd-order effects when deploying new/faster machines. When throttles are opened, the dam can get moved down the river.
- Watch out for 2nd-order effects when deploying new/faster machines. When throttles are opened, the dam can get moved down the river. Artur mentioned that faster pages = more traffic...we see the same thing.
- Some well-known tips and tricks for when the shit hits the fan.
- Before capistrano, before puppet, there was dsh. Quick and dirty.
- Running a command on any arbitrary number of hosts, interactively. Not revolutionary, but useful.
- Better to be mostly up than down for features that aren’t used much.
- Hosting is like $7.95 a month for a blog. Spend the cash.
- We have put squid in front of our search cluster and cached aggressively when we ran close to capacity.