Obvious and Non-Obvious Scalability Issues: Spotify Learnings
•
6 likes•4,205 views
The document summarizes lessons learned by Spotify about scaling infrastructure and operations. Some key points include: starting with letting experts handle data centers when small, streamlining procurement processes, treating capacity in standardized "pods", focusing infrastructure teams on platforms rather than individual services, implementing automated processes for configuration, provisioning and monitoring, and having individual product teams take on operational responsibilities for their own services with guidance from infrastructure teams. The presentation also covers specific scaling challenges faced with storage, networking, and resilience strategies like retry policies and load shedding.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Obvious and Non-Obvious Scalability Issues: Spotify Learnings
Similar to Obvious and Non-Obvious Scalability Issues: Spotify Learnings (20)
Recently uploaded
Recently uploaded (20)
Obvious and Non-Obvious Scalability Issues: Spotify Learnings
- 1. Obvious and Non-Obvious Scalability Issues: Spotify Learnings David Poblador i Garcia @davidpoblador ! ! BcnDevCon13 November 12, 2013
- 3. 2011
- 4. 2011
- 5. 2013
- 6. One order of magnitude bigger in some dimensions
- 7. 1.000.000.000 playlists 400M two years ago 2M new playlists every day
- 8. 6000+ servers 1300 two years ago 20 in 2008
- 9. Available in 32 markets 12 two years ago
- 10. Two years ago less than 10 people in OPSbigger now + Infra more than 10 times
- 11. 4 Data Centers
- 12. More than 20M songs Adding 20K every day
- 13. More than 24M active users 6M paying subscribers
- 14. More than 50 teams building products & features
- 18. 1 Scaling Data Centers Admit that when you are small, there is someone better than you at building datacenters
- 20. 2 Scaling Data Centers Streamline your procurement process
- 22. 3 Scaling Data Centers Have a “unit of capacity” ! We call it POD
- 24. 4 Scaling Data Centers Data Centers are being commoditized ! Chances are that only a few players will deploy DCs in the future. ! Keep an eye on that. Might make sense for your needs
- 28. know your limits Scaling your backend
- 29. examples User AP 60K users 5000 reqs/second AP backend service node AP backend service node User … User Scaling your backend AP
- 30. Do not try to be ‘too smart’
- 31. DNS à la Spotify Do not try to be ‘too smart’
- 32. Error Reporting DHT ring lookup Service Discovery User Distribution Do not try to be ‘too smart’
- 33. User AP DNS GeoIP magic AP User … User Do not try to be ‘too smart’ AP AP
- 34. Do not try to be ‘too smart’
- 35. 8.8.8.8 Do not try to be ‘too smart’
- 36. User AP AP User … User Do not try to be ‘too smart’ AP AP 8.8.8.8
- 37. Storage Devices
- 38. RAM AP 5000 reqs/second AP backend service node AP backend service node AP Storage Devices
- 39. Does not fit in RAM anymore AP ? reqs/second AP big backend service node AP big backend service node AP Storage Devices
- 40. Hard Drives 200 IOPS Storage Devices
- 41. AP ? reqs/second AP big backend service node AP big backend service node AP Storage Devices
- 43. Fusion IO 250,000 IOPS Storage Devices
- 44. Page Cache
- 45. Example AP 5000 reqs/second AP backend service node AP backend service node ! RAM: 32 GB OS RAM: 2 GB ! AP Page Cache Songs: 10M Index size: 10 GB
- 46. AP AP AP AP Page Cache backend service node backend service node Increase in data (songs…) ! Index: approx 13 GB
- 47. Page Cache
- 49. Retry (not much) Back Off Fail Fast Degrade Gracefully
- 50. User AP AP backend service AP backend service User … User AP Retry (not much). Back Off. Fail Fast. Degrade Gracefully
- 51. DDoS’d by your clients 5000 conns/sec User Retry (not much). Back Off. Fail Fast. Degrade Gracefully AP
- 52. Exponential Back Off Retry 5000 conns/sec User Retry (not much). Back Off. Fail Fast. Degrade Gracefully AP
- 53. Fail Fast User AP AP backend service AP backend service User … User AP Retry (not much). Back Off. Fail Fast. Degrade Gracefully
- 54. Degrade gracefully User Acceptable Behaviour AP AP backend service AP backend service User … User AP Retry (not much). Back Off. Fail Fast. Degrade Gracefully
- 55. Test in real world conditions
- 56. Use your most valuable asset Start by sending X% of users to X% of your servers Test in real world conditions
- 57. Automate
- 60. Take a self service approach everywhere
- 61. Configuration Management Databases and Storage Provisioning of Servers Service Discovery Load Balancing Monitoring … Take a self service approach everywhere
- 64. 1 Scaling Operations Start having teams carry operational responsibility for their own services, including on-call duties for the systems they own
- 66. 2 Scaling Operations Infrastructure and Operations provide expert guidance/help on how to run service(s) teams own in production (and everywhere else)
- 68. 3 Scaling Operations Infrastructure and Operations focus the effort on building and extending our platform to create an awesome place to run services
- 71. “Prevent an issue from happening twice” Incident Management Process
- 73. Incident (severity) Postmortem meeting with stakeholders Remediations (urgency) Incident Management Process
- 74. Moltes gràcies! David Poblador i Garcia @davidpoblador ! ! BcnDevCon13 November 12, 2013