Scaling Slack during explosive growth
•Als PPTX, PDF herunterladen•
0 gefällt mir•200 views
Slides from my DevSecOps Days talk (09/2020) about Scaling Slack during explosive growth. In this talk I shared some of the ides on how Slack managed to scale their service during the explosive growth in demand we lived during the start of the covid-19 pandemic with a world wide movement to work from home. In the talk I covered what did I mean with explosive growth, how we had invested in scalability over time in Slack's architecture divided by different layers and what do we do when things go wrong.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Scaling Slack during explosive growth
Ähnlich wie Scaling Slack during explosive growth (20)
Mehr von Javier Turégano Molina
Mehr von Javier Turégano Molina (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Scaling Slack during explosive growth
- 1. Scaling Slack during explosive growth Javier Turegano (he/him) @setoide
- 2. Agenda ● Intro ● Explosive growth ● Investing in scalability ● When things go wrong
- 3. Javier Turegano @setoide (he/him) Sr. Engineering Manager - Cloud Eng - APAC
- 4. It all started as a game ORIGIN
- 5. Today SLACK
- 6. IT and securityMarketing Design HR FinanceFile sharing Dev tools Communications Analytics Support ProductivitySales Slack integrates with 1800+ tools teams use daily
- 7. SOME STATS 130K+ paid customers 150+ countries 65 Fortune 100 customers 2,000+ employees
- 9. A good portion of the planet starts to move to WFH due to covid-19 ORIGIN
- 10. Off-course we have a bot to track this in Slack ORIGIN
- 12. Hack-HHVM Load testing frameworks On-boarding bigger enterprises Data Stores improvements Disaster-theatre pieces Client Re- architecture Cloud Infra Advancements New network architecture Scalability investments over time * This is a made-up timeline as an example. Scaling Slack - The Good, the Unexpected, and the Road Ahead
- 13. Reacting to explosive growth 👩 🏿 💻
- 14. Reacting to explosive growth
- 15. Reacting to explosive growth #help-cloud-econ
- 16. Edge POPs Load Balancing API Data Stores Basic layered architecture Engineering stats as of April 2020
- 17. At the edge
- 18. Client to edge AT THE EDGE Edge POP Apps / Chatbots API Edge POP Websocket HTTPS HTTPS WS HTTPS Achieving massive scale in a brave new (front-end) world
- 19. Message handling and caching AT THE EDGE Edge Cache Golang Edge POP Websocket Websocket Flannel: An Application-Level Edge Cache to Make Slack Scale
- 20. Message handling and caching AT THE EDGE Edge Cache Golang Edge POP Websocket Websocket
- 21. Processing file uploads at the edge AT THE EDGE S3 Edge POP File upload
- 22. Rate limiting & degraded mode AT THE EDGE HTTP/1.1 429 Too Many Requests Accept: application/json, text/plain Content-Type: application/x-www-form-urlencoded Date: Tue, 29 Jan 2019 18:41:22 GMT retry-after: 3 { "ok": false, "error": "ratelimited" } 🚧 Handling Rate Limits with Slack APIs
- 23. Edge POPs Load Balancing API - B Cache Data Stores 150M messages sent per minute during peak hours 13M simultaneous websocket connections per day At the edge
- 24. Scaling compute
- 25. High performance Scaling compute ● Hacklang / HHVM Hacklang at Slack: A Better PHP
- 26. Horizontal Auto Scaling Scaling compute ● Stateless ● Predictive Scaling ● Dynamic Scaling https://aws.amazon.com/blogs/aws/new-predictive-scaling-for-ec2-powered-by- machine-learning/
- 27. API farms Scaling compute ● Different API groups depending on the characteristics of the work API - B API -CAPI - A Load Balancing
- 28. Load Balancing API - B API -CAPI - A Edge POPs Data Stores 14B HTTP requests per day Dedicated API farms
- 29. Async processing + decoupling Scaling compute API - B Job Executors Queue (s)
- 31. Edge POPs Load Balancing API - B API -CAPI - A Cache Data Stores Job Executors Queue (s) 5B background jobs enqueued per day Async jobs
- 32. Data engines
- 33. Specialized data engines Data Engines ● Search Rebuilding Message Search at Slack
- 34. Scaling your stateful tier Data Engines ● Separate reads and writes ● Caching Data Stores (reads) API - B Caching Data Stores (writes)
- 35. Sharding databases with Vitess Data Engines A Journey into Slack’s Database Service
- 36. Edge POPs Load Balancing API - B API -CAPI - A Search Engine Cache Data Shards Cache Job Executors Job Queue 65b database queries per day 9 PB of database storage Scaling our data engines
- 37. When things go wrong
- 38. Incident management at Slack (in Slack) When things go wrong Incident bot and incident channels at the centre of our response #incd-53716 Adam Pretti 3:47PM joined #incd-53716 #incd-53716 Thread #incd-53716 But we also have a couple of backup strategies if/when Slack is not available.
- 39. Users are unable to connect to Slack When things go wrong
- 40. How it felt that day When things go wrong All Hands on Deck
- 41. Edge POPs Load Balancing API - B API -CAPI - A Search Engine Cache Data Shards Cache Job Executors Queue (s) A problem in any of your tiers has the potential to affect your whole service Houston, we’ve got a problem.
- 42. Technical details When things go wrong Load Balancing A Terrible, Horrible, No-Good, Very Bad Day at Slack
- 43. Incident Review process When things go wrong
Hinweis der Redaktion
- Thanks for having me! It’s a pleasure to be here and I hope everybody is doing alright in these difficult days we are living. This is probably going to be the less Security focus talk in the conference but I hope still relevant to you.
- Spaniard Moved to Melbourne 8 years ago after 1 year in freezing Cambridge in the UK I am an Open Source, IT leadership and Devops enthusiast I’ve been with Slack for a year, running the Cloud Engineering team split between Melbourne and San Francisco I’m a co-organizer of the Devopsgirls community.
- Maybe move to the other section
- This is a made-up timeline to use as a talking point. Our capacity to scale is due to the improvements and investments we have done over the years, and those on the timeline can be some examples. One thing that has help is growing with our customers. As we are taking bigger orgs we had to scale our systems to meet their needs. Number of users, messages, etc… @channel example Capacity planning / load testing Disaster piece - Chaos engineering Recommend in detail talk from Demmer, principal engineer on how we’ve implemented some of this improvements.
- Guidance from leadership: look after ourselves, keep Slack up and keep delivering value to customers Engineering teams focus: Looking for ceilings, bottlenecks and known problems in your architecture/infrastructure/services and figure out where we had to scale our systems
- As an example: Some systems had to scale from supporting 100,000 rpms to sometimes double that capacity at peak There are many different techniques that allowed us for the flexibility required to be able to react. We will cover some today.
- Also an important component was managing the increase cost of our infra and how to optimize that #help-cloud-econ
- Even if lego pieces are quite cool, I’ll be using a super simplified version of a system architecture view to walk through the different scalability improvements we applied over time Due to time constraints on my talk a lot of the slides will have a link that expands on the particular topics for those interested All the engineering stats presented are based on figures of April 2020
- We have different types of clients: Apps / Chat-bots (our own and third parties), Slack client for laptops and mobile and web clients. We deploy our own Edge POPs (Points of presence) to terminate those connections closer to our customers. We have Edge POPs in most AWS regions over the world. Two ways of connecting: Websockets and HTTPS
- We also do a lot of processing at the Edge making it faster for users but also offloading a lot of the work from the main APIs. For example, we do a lot of the message massaging/processing/enrichment at the edge. A previous version of our Edge stack was composed of series of Java processes handling Real Time messaging In order to scale an initial process that handles websocket connection and caching was created in Go called Flannel A good example is when a user connects first to a particular team in the morning, we preload a lot of information There is information required for almost any message transaction that is loaded from cache close to 100% times of the times (for example team_ids)
- We can see in this slide how one super common action benefits from caching channels and users when we start typing in the quick-switcher.
- Another example is file upload processing. Instead of doing this centrally we delegate it to our Edge Pops. Upload happens faster as its closer to the user and we can to all the image processing (metadata additions, security checks, setting up permissions, etc…)
- Apps and chatbots and some time even normal clients may have the capability of overloading your API. Two techniques can be used here to control this behaviour: Rate limiting Client in degraded mode We can also implement rate limiting between services using native language capabilities or more recently Envoy
- A good portion of the Slack APIs were originally written in PHP. At some point these were migrated to a combination of: Hack: a strongly typed version of PHP HHVM: a highly performance execution engine for Hack Both originally developed by Facebook and then opensourced. One of the motivations to move to Hack was developer productivity.
- We can divide our API into different farms depending on the tasks that they are performing and that will allow us to adapt the underlying infrastructure. For example: API for collecting statistics For background jobs
- This is the current implementation We are working in simplifying some of the components and using SQS
- Originally developed by YouToube and opensourced. It provides a sharding and topology management solution that sits on top of Mysql You don’t need to modify your application as they connect to VTGates as they were accessing a single database But in the background allows us to shard data not only by team but also by different tables: user, channel, etc… regardless of the database servers serving it We moved from master-master to master-slave and manage failover ---- Kubernetes
- In a nutshell Anyone can trigger an incident using the /assemble command We have a 24x7 rotation of Incident commander and Customer Experience liaisons When paged by assemble they join the channel from where someone called assemble and a video call for rapid categorization is setup Depending on severity we use incident bot to create a new incident channel and notify relevant parties when a high severity Required responders and SMEs are added to the channel /escalate command A lot of the context of how we responded lives in the channel and it’s super valuable to extract lessons from the incident Automated processes to create CAN reports (Conditions, Actions, Needs) and a Jira ticket to coordinate Incident commander makes sure that once we are All Clear someone is assigned to run an Incident Review If Slack is not available for response (Separate Slack workspace, Zoom, Google group)