Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2k6S0kO. Julia Grace was asked to build Slack’s first infrastructure engineering organization in August 2016. The company was two years old and they were approaching the scalability limits of the original infrastructure. Things were starting to break in strange and unpredictable ways. She discusses the architectural and organizational challenges, mistakes and war stories of 2.5 years that followed. Filmed at qconnewyork.com. Julia Grace is currently a senior director of product engineering at Slack focused on building network effects into Slack through shared channels. Prior to joining product engineering she built the infrastructure team at Slack, growing it from from 10 to 100 engineers in three offices in two years.

1. @jewelia
Scaling Slack
Infrastructure
🚀
Julia Grace
Senior Director of Engineering
@jewelia

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
slack-scaling-infrastructure/

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. @jewelia
Phase 0: 2015

5. @jewelia
~2.5M Daily
Active Users

6. @jewelia
Phase 1: 2016

7. @jewelia
~4M Daily Active
Users

8. @jewelia
Phase 1: 2016
Slack was originally designed for teams < 150ppl.
You make very different architectural decisions when you’re building for a team of
100 people vs 500,000.
Before August 2016 we had no Infra team.
Original infrastructure built for Glitch worked very well in 2014/2015.
~150 Engineers total.
Infrastructure investments would come secondary to feature work.

9. @jewelia
Things were starting to break in strange,
unusual
ways.

10. @jewelia
Phase 1: 2016
Example: User Presence
Green dot indicaFng online/away/offline.
Very few people noFce it, unless it’s broken (people expect it to “just work”).
Apps and bots are always online.

11. @jewelia
Phase 1: 2016
User Presence
IniFally broadcast all changes to all users (e.g. “Julia Grace is away”) to the whole
workspace: O(n^2).
Presence was ~80% of all web socket traffic.
Peak volume in late 2016: 16 million messages/minute over web socket.
Presence messages: 13 million/minute.
Rapidly transiFon from broadcast to publish/subscribe.

12. @jewelia
There were many organizational challenges as
well.

13. @jewelia
Phase 1: 2016
How to build engineering-led org in a product-
led company?
Would we be able to get headcount, budget?
How to communicate the value of we are doing to non-technical audiences?
How do we interface with sales?
Infrastructure as a compeFFve advantage.

14. @jeweliahUps://www.flickr.com/photos/pocheco/14833391966

15. @jewelia
Phase 1: 2016
Start internal evangelism on day #1.
I went on an internal PR campaign: Why our work was important, why we needed
to conFnually invest in infrastructure. Make work very visible to execs in other
funcFons.
Followed existing company process.
We did planning, status reporFng, etc. at the same cadence and in the same
meeFngs as product engineering. Don’t try to start a new group and invent new
process.
Identify executive sponsor.

16. @jewelia
Phase 2: 2017

17. @jewelia
Phase 2: 2017
Technology landscape.
Hack/PHP monolith on backend, JavaScript with no libraries on frontend.
1 service: presence and real-Fme messaging.
Building a second service: Go caching service.
These bespoke services each had to handle rate limiFng, traffic management,
deployment.

18. @jewelia
Phase 2: 2017
It was time to change our DB sharding strategy.
MySQL sharded by team/workspace to Vitess sharded by various keys.
Worked great! UnFl we hit scaling limits, significant hotspots.

19. @jewelia
Monolith
Service
A
Service
B

20. @jewelia
Monolith
Service
A
Service
B

21. @jewelia
Monolith
Service
A
Service
B
Who owns this?

22. @jewelia
Communication Risk
The more technically complex,
nuanced a problem is…

23. @jewelia
Communication Risk
The more technically complex,
nuanced a problem is…
The higher the communication risk.

24. @jewelia
Phase 2: 2017
Immense pressure to hire engineers.
Many human SPOFs (single points of failure) because team was so small.
Everyone was overextended and overcommihed.
We had to figure out how to hire Infra
engineers.
All our hiring processes were opFmized around hiring generalists: frontend
backend, iOS, Android, Ops.
We skills do we need and value? How do we test for those skills?

25. @jewelia
Phase 2: 2017
Decided to hire Infra engineering generalists.
Created a take home coding exercise designed
to test:
1. An understanding of servers, networking, and protocols.
2. An understanding of concurrency, performance, and resource constraints, and
an ability to anFcipate future issues and implement soluFons.
3. An ability to write clear, easy to understand code, communicate your approach,
and reason about tradeoffs that you have made.

26. @jewelia
Phase 2: 2017
I wore so many hats. Too many hats.
Similar to my days as a startup CTO!
I was the Engineering Director and
Forming strategy, hiring managers and ICs, evangelizing the org.
…Product Manager and
Internal interface to Product Engineering/PMs building features, externally to
customers with quesFons about the integrity of our infrastructure.
…Program Manager.
Running cross funcFonal iniFaFves.

27. @jewelia
Phase 3: 2018

28. @jewelia
Phase 3: 2018
“0 to 1” was over. Now time for “1 to ∞”.
ReacFve to ProacFve.
Transition from few teams to an org in 3 offices.
Team nearly 100 engineers by end of year.
Now included Data, Machine Learning, Search Infrastructure
Many orders of magnitude better performance
Things were not breaking all the time.

29. @jewelia
Phase 3: 2018
Services model matured significantly.
SLAs for services, consistent deployment processes, etc.
Mature incident response process.
Dividing into sub-teams made sense.
Data Stores & Cache Infra, Service Mesh & Web Serving, Distributed Messaging.

30. @jewelia
Phase 3: 2018
Hired Director Specialists…
Had to quickly learn how to hire senior leaders whose jobs you haven’t done before.
How to do this well: talk to a lot people who currently do the job you’re trying to hire
for, deeply understand the talent market.
and Product Managers…
and did an acquisition.

31. @jewelia
Phase 3: 2018
Challenge: coherency across a large
organization.
Example: overlap between Machine Learning and Frontend Infra was NULL.
Difficult to have a unified vision.
Stakeholders were each org were different for each part of the org; Data Infra
organizaFon worked closely with G&A (finance), Search Infra did not.
I should have done more re-orgs!

32. @jewelia
2016:

33. @jewelia
2016:
2017:

34. @jewelia
2016:
2017:
2018:

35. @jewelia
Today
Infra has been around for ~3 years
400M async jobs processed/day to 2.5B
3M DAU (daily active users) to 10M DAU
1M simultaneously connected users to 7.5M
10 to ~100 engineers in SF, NYC, YVR
Generalist (ICs, Managers) to specialists
1 amazing team

36. @jewelia
Thank You!

37. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
slack-scaling-infrastructure/