This document summarizes Phil Calçado's experience with microservices at SoundCloud over three years. It describes how SoundCloud moved to microservices for rapid provisioning and deployment. This created monitoring challenges as services became more distributed. SoundCloud developed its own telemetry tools to address these issues. The document also discusses lessons learned, such as standardizing dashboards and operations, and how initial microservice implementations at SoundCloud were imperfect but improved over time through experience.

1. No free lunch, indeed:
Three years
Phil Calçado
SoundCloud
of microservices
at SoundCloud

2. No free lunch, indeed:
Three years
Phil Calçado
SoundCloud
of microservices
at SoundCloud

5. > 11 hours of audio uploaded
every minute
~ 300 million people every month

6. heaps have been written
about microservices in
the past year-ish

8. tl;dr
• Rapid provisioning
• Basic Monitoring
• Rapid application deployment

9. These make sense
• Rapid provisioning
• Basic Monitoring
• Rapid application deployment
this makes sense
why does it make me so
nervous?

10. the SoundCloud story
you might know

11. we moved to
microservices
because $reasons
http://bit.ly/how_we_ended_up_with_microservices

13. the pre-history

14. SoundCloud, circa 2011

15. let’s prepare for the
“microservices
explosion"

16. #1 provisioning

17. what was cool in 2010-11

18. what was cool in 2010-11
doozer lxc 12factor.net

19. much better than
anything else at the
time

20. a problem
no resource limits (i.e. cgroups)
+
naïve scheduling
=
loud neighbour in your own
datacentre

21. a problem
made for
most of our services migrated to

22. the problem
time
start work
on v1
v1 100%
deployed
start work
on v2

23. attempt #1 - before we
go sophisticated, let’s
simplify what we have

24. warmed up
pool
machine
intake

25. just a bit of glue code

26. just a bit of glue code
a LOT

27. machine
intake

28. #2 telemetry

29. state of telemetry tools
circa 2011-12 wasn’t
great

30. StatsD

31. let’s build our own!

33. but that’s not what
broke…

34. obvious with a monolith
monolith

35. not so much now
?
?
?

36. standardise dashboards

37. standardise operations
https://twitter.github.io/twitter-server/Features.html#http-admin-interface

38. visualise

39. #3 deployment

41. > git
SquashFS
> make
unit tests
integratio
n tests
acceptanc
e tests
perf tests
> make
/dev/null

42. we ended up with 7
different deployment
scripts

43. > make > gitunit tests
integratio
n tests
unit tests
integratio
n tests
acceptanc
e tests
perf tests

44. containers let you
spawn your mini-
SoundCloud

45. but why was I so
nervous?

46. because we
messed up

47. there are simple and incremental
ways to address these
• Rapid provisioning
• Basic Monitoring
• Rapid application deployment

48. “wtf? do you think Netflix
got it right the first time?”
— @adrianco

49. some of the good things

50. Q&A