SoundCloud is the largest repository of audio on the web, used by more than 200 million people every month, who upload more than 11 hours of audio every minute. Like so many others, we have migrated from a typical monolithic architecture to micro-services. While the benefits brought by this style of SOA to our productivity and reliability are clear, the architecture required some non-obvious changes in the way we operate systems, and a way to tackle the overhead associated with having hundreds of small moving parts to serve every request. In this talk we’ll share the toolkit and strategy SoundCloud uses to keep its micro-services explosion manageable. What do we do about the operations overhead? How to spread devops skills across teams to support the “you build it, you run it” vision? How to deal with breaking changes and asynchronous behaviours? How to deal with chatty interactions? Which protocol? How do I even get a diagram telling me how all this stuff is put together? http://qconlondon.com/presentation/no-free-lunch-indeed-three-years-micro-services-soundcloud

1. No free lunch, indeed:
Three years
Phil Calçado
SoundCloud
of microservices
at SoundCloud

4. > 11 hours of audio uploaded
every minute
~ 300 million people every month

5. heaps have been written
about microservices in
the past year-ish

7. tl;dr
• Rapid provisioning
• Basic Monitoring
• Rapid application deployment

8. These make sense
• Rapid provisioning
• Basic Monitoring
• Rapid application deployment
this makes sense
why does it make me so
nervous?

9. the SoundCloud story
you might know

11. the pre-history

12. SoundCloud, circa 2011

13. let’s prepare for the
“microservices
explosion"

14. #1 provisioning

15. what was cool in 2010-11

16. what was cool in 2010-11
doozer lxc 12factor.net

17. much better than
anything else at the
time

18. a problem
no resource limits (i.e. cgroups)
+
naïve scheduling
=
loud neighbour in your own
datacentre

19. a problem
made for
most of our services migrated to

20. the problem
time
start work
on v1
v1 100%
deployed
start work
on v2

21. before we go
sophisticated, let’s
simplify what we have

22. warmed up
pool
machine
intake

23. not the final solution,
but will buy us some
time

24. #2 telemetry

25. state of telemetry tools
circa 2011-12 wasn’t
great

26. StatsD

27. let’s build our own!

29. but that’s not what
broke…

30. obvious with a monolith
monolith

31. not so much now
?
?
?

32. standardise dashboards

33. standardise operations
https://twitter.github.io/twitter-server/Features.html#http-admin-interface

34. visualise

35. add management lines,
all the way up, to
escalation policies

36. #3 deployment

38. > git
SquashFS
> make
unit tests
integratio
n tests
acceptanc
e tests
perf tests
> make
/dev/null

39. we ended up with 7
different deployment
scripts

40. > make > gitunit tests
integratio
n tests
unit tests
integratio
n tests
acceptanc
e tests
perf tests

41. containers let you
spawn your mini-
SoundCloud

42. but why was I so
nervous?

43. because we
messed up

44. there are simple and incremental
ways to address these
• Rapid provisioning
• Basic Monitoring
• Rapid application deployment

45. “uh? do you think
Netflix got it right the
first time?"

46. some good things

47. Q&A