SoundCloud's web and mobile properties are all built on top of our Public API. While building the user-interface atop a RESTful layer has proven itself a sound decision, the one-size-fits-all nature of a Public API is not ideal. When it comes to data transfer and HTTP resource modelling, each client has their own needs, and in the end hacks and workarounds have to be implemented in both clients and servers. Feature development also becomes complicated, with coordination between multiple teams required for every single little feature. SoundCloud is now moving to a different model, where clients have their own façade APIs, modelled after their core characteristics and needs. We are also using the architecture to break away from Conway’s Law and building more cross-functional, end-to-end teams. From: http://gotocon.com/berlin-2013/presentation/APIs:%20The%20Problems%20with%20Eating%20your%20Own%20Dog%20food To implement this, a lot of change in our architecture, tech stack and development processes were required. In this talk we will explore the challenges we had, the options we investigated and how ultimately SoundCloud decided to move forward.

1. The problems with
eating your own
dog food
Phil Calçado
SoundCloud
Thursday, October 17, 13

2. Thursday, October 17, 13

3. >11 hours of
audio uploaded
every minute
Thursday, October 17, 13

4. ~200
million users
every month
Thursday, October 17, 13

5. Thursday, October 17, 13

6. Thursday, October 17, 13

7. http://bit ly/tech-behind-new-twittercom
Thursday, October 17, 13

8. Thursday, October 17, 13

9. Just before
the beta
release
Thursday, October 17, 13

10. http://bit ly/improving-performance-twittercom
Thursday, October 17, 13

11. Thursday, October 17, 13

12. Ship it?
Thursday, October 17, 13

13. Ship it!
(just overprovision the api)
Thursday, October 17, 13

14. ~3 back-end requests
Thursday, October 17, 13

15. ~159 back-end requests
Thursday, October 17, 13

16. First to break
were the
haproxy
Thursday, October 17, 13

17. Then
memcached
Thursday, October 17, 13

18. Then
Rails/MySQL
Thursday, October 17, 13

19. Rails
won’t go away
need faster api
Thursday, October 17, 13

20. SoundCloud.com
Thursday, October 17, 13

21. Sounds Social Activity
& Sets Graph Stream
Premium
Search API
Features
Thursday, October 17, 13

22. Social
Graph
Sounds
& Sets
API
Activity
Stream
Premium
Features
Thursday, October 17, 13
Search

23. 100%
75%
50%
25%
0%
Original
Database
Thursday, October 17, 13
Services Extracted
Other
HTTP

24. 100%
75%
50%
25%
0%
Original
Database
Thursday, October 17, 13
Services Extracted
Other
HTTP

25. about the same, tbh
100%
75%
50%
25%
0%
Original
Database
Thursday, October 17, 13
Services Extracted
Other
HTTP

26. Rails
won’t go away
need faster Rails
Thursday, October 17, 13

27. Thursday, October 17, 13

28. lots of I/O
concurrency friendly
Thursday, October 17, 13

29. Rails
doesn’t like
parallelism
Thursday, October 17, 13

30. time
Thursday, October 17, 13

31. Can we
make it like
concurrency?
Thursday, October 17, 13

32. time
Thursday, October 17, 13

33. time
Thursday, October 17, 13

34. Thursday, October 17, 13

35. Thursday, October 17, 13

36. What
else
is there?
Thursday, October 17, 13

37. Thursday, October 17, 13

38. Thursday, October 17, 13

39. Thursday, October 17, 13

40. Thursday, October 17, 13

41. Thursday, October 17, 13

42. time
Thursday, October 17, 13

43. time
Thursday, October 17, 13

44. Thursday, October 17, 13

45. What
about
the network?
Thursday, October 17, 13

46. GET /tracks/trending/music
Thursday, October 17, 13

47. GET /tracks/trending/music
GET /tracks/genres
Thursday, October 17, 13

48. GET /tracks/trending/music
GET /tracks/genres
GET /tracks/genres/jazz
Thursday, October 17, 13

49. GET /tracks/trending/music
GET /tracks/genres
GET /tracks/genres/jazz
GET /tracks/123
Thursday, October 17, 13

50. What
about
a custom API?
Thursday, October 17, 13

51. GET /tracks/suggested
Thursday, October 17, 13

52. GET /tracks/suggested
GET /tracks/123
Thursday, October 17, 13

53. api.soundcloud.com
iOS
Thursday, October 17, 13
Android Desktop Widget 3rd Party

54. api-mobile
iOS
Thursday, October 17, 13
api web
api-partn
Android Desktop Widget 3rd Party

55. What
comes
next?
Thursday, October 17, 13

56. Stop
babysitting
Rails
Thursday, October 17, 13

57. Enable &
empower
app devs
Thursday, October 17, 13

58. “Experience
based”
api?
Thursday, October 17, 13

59. phil calçado
http://philcalcado.com
@pcalcado
www.soundcloud.com
Thursday, October 17, 13