In a microservices architecture, different services usually have different availabilities. It is often hard to see how the availability of a single service affects the availability of the overall system. Without a clear idea about the availability requirements of individual services, even a seemingly subtle degradation of a service can cause a critical outage. Unfortunately these are discovered only after thorough post-mortems. At SoundCloud we kicked off a project called “Availability Objectives”. An availability objective is the minimum availability a service is allowed to have. These objectives are calculated based on the requirements of the clients of those services. We started by visiting all of our services and setting an availability objective for each of them. We built tools to expose the availability of these services and to flag the ones that drop below their objectives. As a result, we can now make informed decisions about the integration points we need to improve first. This talk will share the insights we gained via this project and how it affected our overall availability and engineering productivity.

1. Bora Tunca
Availability Objectives of
SoundCloud’s Microservices

2. SoundCloud is the largest
online audio distribution
platform

3. ● Backend developer
● @SoundCloud ~3 years
● Core engineering
Bora Tunca
@grandbora

4. Microservices

6. ~140 services
~140 engineers

7. How do we improve
availability
in a microservices architecture?

10. MS
API
MS
MS
MS
MS MS

11. Post Mortem Meetings
help us spot “risky” services

12. Service Level Objectives
define an availability objective for each microservice

13. If the availability of a service
drops below its objective,
we raise the red flag.

14. Service Level Objectives
● SLO infrastructure
● Usage analyses

15. SLO Infrastructure

16. Availability
● Error Rate Threshold
● Latency Threshold 99th percentile

17. SLO Parameters
SLO_ERROR_RATE_THRESHOLD=0.0001
SLO_LATENCY_P99_THRESHOLD_SECONDS=0.500
SLO_AVAILABILITY_TARGET=0.99999

18. SLO Metrics
● TotalTransactions
● FailedTransactions
● SlowTransactions

19. Availability
(FailedTransactions / TotalTransactions) <
&&
(SlowTransactions / TotalTransactions) < 0.01
99th percentile
error rate
threshold

20. SLO
Wrapper
MS
Req Res
Prometheus
Metrics
Services
Directory
Availability

22. Usage Analyses

23. API API API
Microservices

24. API API API
MS MS MS MS MS MS MS
MS MS MS MS MS MS MS MS
MS MS MS MS MS MS MS
MS MS MS MS MS MS
Edge
Value
Added
Data

25. MSMS
API API API
MS MS MS MS MS
MS MS MS MS MS MS MS MS
MS MS MS MS MS MS MS
MS MS MS MS MS MS
Edge
Value
Added
Data

26. Graceful Degradation
Tracks
API
99.995%95%
Stats

27. Learnings

28. ● Without a formal process identifying risky services is hard
● Don’t over-engineer your services
● Let the data guide your engineering decisions

29. THANK YOU
bora@soundcloud.com