At SoundCloud, we’ve found that teams move faster when we’ve moved away from a monolith architecture to one based around microservices. Unfortunately, this new type of architecture has been prone to cascading failures when breaking changes go unnoticed in one of our services’ API’s. These failures have had a devastating impact on our system’s uptime, but we’ve found that we can mitigate some of this risk by introducing consumer driven contract tests. Consumer driven contract tests allow each consumer service and client to define their expectations and interactions with each provider service upstream, and for provider services to verify all of these contracts as part of their build pipeline. Breakage avoided. In this talk we’ll go through SoundCloud’s process of breaking the monolith into microservices, then see how PACT-based contract tests were introduced, and discuss some of the challenges of adopting such tests in an already-established microservices culture.

1. Making SoundCloud’s µ-services safe(r) to deploy
Move Fast and Consumer-Driven-
Contract-Test Things
XP Days Ukraine 2017

2. 2.5 years @ SoundCloud Monetization Team Backend Engineer
@alonpeer

3. About SoundCloud
a cloud full of sounds
135M tracks, 175M listeners, 12 hours/minute
300+ services / 50+ teams

4. Agenda
● Testing, monolith style
● My first steps @ SoundCloud
● Introducing contract tests
● Pactifying our services
● Caveats
● QA Q&A

5. Testing, monolith style

6. ● One service, one client
● API changes are easy
● One client team to sync with for migrations
● API deprecation is easy
Testing, monolith style
The good ol’ days

7. ● Different requirements per client
=> Code complexity increases
● Harder to deploy without breaking at least one client
Testing, monolith style
Mo clients, mo problems

8. ● More manual QA
expensive, slow, prone to human error,
doesn’t scale
● More end-to-end tests
Maintenance nightmare, flaky, slow,
creates bottlenecks
Testing, monolith style
Mo clients, mo problems

9. My first steps @ SoundCloud

10. My first steps @ SoundCloud
Whoa, micro-services

11. My first steps @ SoundCloud
Testing strategy
E2E
Unit
Integration

12. My first steps @ SoundCloud
Digging into post-mortems
© ofsmallthings

13. My first steps @ SoundCloud
“A lack of trust in the acceptance tests caused us to largely
ignore the warnings they generated.”
“We couldn’t figure out the impact of the broken acceptance
tests, and assumed the only problem was the tests themselves,
rather than the underlying code.”
“The commit went through as there weren't any tests for
the serialisation from the client to the backend.”
Digging into post-mortems
© ofsmallthings

14. Introducing contract tests

15. Introducing contract tests
My daily routine
© LockeSteerpike

16. Introducing contract tests
My daily routine
© rickandmorty.com

17. Unit tests are not enough
Introducing contract tests
“#cake” >> {
result = call(“cake”)
assert(result == “lie”)
}
GET “/cake”:
return “lie”

18. Unit tests are not enough
Introducing contract tests
GET “/cake”:
return “truth”
“#cake” >> {
result = call(“cake”)
assert(result == “lie”)
}

19. Unit tests are not enough
Introducing contract tests
import JsonLibFoo
GET “/cake”:
return toJson(current_state)
// { “current_state” : “lie” }
import JsonLibFoo
“#cake” >> {
result = fromJson(call(“cake”))
assert(result == “lie”)
}

20. Unit tests are not enough
Introducing contract tests
import JsonLibBar
GET “/cake”:
return toJson(current_state)
// { “currentState” : “lie” }
import JsonLibBar
“#cake” >> {
result = fromJson(call(“cake”))
assert(result == “lie”)
}

21. Introducing contract tests
How this works?
Consumer Provider
End-to-End Test

22. Introducing contract tests
How this works?
Consumer Provider
Unit Test Unit Test

23. Introducing contract tests
How this works?
Consumer Provider
Unit Test Unit Test

24. Introducing contract tests
How this works?
➢ Requesting “/cake”.
➢ Expecting a JSON response
with a key “current_state”.
➢ Asserting deserialization
of the response succeeds.
import JsonLibFoo
GET “/cake”:
return toJson(current_state)
// { “current_state” : “lie” }
Consumer
Provider

25. Pactifying our services
http://pact.io

26. What’s Pact?
Pactifying our services
● A family of frameworks designed for consumer driven
contract testing.
● Supports JSON over HTTP.
● Impl. for Java, Scala, Go, Ruby, .Net, Swift, JS etc.
● Tests run locally, no external dependencies.

27. What’s Pact?
Pactifying our services
Step 1: Define consumer expectations
© pact.io

28. What’s Pact?
Pactifying our services
Step 2: Verify expectations on provider
© pact.io

29. Consumer: client code (LikesClient)

30. Consumer: client code (LikesClient)

31. Consumer: client code (LikesClient)

32. Consumer: client code (LikesClient)

33. Consumer: client code (LikesClient)

34. Consumer: client code (LikesClient)

35. Consumer: client unit test

36. Consumer: client unit test

37. Consumer: client unit test

38. Consumer: client unit test

39. Consumer: client contract unit test

40. Consumer: client contract unit test

41. Consumer: client contract unit test

42. Consumer: client contract unit test

43. Consumer: client contract unit test

44. Consumer: client contract unit test

45. Consumer: client contract unit test

46. Consumer: client contract unit test

47. Consumer: my-consumer-likes.json

48. Consumer: my-consumer-likes.json

49. Consumer: my-consumer-likes.json

50. Consumer: my-consumer-likes.json

51. Consumer: my-consumer-likes.json

52. Consumer: my-consumer-likes.json

53. Consumer: my-consumer-likes.json

54. Pactifying our services
Provider side verification
● Collect pact files from consumers and verify them all.
● Keep your provider environment isolated.
○ Dockerized databases.
○ Test doubles for upstream services.

55. Pactifying our services
Provider states

56. Pactifying our services
Provider states
Likes
App
Likes
DB

57. Pactifying our services
Provider states
Set state:
“User 1000 has liked 2 tracks”
Likes
App
Likes
DB

58. Pactifying our services
Provider states
INSERT
likes
Set state:
“User 1000 has liked 2 tracks”
Likes
App
Likes
DB

59. Pactifying our services
Provider states
INSERT
likes
Set state:
“User 1000 has liked 2 tracks”
HTTP Request: GET /likes/1000
Likes
App
Likes
DB

60. Pactifying our services
Provider states
HTTP Request: GET /likes/1000
HTTP Response: 200 OK
INSERT
likes
Set state:
“User 1000 has liked 2 tracks”
Likes
App
Likes
DB

61. Pactifying our services
Provider states
Likes
App
Likes
DB
HTTP Response: 200 OK
INSERT
likes
Set state:
“User 1000 has liked 2 tracks”
HTTP Request: GET /likes/2000
HTTP Response: 404 Not Found
Set state:
“User 2000 has liked no tracks”
HTTP Request: GET /likes/1000
DELETE
likes

62. Pactifying our services
Pact broker
● Share pact files between projects.
● API for pact frameworks to fetch all pacts by provider.
● Support for versioning and tagging.
● Useful UI and visualization of the network.

63. Pactifying our services
Pact broker
© pact broker

64. Pactifying our services
Pact broker
© pact broker

65. Pactifying our services
Pact broker
© pact broker

66. Pactifying our services
Our CI pipeline
Push new
change
Generate
consumer
contracts
Deploy
Upload
pacts & tag
with ‘prod’
Consumer pipeline

67. Pactifying our services
Our CI pipeline
Push new
change
Deploy
Upload
pacts & tag
with ‘prod’
Consumer pipeline
Push new
change
Verify all
‘prod’-tagged
pacts
Deploy
Provider pipeline
Generate
consumer
contracts

68. ● Communication is key.
Caveats

69. ● Communication is key.
● New moving parts.
Caveats

70. ● Communication is key.
● New moving parts.
● Learning curve per pact framework.
Caveats

71. ● Communication is key.
● New moving parts.
● Learning curve per pact framework.
● Frameworks are WIP.
Caveats

72. ● Communication is key.
● New moving parts.
● Learning curve per pact framework.
● Frameworks are WIP.
● Provider side setup is time consuming.
Caveats

73. ● Communication is key.
● New moving parts.
● Learning curve per pact framework.
● Frameworks are WIP.
● Provider side setup is time consuming.
● Automating consumer-triggered provider verification.
Caveats

74. ● Communication is key.
● New moving parts.
● Learning curve per pact framework.
● Frameworks are WIP.
● Provider side setup is time consuming.
● Automating consumer-triggered provider verification.
● Adoption is essential.
Caveats

75. Recap

76. Recap
Write contract tests
Consum
ers

77. Recap
Write contract tests Generate pact files
Consum
ers

78. Recap
Write contract tests Generate pact files
Publish to broker
Consum
ers

79. Recap
Write contract tests Generate pact files
Publish to brokerVerify pacts
Consum
ers
Providers

80. Recap
Write contract tests Generate pact files
Publish to brokerVerify pacts
Consum
ers
Providers

81. QA Q&A
@alonpeer