As much as cloud-native applications and microservices help us be more productive and resilient and grow to unprecedented scales, they also bring an entirely new class of challenges. Let’s explore how the challenge of debugging applications has changed in a highly distributed world. From: https://www.dashcon.io/agenda/ten-years-of-failing-microservices/

1. Ten Years of Failing
Microservices
Phil Calçado - @pcalcado
Meetup/WeWork

3. What is a microservice?

4. Monolith

5. Entry Point
Service A Service B

6. Entry Point
Service A Service B
Service C Service D
Service E

7. Microservices are
highly-distributed
applicaDon architectures.

8. Debugging
monolithic
applicaDons

9. Monolith
The bug you are
looking for is here

11. Debugging
highly-distributed
applicaDons

12. Entry Point
Service A Service B
Service C Service D
Service E
🤔
🤔 🤔
🤔
🤔🤔

13. Entry Point
Service A Service B
Service C Service D
Service E
🤔
🤔 🤔
🤔
🤔🤔
Most bugs you’ll find in
microservices are sDll
isolated in a single
service.

14. But which one?

15. "

16. Some tools to find out which
service to look at when
debugging

17. Staging environments

18. "I know, we need a staging environment”
"We need to fix the staging environment”
“Staging is broken, let’s build another staging”
“I need you to use the first staging because
the new staging isn’t ready”
“Don’t use staging, it’s all broken"
"I know, we need a staging environment”

19. "I know, we need a staging environment”
"We need to fix the staging environment”
“Staging is broken, let’s build another staging”
“I need you to use the first staging because
the new staging isn’t ready”
“Don’t use staging, it’s all broken"
"I know, we need a staging environment”

20. "I know, we need a staging environment”
"We need to fix the staging environment”
“Staging is broken, let’s build another staging”
“I need you to use the first staging because
the new staging doesn’t”
“Don’t use staging, it’s all broken"
"I know, we need a staging environment”

21. "I know, we need a staging environment”
"We need to fix the staging environment”
“Staging is broken, let’s build another staging”
“I need you to use the first staging because
the new staging isn’t ready”
“Don’t use staging, it’s all broken"
"I know, we need a staging environment”

22. "I know, we need a staging environment”
"We need to fix the staging environment”
“Staging is broken, let’s build another staging”
“I need you to use the first staging because
the new staging isn’t ready”
“Don’t use staging, it’s all broken"
"I know, we need a staging environment”

23. "I know, we need a staging environment”
"We need to fix the staging environment”
“Staging is broken, let’s build another staging”
“I need you to use the first staging because
the new staging isn’t ready”
“Don’t use staging, it’s all broken"
"I know, we need a staging environment”

24. Useful tool:
Request Tracing

25. How to correlate logs across
services?

28. Entry Point
Service A Service B
Service C Service D
Service E
req1
req1-right1
req1-leV1
req1-leV2
req1-right1-leV1 req1-right1-right1

29. 2018-01-11 18:01:02.122 UTC - INFO - req1-right1-right1 - User [14523] deleted by user [56432]
2018-01-11 18:01:02.132 UTC - INFO - req1-right1-right1 - User [12] made admin
2018-01-11 18:01:03.002 UTC - INFO - req1-right1-right1 - User [3522] deleted by user [56432]
2018-01-11 18:01:03.341 UTC - INFO - req1-right1-right1 - User [14523] created via Facebook
2018-01-11 18:01:03.176 UTC - INFO - req1-right1-right1 - User [5643] deleted by user [1]
2018-01-11 18:01:04.265 UTC - INFO - req1-right1-right1 - User [4577] deleted by user [7544]
2018-01-11 18:01:04.531 UTC - INFO - req1-right1-right1 - User [3245] deleted by user [34]
2018-01-11 18:01:06.001 UTC - INFO - req1-right1-right1 - User [14523] deleted by user [56432]
Add span to your log lines

30. $ uncompress log | grep req1-right1
Search your logs by span
2018-01-11 18:01:02.122 UTC - INFO - req1-right1-right1 - User [14523] deleted by user [56432]
2018-01-11 18:01:02.132 UTC - INFO - req2-right1-right1 - User [12] made admin
2018-01-11 18:01:03.002 UTC - INFO - req1-right1-right1 - User [3522] deleted by user [56432]
2018-01-11 18:01:04.143 UTC - ERROR - req1-right1-left1 - Failed to delete picture [3522.jpg] from CDN
2018-01-11 18:01:06.001 UTC - INFO - req1-right1-right1 - User [14523] deleted by user [56432]

31. Useful tool:
Telemetry

32. As the number of services or
users grow, it is expensive to
have verbose logging

33. As you grow:
•Verbose logging becomes
expensive
•There’s so much going on you’ll
be lost

34. The first place to look at should
be the telemetry dashboards

35. Entry Point
Service A Service B
Service C Service D
Service E
AutomaDcally create a
dashboard for each service

36. Useful tool:
Request/Response
Capturing

37. Service
Request
Response
=>{ ,
=> ,
=> }
Storage
Queryable Interface

38. I was really against this idea.

39. Service
Request
Response
=>{ ,
=> ,
=> }
Storage
Queryable Interface
Service owns the storage

40. Service
Request
Response
=>{ ,
=> ,
=> }
Storage
Queryable Interface
Network owns the storage

41. Microservices dirty
liale secrets

42. ~1/10 of all my
engineering teams was
dedicated to building
tooling

43. Polyglot programming
has died

44. You end up with vendor
lock-in with your own
tools

45. …but in 2018 things aren’t
nearly as bad anymore.

46. "Higher-level networking”

47. Serverless?

48. Do we have Dme for
quesDons?