Distributed microservices introduce new challenges: failure modes are harder to anticipate and resolve. In this session, we present a “Chaos Debugging” framework enabled by three open source projects: Gloo Shot, Squash, and Loop to help you increase your microservices’ “immunity” to issues. Gloo Shot integrates with any service mesh to implement advanced, realistic chaos experiments. Squash connects powerful and mature debuggers (gdb, dlv, java debugging) to your microservices while they run in Kubernetes. Loop extends the capability of your service mesh to observe your application and record full transactions for sandboxed replay and debugging. Come to this demo-heavy talk to see how together, Squash, Gloo Shot, and Loop allow you to trigger, replay, and investigate failure modes of your microservices in a language agnostic and efficient manner without requiring any changes to your code.

1. 1 | Copyright © 2019
CHAOS DEBUGGING
CHRISTIAN POSTA – FIELD CTO, SOLO.IO

2. 2 | Copyright © 2019
CHRISTIAN POSTA
• Field CTO @ solo.io
• Author of a few books
• Contributor to many open-source projects
• Architect, blogger, speaker, mentor, leader
https://bit.ly/istio-in-action
@christianposta
christian@solo.io
https://blog.christianposta.com
https://slideshare.net/ceposta

3. 3 | Copyright © 2019
Move fast, safely
https://puppet.com/resources/whitepaper/state-of-devops-report

4. 4 | Copyright © 2019 @christianposta
Service Mesh

5. 5 | Copyright © 2019
Service mesh technologies typically provide:
• Service discovery / Load balancing
• Secure service-to-service communication
• Traffic control / shaping / shifting
• Policy / Intention based access control
• Traffic metric collection
• Service resilience
• API / programmable interface

6. 6 | Copyright © 2019 @christianposta
Service Mesh Interface (SMI)
https://github.com/deislabs/smi-spec https://supergloo.solo.io
https://servicemeshhub.io

7. 7 | Copyright © 20197 | Copyright © 2019
THE PROBLEM:
DEBUGGING MICROSERVICES APPLICATIONS IS HARD.

8. 8 | Copyright © 2019
THE PROBLEM
A MONOLITHIC APPLICATION
CONSISTS OF A SINGLE
PROCESS
AN ATTACHED DEBUGGER
ALLOWS VIEWING THE
COMPLETE STATE OF THE
APPLICATION DURING RUNTIME
A MICROSERVICES APPLICATION
CONSISTS OF POTENTIALLY
HUNDREDS OF PROCESSES
IS IT POSSIBLE TO GET A
COMPLETE VIEW OF THE STATE
OF A SUCH APPLICATION?!

9. 9 | Copyright © 2019
THE PROBLEM

10. 10 | Copyright © 2019
THE PROBLEM

11. 11 | Copyright © 201911 | Copyright © 2019
SQUASH DEMO

12. 12 | Copyright © 2019
SQUASH
DEFAULT MODE SECURE MODE

13. 13 | Copyright © 2019
SQUASH DEFAULT MODE
Node
Namespace: ns-a Namespace: squash
s-dlvc1

14. 14 | Copyright © 2019
-> ls -l /proc/self/ns
total 0
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 net -> net:[4026532009]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 pid -> pid:[4026531836]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 pid_for_children -> pid:[4026531836]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 user -> user:[4026531837]
lrwxrwxrwx 1 idit idit 0 Dec 7 01:14 uts -> uts:[4026531838]
-> inod of mnt namespace (unique identifier to the container namespace)
via CRI api call ExecSyncRequest
Node
Namespace: ns-a
s-dlv
CRI
c1
we need to translate the pid of the process (application that run in the container) to the host pid namespace
to allow debugger to attach.
Namespace: Squash

15. 15 | Copyright © 2019
SQUASH SECURE MODE
Node
Namespace: ns-a Namespace: squash
s-dlvc1
CRD Intent
squash

16. 16 | Copyright © 2019
DOCS: HTTPS://SQUASH.SOLO.IO
GITHUB: HTTPS://GITHUB.COM/SOLO-IO/ SQUASH
COMMUNITY: HTTPS://SLACK.SOLO.IO

17. 17 | Copyright © 2019
CAN I USE SQUASH TO DEBUG IN PRODUCTION ?

18. 18 | Copyright © 201918 | Copyright © 2019
DEBUGGING IN PRODUCTION

19. 19 | Copyright © 2019
DB S3
DEBUGGING IN PRODUCTION
CLUSTER
POD 1 POD 2
> ONLY HEADER WILL BE SENT
> SAMPLING
POD 3 POD 4

20. 20 | Copyright © 2019
DB S3
P P P P
DEBUGGING IN PRODUCTION
CLUSTER
POD 1 POD 2 POD 3 POD 4
> ONLY HEADER WILL BE SENT
> SAMPLING

21. 21 | Copyright © 2019
DB S3
P P P P
DEBUGGING IN PRODUCTION
CLUSTER
POD 1 POD 2 POD 3 POD 4

22. 22 | Copyright © 2019
DB S3
P P P P
DEBUGGING IN PRODUCTION
CLUSTER
POD 1 POD 2 POD 3 POD 4

23. 23 | Copyright © 2019
DEBUGGING IN PRODUCTION
+

24. 24 | Copyright © 201924 | Copyright © 2019
LOOP DEMO

25. 25 | Copyright © 2019
DOCS: COMING REAL SOON …
GITHUB: COMING REAL SOON …
COMMUNITY: HTTPS://SLACK.SOLO.IO

26. 26 | Copyright © 201926 | Copyright © 2019
PROACTIVE DEBUGGING

27. 27 | Copyright © 2019
CHAOS ENGINEERING
THINK OF A VACCINE OR A FLU SHOT
INJECT YOURSELF WITH SOMETHING HARMFUL
IN ORDER TO PREVENT A FUTURE ISSUE.
CAREFULLY INJECTING THIS HARM INTO YOUR SYSTEMS
TO TEST THE SYSTEM’S ABILITY TO RESPOND TO IT.
“BREAK THINGS ON PURPOSE" IN ORDER TO LEARN
HOW TO BUILD MORE RESILIENT SYSTEMS.

28. 28 | Copyright © 2019
PROBLEMS WITH CHAOS ENGINEERING TODAY?
LANGUAGE SPECIFIC CODE MODIFICATION
1 2

29. 29 | Copyright © 2019
NETWORK ABSTRACTION
EAST-WEST
TRAFFIC
NORTH-SOUTH
TRAFFIC
SERVICE
I
SERVICE
II
SERVICE
III
SERVICE
IV
SERVICE
V

30. 30 | Copyright © 2019
CONTROL EXPERIMENT
⍄ DEFINE EXPERIMENTS (SET OF: MESSAGE DELAYS, NETWORK FAULTS)
⍄ RUN EVERY INTERVAL (E.G. EVERY FRIDAY AT 9PM)
⍄ GATHERED METRICS – COMPARE BASELINE
⍄ STOP EXPERIMENT IF CONDITION REACHED

31. 31 | Copyright © 2019
GLOOSHOT
GLOOSHOT ALLOWS YOU TO PERFORM CHAOS EXPERIMENTS AT THE SERVICE MESH
LEVEL.
DEFINE ERROR CONDITIONS IN TERMS OF SUCH FAILURE MODES:
⍄ MESSAGE DELAYS
⍄ NETWORK FAULTS.
RUN EXPERIMENTS UNTIL A STOP CONDITION IS MET.
GLOOSHOT INTERFACES WITH ALL MAJOR SERVICE MESHES THROUGH SERVICE MESH
INTERFACE (SMI).

32. 32 | Copyright © 201932 | Copyright © 2019
GLOOSHOT DEMO

33. 33 | Copyright © 2019
DOCS: HTTPS://GLOOSHOT.SOLO.IO
GITHUB: HTTPS://GITHUB.COM/SOLO-IO/GLOOSHOT
COMMUNITY: HTTPS://SLACK.SOLO.IO

34. 34 | Copyright © 201934 | Copyright © 2019
WHAT IS THE BEST WAY TO START?

35. 35 | Copyright © 2019
SERVICE MESH HUB

36. 36 | Copyright © 2019

37. 37 | Copyright © 201937 | Copyright © 2019
SUMMARY

38. 38 | Copyright © 2019
SUMMARY
DEBUG CAPTURE FIND
SQUASH
DEBUGGER FOR
MICROSERVICES AND
SERVERLESS
RECORD, REPLY, AND
DEBUG ERRONEOUS
REQUESTS OUTSIDE
PRODUCTION
CHAOS ENGINEERING
FOR SERVICE MESH
LOOP GLOOSHOT
SHARE
FIRST INDUSTRY SERVICE
MESH HUB TO
ACCELERATE ADOPTION,
ADVANCE INNOVATION
AND FOSTER
COLLABORATION
SM HUB

39. 39 | Copyright © 2019
MORE FROM SOLO
GATEWAY GRAPHQL MESH
GLOO
AN ENVOY-POWERED API
GATEWAY, A K8S INGRESS, A
MIGRATION FACILITATOR, AND
MORE
PERFORM YOUR
QUERIES ACROSS ALL YOUR
APIS WITHOUT WRITING CODE
THE SIMPLEST WAY TO
INSTALL AND MANAGE
ONE OR MORE SERVICE
MESH TECHNOLOGIES.
SQOOP SUPERGLOO

40. 40 | Copyright © 201940 | Copyright © 2019
@SOLOIO_INC
HTTPS://GITHUB.COM/SOLO-IO
CHRISTIAN POSTA
@CHRISTIANPOSTA