Building add-ons for Atlassian products today means building a Connect add-on and running it as a service in your own infrastructure, or a PaaS provider’s infrastructure, or (more commonly) a set of microservices. While this has many benefits, the transition from monolithic to distributed systems brings with it additional failure modes that simply do not manifest in the world of local function calls. Join Atlassian developer Diego Berrueta for a walk-through of 5 resilience techniques that will help keep your services rock-solid in the face of unreliable, slow, or faulty systems. Diego Berrueta, Engineering Principal, Atlassian

1. 5 Essential Techniques for
Building Fault-tolerant Systems
DIEGO BERRUETA | ENGINEERING PRINCIPAL | ATLASSIAN

2. Preston Rhea; Flickr (www.flickr.com/photos/prestonrhea/), CC-by

3. ALERT!

5. Incident response

6. REVERSE PROXY
CASCADING FAILURE

7. CASCADING FAILURE
REVERSE PROXY

8. aotaro; Flickr (www.flickr.com/photos/aotaro/), CC-by

9. Fault-tolerant system
Continues to operate in the event of faults in some
of its components
Fault
Deviation from the normal state, either due to
internal or external causes
Failure
Observable impact of a fault in a system, usually
manifests as reduced availability

10. Fault-tolerant system
Continues to operate in the event of faults in some
of its components
Fault
Deviation from the normal state, either due to
internal or external causes
Failure
Observable impact of a fault in a system, usually
manifests as reduced availability

11. Fault-tolerant system
Continues to operate in the event of faults in some
of its components
Fault
Deviation from the normal state, either due to
internal or external causes
Failure
Observable impact of a fault in a system, usually
manifests as reduced availability

12. Bob Yeats (OSU); Flickr (www.flickr.com/photos/oregonstateuniversity/), CC-by
Faults happen
Preventing them may be technically or
economically impractical

13. Systems can be designed and
built to tolerate faults
FAULT TOLERANCE

14. STABLE SYSTEM UNSTABLE SYSTEM

15. Essential
techniques
Contain
Fail fast
Escape
Adjust
Learn

16. Contain the fault
Using physical or logical barriers

17. Norlando Pobre; Flickr (www.flickr.com/photos/npobre/), CC-by

18. Shawn O’Neil; Flickr (www.flickr.com/photos/oneilsh/), CC-by

19. Pool separation
Different pools for each
task or dependency
Brian Cantoni; Flickr (www.flickr.com/photos/cantoni/), CC-by

20. Asynchronous communication
Sender and receiver fail independently,
receiver can catch up later
Synchronous communication
Propagates failures, context may be lost

21. Contain:
in practice

22. CLIENT ISOLATION
NEW CLIENT
CLIENT A
CLIENT B

23. Avoid SPOF
Find the components which
compromise the system
How to contain faults
Invest in redundancy
Improve availability by having
more than one of everything
Build bulkheads
Set up logic walls to
reduce the blast radius

24. Essential
techniques
Contain
Fail fast
Escape
Adjust
Learn

25. A quick error is better
than a slow response
FAIL FAST PRINCIPLE

26. Validate early
Anticipate problems and
change course
Carl Wycoff; Wikimedia Commons, CC-by

27. Reject politely
Fast refusal is better
than slow error
Taber A.B.; Flickr (www.flickr.com/photos/andrewbain/), CC-by

28. Never wait long
Protect long tasks
with timeouts
Robert C-B.; Flickr (www.flickr.com/photos/29233640@N07/), CC-by

29. Watch out
for slowness
In-process locks
Database queries
Sockets
Remote APIs
Albert Herring, Wikimedia Commons, CC-by

30. Watch out
for slowness
In-process locks
Database queries
Sockets
Remote APIs
Albert Herring, Wikimedia Commons, CC-by

31. Watch out
for slowness
In-process locks
Database queries
Sockets
Remote APIs
Albert Herring, Wikimedia Commons, CC-by

32. Watch out
for slowness
In-process locks
Database queries
Sockets
Remote APIs
Albert Herring, Wikimedia Commons, CC-by

33. Fail fast:
in practice

34. DATABASE FAILOVER
SYNC

35. Decline service
When overloaded, ask clients
to come back later
How to fail fast
Never wait long
Set a timeout for blocking
calls and slow operations
Validate early
Avoid starting something that
cannot be completed

36. Essential
techniques
Contain
Fail fast
Escape
Adjust
Learn

37. Circuit breakers
Separate the failing parts

38. REST CIRCUIT-BREAKER
CLIENT SERVER

39. Fault-tolerance libraries
Circuit breaking
Avoid cascading failures during
periods of turbulence
Monitoring and alerting
Observe the behaviour of all your
dependencies
Timeouts
Time-bound any operation
Fall-back
Recover using an alternative path

40. Escape:
in practice

41. SERVICE EVOLUTION
TENANT ATENANT A
TENANT ATENANT B
TENANT ATENANT C

42. Anticipate failure
If it is not going to work,
do not even try
How to escape
Degrade gracefully
A cached result or a default
value may be an alternative
Detect problems
Compare all interactions
against error thresholds

43. Essential
techniques
Contain
Fail fast
Escape
Adjust
Learn

44. Communication
Share initial expectations
and status updates

45. Adjust
Page size
Growth
Flow
Retry
Paginate queries
Avoid unbounded database and
remote queries

46. Adjust
Page size
Growth
Flow
Retry
Beware of things that grow
Clean up old data and set size limits

47. Adjust
Page size
Growth
Flow
Retry
Negotiate speed
Adjust dynamically using rate limits,
request throttling, buffering and
autoscaling

48. Adjust
Page size
Growth
Flow
Retry
Insist smartly
Retry transient errors with timeouts
and back-off

49. Adjust:
in practice

50. KILLER HEALTH CHECK

51. Insist smartly
Transient errors can be
retried with back-off
How to adjust
Report availability
Apply back pressure to
prevent congestion
Negotiate size
Limit the cost of the job

52. Essential
techniques
Contain
Fail fast
Escape
Adjust
Learn

53. Monitor the behaviour
Define metrics, observe trends, set up
alerts, capture logs
Learn from
experience
Observe
Test
Reflect

54. Test with chaos
Verify your hypothesis about
fault-tolerance by continuously
introducing chaos
Learn from
experience
Observe
Test
Reflect

55. Learn from mistakes
Each incident is an opportunity to
make the system more robust
Learn from
experience
Observe
Test
Reflect

56. Learn:
in practice

57. PARTIAL FAILURE
1
2
3

58. PARTIAL FAILURE
1 32
2
2

59. Monitor and alert
Understand the behaviour of
the system in production
How to learn
Reflect on incidents
Analyse the root cause and
prevent recurrences
Test what if…?
Deliberately introduce chaos
to assess fault-tolerance

60. Essential
techniques
Contain
Fail fast
Escape
Adjust
Learn

61. Hope is not a strategy
Test, observe, reflect
Life starts after releasing
“Code complete” is not “production ready”
Be cynical
Do not trust anybodyRobustness
is an attitude

62. Some disasters
can be prevented
Build and test with failure in mind
Faults are unavoidable
Any possible fault
will eventually happen

63. Leo Hidalgo; Flickr (www.flickr.com/photos/ileohidalgo/), CC-by

64. Release it!
Michael Nygard
Site Reliability
Engineering
Google
References

65. Thank you!
DIEGO BERRUETA | ENGINEERING PRINCIPAL | ATLASSIAN