This document discusses reliability patterns for microservices including retrying operations, handling transient faults, implementing a circuit breaker pattern, and providing fallbacks. It describes strategies like exponential backoff with jitter for retrying and using a circuit breaker to prevent repeatedly failing operations. The document concludes with a hands-on exercise to apply these patterns when requesting product details from multiple services.

1. FOR FUN AND PROFIT
R E L I A B I L I T Y
PATTERNS
D E V O P S L I S B O N 1 2 . 0 3 . 2 0 18
L I S B O N , P O RT U G A L
L U I S M I N E I R O
@voidmaze

2. WHAT'S THIS?

3. WHAT ABOUT THIS?

4. TYPICAL TRAFFIC BURST

5. What happens when the following operation fails?
Cart cart = restTemplate.getForObject(url, Cart.class);
H A N D L I N G FA U LT S

6. for (int i = 1; i <= numRetries; i++) {
try {
return restTemplate.getForObject(url, Cart.class);
} catch (RestClientException e) {
LOG.error("failed to get cart", e);
if (i >= numRetries) {
throw e;
}
}
}
R E T RY I N G

7. We should only retry if the problem is due to
a network failure or server overload
TRANSIENT FAULTS

8. for (int i = 1; i <= numRetries; i++) {
try {
return restTemplate.getForObject(url, Cart.class);
} catch (RestClientException e) {
LOG.error("failed to get cart", e);
if (i >= numRetries || isNonTransientFault(e)) {
throw e;
}
}
}
B E T T E R R E T RY

9. EVEN BETTER RET RY

10. long computeWaitTime(int retryNumber, int maxWaitTime) {
int delay = WAIT_TIME_MULTIPLIER * 2^retryNumber;
return min(maxWaitTime, delay + random.nextInt(delay));
}
...
for (int i = 1; i <= numRetries; i++) {
try {
return restTemplate.getForObject(url, Cart.class);
} catch (RestClientException e) {
LOG.error("failed to get cart", e);
if (i >= numRetries || isNonTransientFault(e)) {
throw e;
}
sleep(computeWaitTime(i, MAX_WAIT_TIME));
}
}
TRUNCATED EXPONENTIAL BACKOFF WITH JITTER

11. WHEN TRANSIENT FAULTS BECOME PERMANENT

12. C I R C U I T B R E A K E R PAT T E R N
The circuit breaker pattern can prevent an application from
repeatedly trying to execute an operation that's likely to fail

13. C I R C U I T B R E A K E R C L O S E D
Closed State
The requests from the application are forwarded to the target
TA R GET

14. C I R C U I T B R E A K E R C L O S E D
Closed State
The requests from the application are forwarded to the target
TA R GET

15. C I R C U I T B R E A K E R C L O S E D
Closed State
The requests from the application are forwarded to the target
TA R GET

16. C I R C U I T B R E A K E R C L O S E D
Closed State
The requests from the application are forwarded to the target
TA R GET

17. C I R C U I T B R E A K E R O P E N
Open State
The request from the application fails immediately and an
exception is returned to the application.
TA R GET

18. C I R C U I T B R E A K E R H A L F - O P E N
Half-Open State
A limited number of requests from the application are allowed
to pass through and invoke the operation.
TA R GET

19. C I R C U I T B R E A K E R H A L F - O P E N
Half-Open State
A limited number of requests from the application are allowed
to pass through and invoke the operation.
TA R GET

20. C I R C U I T B R E A K E R H A L F - O P E N
Half-Open State
A limited number of requests from the application are allowed
to pass through and invoke the operation.
TA R GET

21. C I R C U I T B R E A K E R O P E N
Open State
The request from the application fails immediately and an
exception is returned to the application.
TA R GET

22. private double doSomeMath(int result) {
if(result != 0) {
return 42 / result;
}
return Double.NaN;
}
THE MOST IMPORTANT QUESTION

23. PRODUCT DETAI L PAG E

24. PRODUCT DETAI L PAG E

25. PRODUCT DETAI L PAG E

26. circuitBreaker.call((url) -> {
for (int i = 1; i <= numRetries; i++) {
try {
return restTemplate.getForObject(url, Product.class);
} catch (RestClientException e) {
LOG.error("failed to get product details", e);
if (i >= numRetries || isNonTransientFault(e)) {
throw e;
}
sleep(computeWaitTime(i, MAX_WAIT_TIME));
}
}
throw new NoMoreRetriesException();
}).fallback(() -> "a Partner");
PUTTING IT ALL TOG ET HER

27. H A N D S - O N E X E R C I S E
R E T RY
C I R C U I T
BREAKER
FALLBACK
PRODUCT
D E TA I L PA G E
TA R GET

28. EXERCISE - PR OD UC T D ETA IL PAGE

29. EXERCISE - PR OD UC T D ETA IL PAGE
B R A N D D ATA
PRO DUCT DATA
WISH LIST
SI ZE RE COMME NDATION
C A R T
D E L I V E RY O P T I O N S
S IZ E SE LE CTO R

30. EXERCISE - PR OD UC T D ETA IL PAGE
GROUP 2: BR AN D D ATA
GROUP 3: PR OD U C T D ATA
GROUP 4: WISH LIST
GROUP 1: SI ZE R E C OMME NDAT IO N
GROUP 5: C A RT
GROUP 7: D E L I V E RY O P T I O N S
GROUP 6: SI ZE S E LE C TOR
GROUPS TASKS
1.Retries
Retryable operation?
How many times?
2.Circuit breaker
Global circuit breaker?
3.Fail fast
Type of fallback
Delegate to frontend?
2 0
MIN

31. THANK YOU
QUESTIONS?
WE'RE HIRING
https://jobs.zalando.com

32. @ZalandoTech
https://tech.zalando.com
LUIS MINEIRO
@voidmaze
luis@zalando.de
1 2 . 0 3 . 2 0 1 8