Fault tolerance in general is a challenging topic. Yet we need fault toleranct designs more badly than ever in order to provide robust, highly available systems - especially in times of scale out systems becoming more and more popular.
Unfortunately, most developers do not care too much about a fault tolerant design, either because they are scared by the complexity of the realm or because they do not care enough. One of the problems is that a lack of fault tolerant design does not hurt a lot in development or in QA, but it hurts a lot in production - as Michael Nygard said: "It's all about production!" (at least figuratively).
In this presentation I do *not* try to give a general introduction to fault tolerant design. Instead I pick a few generic case studies that demonstrate the results of missing fault tolerant design, try to sensitize a bit about the production relevance of fault tolerant design and then go along with a few selected patterns. I picked a few patterns which are surprisingly easy to implement and help to mitigate the problems of the former case studies.
This way I try to show two things:
1. A piece of architecture or design as a pattern is not necessarily hard to implement. Sometimes the code is written quicker than it takes to explain the pattern beforehand.
2. Even if fault tolerant design as a general topic might be hard, some parts of it can be implemented very easily and it's more than worth the coding effort if you look how much better your system behaves in production just from adding those few lines of code.
7. Timeouts (1)
// Basics
myObject.wait(); // Do not use this by default
myObject.wait(TIMEOUT); // Better use this
// Some more basics
myThread.join(); // Do not use this by default
myThread.join(TIMEOUT); // Better use this
8. Timeouts (2)
// Using the Java concurrent library
Callable<MyActionResult> myAction = <My Blocking Action>
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<MyActionResult> future = executor.submit(myAction);
MyActionResult result = null;
try {
result = future.get(); // Do not use this by default
result = future.get(TIMEOUT, TIMEUNIT); // Better use this
} catch (TimeoutException e) { // Only thrown if timeouts are used
...
} catch (...) {
...
}
12. Circuit Breaker (1)
Client
Resource
Circuit Breaker
Request
Resource unavailable
Resource available
Closed
Open
Half-Open
Lifecycle
13. Circuit Breaker (2)
Closed
on call / pass through
call succeeds / reset count
call fails / count failure
threshold reached / trip breaker
Open
on call / fail
on timeout / attempt reset
trip breaker
Half-Open
on call / pass through
call succeeds / reset
call fails / trip breaker
trip breaker
attempt reset
reset
Source: M. Nygard, „Release It!“
14. Circuit Breaker (3)
public class CircuitBreaker implements MyResource {
public enum State { CLOSED, OPEN, HALF_OPEN }
final MyResource resource;
State state;
int counter;
long tripTime;
public CircuitBreaker(MyResource r) {
resource = r;
state = CLOSED;
counter = 0;
tripTime = 0L;
}
...
15. Circuit Breaker (4)
...
public Result access(...) { // resource access
Result r = null;
if (state == OPEN) {
checkTimeout();
throw new ResourceUnavailableException();
}
try {
r = r.access(...); // should use timeout
} catch (Exception e) {
fail();
throw e;
}
success();
return r;
}
...
21. Fail Fast (2)
Client
Resources
Expensive Action
Request
Fail Fast Guard
Uses
Check availability
Forward
22. Fail Fast (3)
public class FailFastGuard {
private FailFastGuard() {}
public static void checkResources(Set<CircuitBreaker> resources) {
for (CircuitBreaker r : resources) {
if (r.getState() != CircuitBreaker.CLOSED) {
throw new ResourceUnavailableException(r);
}
}
}
}
23. Fail Fast (4)
public class MyService {
Set<CircuitBreaker> requiredResources;
// Initialize resources
...
public Result myExpensiveAction(...) {
FailFastGuard.checkResources(requiredResources);
// Execute core action
...
}
}
27. Shed Load (2)
Server
Too many Requests
Gate Keeper
Monitor
Requests
Request Load Data
Monitor Load
Shedded Requests
Clients
28. Shed Load (3)
public class ShedLoadFilter implements Filter {
Random random;
public void init(FilterConfig fc) throws ServletException {
random = new Random(System.currentTimeMillis());
}
public void destroy() {
random = null;
}
...
29. Shed Load (4)
...
public void doFilter(ServletRequest request,
ServletResponse response,
FilterChain chain)
throws java.io.IOException, ServletException {
int load = getLoad();
if (shouldShed(load)) {
HttpServletResponse res = (HttpServletResponse)response;
res.setIntHeader("Retry-After", RECOMMENDATION);
res.sendError(HttpServletResponse.SC_SERVICE_UNAVAILABLE);
return;
}
chain.doFilter(request, response);
}
...
37. // Do or wait variant
ProcessingState state = initBatch();
while(!state.done()) {
int load = getLoad();
if (load > THRESHOLD) {
waitFixedDuration();
} else {
state = processNext(state);
}
}
void waitFixedDuration() {
Thread.sleep(DELAY); // try-catch left out for better readability
}
Deferrable Work (3)
38. // Adaptive load variant
ProcessingState state = initBatch();
while(!state.done()) {
waitLoadBased();
state = processNext(state);
}
void waitLoadBased() {
int load = getLoad();
long delay = calcDelay(load);
Thread.sleep(delay); // try-catch left out for better readability
}
long calcDelay(int load) { // Simple example implementation
if (load < THRESHOLD) {
return 0L;
}
return (load – THRESHOLD) * DELAY_FACTOR;
}
Deferrable Work (4)
50. Further reading
1. Michael T. Nygard, Release It!,
Pragmatic Bookshelf, 2007
2. Robert S. Hanmer,
Patterns for Fault Tolerant Software,
Wiley, 2007
3. James Hamilton, On Designing and
Deploying Internet-Scale Services,
21st LISA Conference 2007
4. Andrew Tanenbaum, Marten van Steen,
Distributed Systems – Principles and
Paradigms,
Prentice Hall, 2nd Edition, 2006