SHARDING WITH AKKA CLUSTER: FROM THEORY TO PRODUCTION Writing distributed applications is very hard, especially when you start developing them as single-noded ones. Programmers tend to focus on functionalities first, leaving the scalability issues for later. Fortunately, Akka gives us many tools for scaling out and we can use them very early in the development process. You will learn how to take advantage of these features. The talk is divided into two sections. In the first section you will learn how to transform a single-noded app into a scalable one. During live coding session we will create both versions from scratch and guide you through the most important architectural decisions. Then we will move on to more important problems. During the second section we would like to simulate missile defence system and protect planet Earth against hostile aliens. Under the hood there will be Scala and Akka Cluster with sharding extension. After describing the application and its internals, we will simulate network and power failures to provide some idea how fast the application can recover.

1. Sharding with Akka Cluster
From Theory to Production
Michał Płachta
@miciek
Krzysztof Otrębski
@kotrebski

2. What’s inside?
● Creating web service using actor model
● ...making it scalable
● Playing with more complex app
● ...validating its fault tolerance

3. Akka Tutorial
● actor ~= thread
● actorRef.tell
● actorRef.ask
● actors create children
ActorRef
Sender 1 Sender 2
ask tell
enqueue
MailboxActor
dequeue

4. Scala Tutorial
● case class
● pattern matching
case class Junction(id: Int)
public class Junction {
private final int id;
public Junction(int id) {
this.id = id;
}
public int getId() {
return id;
}
// hashCode
// equals
// copy
}
msg match {
case Junction(id) =>
// this will execute
// when msg is Junction
case SomeOtherType =>
}

5. First example: Sorter
scan <containerId> -> HTTP -> push right or not
http://i.imgur.com/mctb4HC.gifv

6. Sorter Web Service
http://localhost:8080/junctions/<junctionId>/decisionForContainer/<containerId>
returns JSON
{ “direction”: left | right | straight | ... }
Assumptions:
● 5-10 ms to make a decision
● business logic defined - focus on performance

7. Let’s code it!

8. Step 1: Just REST...
RestInterface
HTTP Requests HTTP Responses
● One Actor = One Thread
● Blocking inside receive method
● Low throughput...

9. Throughput testing
/junctions/1/decisionForContainer/1
/junctions/2/decisionForContainer/4
/junctions/3/decisionForContainer/5
/junctions/4/decisionForContainer/2
/junctions/5/decisionForContainer/7
2000 requests
2000 requests
2000 requests
2000 requests
2000 requests
in parallel
cat URLs.txt | parallel -j 5 'ab -ql -n 2000 -c 1 -k {}'
GNU Parallel ApacheBench

10. Let’s test it!

11. Step 1: Just REST...
RestInterface
HTTP Requests HTTP Responses
± % cat URLs.txt | parallel -j 5 'ab -ql -n 2000 -c 1 -k {}' | grep 'Requests per second'
Requests per second: 34.78 [#/sec] (mean)
Requests per second: 34.22 [#/sec] (mean)
Requests per second: 33.77 [#/sec] (mean)
Requests per second: 33.82 [#/sec] (mean)
Requests per second: 33.98 [#/sec] (mean)

12. Let’s improve
performance!

13. Step 1.5: Logic in another actor
RestInterface
HTTP Requests HTTP Responses
SortingDecider

14. Step 2: One actor per junction
RestInterface
HTTP Requests HTTP Responses
DecidersGuardian
SortingDecider
SortingDecider
SortingDecider
<junctionId>=1 ... <junctionId>=5

15. Step 2: One actor per junction
± % cat URLs.txt | parallel -j 5 'ab -ql -n 2000 -c 1 -k {}' | grep 'Requests per second'
Requests per second: 67.36 [#/sec] (mean)
Requests per second: 69.03 [#/sec] (mean)
Requests per second: 67.75 [#/sec] (mean)
Requests per second: 66.88 [#/sec] (mean)
Requests per second: 66.28 [#/sec] (mean)

16. Now what?
● non-blocking
● concurrent
● scaling up works
● scaling out?
RestInterface
HTTP Requests HTTP Responses
DecidersGuardian
SortingDecider
SortingDecider
SortingDecider
<junctionId>=1 ... <junctionId>=5

17. Manual scaling out
RestInterface
HTTP Requests HTTP Responses
DecidersGuardian
SortingDecider
SortingDecider
SortingDecider
<junctionId>=1 ... <junctionId>=3
RestInterface
HTTP Requests HTTP Responses
DecidersGuardian
SortingDecider
SortingDecider
SortingDecider
<junctionId>=4 ... <junctionId>=6

18. Enter Sharding
RestInterface
HTTP Requests HTTP Responses
ShardRegion
SortingDecider
SortingDecider
SortingDecider
<junctionId>=h(m) ... <junctionId>=h(m)
RestInterface
HTTP Requests HTTP Responses
ShardRegion
SortingDecider
SortingDecider
SortingDecider
<junctionId>=h(m) ... <junctionId>=h(m)
...

19. Let’s shard it!

20. Step 3: Sharded web service
± % cat URLs.txt | parallel -j 5 'ab -ql -n 2000 -c 1 -k {}' | grep 'Requests per second'
Requests per second: 106.80 [#/sec] (mean)
Requests per second: 108.15 [#/sec] (mean)
Requests per second: 100.60 [#/sec] (mean)
Requests per second: 99.92 [#/sec] (mean)
Requests per second: 100.07 [#/sec] (mean)

21. Sharding
● automatic distribution
● no need to know who is where
● no need to know how many nodes are there
● rebalancing
● migration

22. Alien Invasion

23. https://it.wikipedia.org/wiki/File:Richard-feynman.jpg

24. ...it seems to have been the general policy then to
not try to repeat psychological experiments,
Richard Feynman

25. How does failover
work on
production?

26. What is on the screen

27. The structure of the cluster

28. Some important cluster settings
akka {
cluster {
auto-down-unreachable-after = 20s
}
}

29. Let's see some
action

30. What you know now
How Akka Cluster and Sharding works
It's good to run failover tests before going on protoduction
Close your nodes gracefully

31. How can I run those apps?
● https://github.com/miciek/akka-sharding-example
● https://github.com/otrebski/reactive-missile-defend

32. Thank you!
Any questions?
Michał Płachta
@miciek
Krzysztof Otrębski
@kotrebski