A presentation about deploy, scaling and the coordination problem. We will focus on redis as a coordination system in order to simplify the migration to ETCd as coordination system

1. CloudConf
Turin 16th March 2017
Deploy, scale and coordinate a microservice oriented
application
cloudconf.it

2. Walter Dal Mut
walter.dalmut @ corley.it
Solution Architect @ Corley

3. Microservices

4. They're coming outta the walls

5. Docker Swarm

6. Issue a new process (deploy)
$ docker service create
--name app
registry.walterdalmut.com/app:v1
$ ./app &

7. Run different processes (scale)
$ docker service scale app=10 $ ./app &
$ ./app &
$ ./app &

8. What about the locking system?
In a distributed system: locks (distributed locks) are
the foundations for activity synchronization

9. What about coordination?

10. A service have its own con guration at
launch
I am here with this address, port etc...
A service require other service
con gurations
Where the database is, which password i should use, etc...
The service should reports its owns status
I am alive and responsive (for healthcheck)
other services can check the health report for the maintenance mode
or to shortcircuit the service dependency

11. To expose the coordination problem we create an
application

12. Read the twitter stream
#cloudconf2017

13. users that tweet with this handle
create a reserved API service [ JSON over HTTP ]
GET /tweet - list my tweets
POST /tweet - record a new
tweet
The database to store tweets is self-contained in the API service

14. Now every service is exposed with a unique pair
address:port in the swarm

15. every box is a container (service)
blinks for activities (publish new tweets)

16. Multiple services
(1)
(1)
(1..*)
(n)
(1..*)
Stream readerer
Distributed queue
Service Worker (need a distributed lock)
a lock identi es the service deployment
progress
on missing => service deploy
on existing => publish messages
Per user container
A proxy to list users and redirects requests

17. HTTP framework
> GET /user/walterdalmut HTTP/1.1
> Host: cluster.corsi.walterdalmut.com:30000
> User-Agent: curl/7.47.0
> Accept: */*
< HTTP/1.1 302 Found
< location: http://cluster.corsi.walterdalmut.com:30002/v1/tweet
< vary: origin
< cache-control: no-cache
< content-length: 0
< Date: Sun, 12 Mar 2017 11:26:50 GMT
< Connection: keep-alive

18. CloudConf2017 Example

21. Every user have its own network address and
port
Every user expose its own API
How do we connect services together?

22. DNS as a coordination system
DNS is a good solution to point things in a network
DNS SRV expose a service address con guration
$ dig srv _auth._tcp.walterdalmut.com +short
1 10 8080 1.api.walterdalmut.com
1 10 8080 2.api.walterdalmut.com

23. And service con gurations?
host: db.mynet.local
port: 3306
username: root
password: root
dbname: example

24. K / V
Several coordination systems available
Etcd is one of the most interesting coordination system available
Consul integrates di erent things together like: DNS, KV, etc...
many other: zookeeper, etc...

25. Redis as a coordination service

26. distribute con gurations at paths
$ cat mydb.conf | redis-cli set /path/to/disk/mydb.conf -

27. Where is my `ls` command now?
$ redis-cli keys /path/*
1) "/path/to/disk/mydb.conf"

28. Get my con guration back
$ redis-cli get /path/to/disk/mydb.conf
host: db.mynet.local
port: 3306
username: root
password: root
dbname: example

29. How to report the application status?
Healthchecks

30. Con gurations can also expires
Dead man switch
application reports continuously
cat mydb.conf | SETEX /path/to/disk/mydb.conf 30 -
EXPIRE /path/to/disk/mydb.conf 30
... sleep 20
EXPIRE /path/to/disk/mydb.conf 30
...

31. Services links together
Can i watch for con guration changes?
refresh my services on updates

32. Redis Keyspace Noti cations
or in your con guration le
CONFIG SET notify-keyspace-events AKE

33. Listen for my con guration changes
SUBSCRIBE __keyspace@0__:/path/to/disk/mydb.conf

34. Here the event
$ cat mydb.conf | redis-cli set /path/to/disk/mydb.conf -
1) "message"
2) "__keyspace@0__:/path/to/disk/mydb.conf"
3) "set"

35. Distributed locks
In a single node for redis
NX - if not exists
PX 30000 - expires in 30000 ms
SET /etc/lock/username/.lock {random_value} NX PX 30000

36. After 30 seconds the lock expires
SET /etc/lock/walterdalmut/.lock 3891573 NX PX 30000
OK
SET /etc/lock/walterdalmut/.lock 2857152 NX PX 30000
(nil)

37. How do i release the lock?
DEL /etc/lock/walterdalmut/.lock

38. How do i extend the lock?
EXPIRE /etc/lock/walterdalmut/.lock 30

39. How do i watch for lock release?
Lock releases
SUBSCRIBE __keyspace@0__:/etc/lock/walterdalmut/.lock
1) "message"
2) "__keyspace@0__:/etc/lock/walterdalmut/.lock"
3) "del"
1) "message"
2) "__keyspace@0__:/etc/lock/walterdalmut/.lock"
3) "expired"

40. Thank you for listening