Recently, ArangoDB integrated its cluster management with Apache Mesos. This makes it now possible to launch an ArangoDB cluster on a Mesos cluster with a single, albeit complex shell command. In a DCOS-enabled Mesosphere cluster this is even easier, because one can use the dcos subcommand for ArangoDB, which essentially turns a Mesosphere cluster into a single, large computer. In this talk I explain the whole setup and show (live on stage) how to deploy ArangoDB clusters on Amazon Web Services, and how we used this to scale ArangoDB up until it could sustain 1000000 document writes per second.

1. Scaling ArangoDB on
Mesosphere DCOS
Max Neunhöffer
Hamburg, 1 October 2015
www.arangodb.com

2. Features
is a multi-model database (document store & graph database),

3. Features
is a multi-model database (document store & graph database),
offers convenient queries (via HTTP/REST and AQL),

4. Features
is a multi-model database (document store & graph database),
offers convenient queries (via HTTP/REST and AQL),
including joins between different collections,

5. Features
is a multi-model database (document store & graph database),
offers convenient queries (via HTTP/REST and AQL),
including joins between different collections,
configurable consistency guarantees using transactions

6. Features
is a multi-model database (document store & graph database),
offers convenient queries (via HTTP/REST and AQL),
including joins between different collections,
configurable consistency guarantees using transactions
API extensible by JS code in the Foxx Microservice Framework.

7. Replication and Sharding — horizontal scalability
ArangoDB provides
easy setup of (asynchronous) replication,
sharding with automatic data distribution
MongoDB-style replication in the cluster,
full integration with Apache Mesos and Mesosphere.

8. Replication and Sharding — horizontal scalability
ArangoDB provides
easy setup of (asynchronous) replication,
sharding with automatic data distribution
MongoDB-style replication in the cluster,
full integration with Apache Mesos and Mesosphere.
Work in progress:
synchronous replication in cluster mode,

9. Replication and Sharding — horizontal scalability
ArangoDB provides
easy setup of (asynchronous) replication,
sharding with automatic data distribution
MongoDB-style replication in the cluster,
full integration with Apache Mesos and Mesosphere.
Work in progress:
synchronous replication in cluster mode,
fault tolerance by automatic failover and

10. Replication and Sharding — horizontal scalability
ArangoDB provides
easy setup of (asynchronous) replication,
sharding with automatic data distribution
MongoDB-style replication in the cluster,
full integration with Apache Mesos and Mesosphere.
Work in progress:
synchronous replication in cluster mode,
fault tolerance by automatic failover and
zero administration by a self-reparing and self-balancing cluster architecture,

11. Replication and Sharding — horizontal scalability
ArangoDB provides
easy setup of (asynchronous) replication,
sharding with automatic data distribution
MongoDB-style replication in the cluster,
full integration with Apache Mesos and Mesosphere.
Work in progress:
synchronous replication in cluster mode,
fault tolerance by automatic failover and
zero administration by a self-reparing and self-balancing cluster architecture,
all based on the Apache Mesos infrastructure.

12. Task Task
Task Task
Task Task
Task Task
TaskTask
Task Task
TaskTask
Task Task
TaskTask
Task
TaskTask
Task
Task Task
TaskTask
Task
Mesos Agent Mesos Agent
Mesos Agent Mesos Agent
Mesos Agent
Mesos Agent
Mesos Agent
Mesos Agent
Mesos Master Mesos Master Mesos Master
(leader)
(leader)
Zookeeper Zookeeper Zookeeper

13. Task Task
Task Task
Task Task
Task Task
TaskTask
Task Task
TaskTask
Task Task
TaskTask
Task
TaskTask
Task
Task Task
TaskTask
Task
Mesos Agent Mesos Agent
Mesos Agent Mesos Agent
Mesos Agent
Mesos Agent
Mesos Agent
Mesos Agent
Mesos Master
Zookeeper
Mesos Master
(leader)
(leader)
Zookeeper
Mesos Master
Zookeeper

14. Task Task
Task
Task Task
TaskTask
Task Task
TaskTask
Task Task
TaskTask
Task
TaskTask
Task
Task Task
TaskTask
Task
Task
Task Task
Task Task
Task
Mesos Agent
Mesos Agent Mesos Agent
Mesos Agent
Mesos Agent
Mesos Agent
Mesos Agent
Mesos Master Mesos Master Mesos Master
(leader)
(leader)
Zookeeper Zookeeper Zookeeper
Mesos Agent

15. dcos CLI Marathonschedules frameworks
starts (this is a lie)
Mesos Agent Mesos Master
Zookeeper
registers
stores state
Framework
Task

16. dcos CLI Marathonschedules frameworks
1. reports free resources
4. tells to execute
3. accepts or
resource offers
2. makes
declines them
starts (this is a big lie)
executes
(this is a small lie)
Mesos Agent Mesos Master
Zookeeper
Framework
Task

17. dcos CLI Marathonschedules frameworks
1. reports free resources
4. tells to execute
3. accepts or
resource offers
2. makes
declines them
executes
(this is a small lie)
actually, Marathon is a framework
starts (this is a big lie)
actually, it uses an "executor"
Mesos Agent Mesos Master
Zookeeper
Framework
Task

18. dcos CLI Marathonschedules frameworks
starts
Mesos Agent Mesos Master
Zookeeper
Framework
Task
notices
reports
reports

19. dcos CLI Marathonschedules frameworks
restarts
Mesos Agent Mesos Master
Zookeeper
Framework
Task

20. dcos CLI Marathonschedules frameworks
restarts
Mesos Agent Mesos Master
Zookeeper
Framework
Task
gets state
and reconciles
reconnects

21. Persistent primitives
offer received 20151001-105738-2905319616-5050-2640-O0 with
cpus(*):4; mem(*):10895; disk(*):119761; ports(*):[31000-32000]}

22. Persistent primitives
offer received 20151001-105738-2905319616-5050-2640-O0 with
cpus(*):4; mem(*):10895; disk(*):119761; ports(*):[31000-32000]}
trying to reserve 20151001-105738-2905319616-5050-2640-O0 with
cpus(arangodb, pri):0.2; mem(arangodb, pri):512; disk(arangodb, pri):512}

23. Persistent primitives
offer received 20151001-105738-2905319616-5050-2640-O0 with
cpus(*):4; mem(*):10895; disk(*):119761; ports(*):[31000-32000]}
trying to reserve 20151001-105738-2905319616-5050-2640-O0 with
cpus(arangodb, pri):0.2; mem(arangodb, pri):512; disk(arangodb, pri):512}
offer received 20151001-105738-2905319616-5050-2640-O1 with
cpus(*):3.8; mem(*):10383; disk(*):119249; ports(*):[31000-32000];
cpus(arangodb, pri):0.2; mem(arangodb, pri):512; disk(arangodb, pri):512}

24. Persistent primitives
offer received 20151001-105738-2905319616-5050-2640-O0 with
cpus(*):4; mem(*):10895; disk(*):119761; ports(*):[31000-32000]}
trying to reserve 20151001-105738-2905319616-5050-2640-O0 with
cpus(arangodb, pri):0.2; mem(arangodb, pri):512; disk(arangodb, pri):512}
offer received 20151001-105738-2905319616-5050-2640-O1 with
cpus(*):3.8; mem(*):10383; disk(*):119249; ports(*):[31000-32000];
cpus(arangodb, pri):0.2; mem(arangodb, pri):512; disk(arangodb, pri):512}
trying to make 20151001-105738-2905319616-5050-2640-O1 persistent for
disk(arangodb, pri)[AGENCY_c2bb93ce-46d3-4802-bed5-cf254c6f16df:dataxyz]:512

25. Persistent primitives
offer received 20151001-105738-2905319616-5050-2640-O0 with
cpus(*):4; mem(*):10895; disk(*):119761; ports(*):[31000-32000]}
trying to reserve 20151001-105738-2905319616-5050-2640-O0 with
cpus(arangodb, pri):0.2; mem(arangodb, pri):512; disk(arangodb, pri):512}
offer received 20151001-105738-2905319616-5050-2640-O1 with
cpus(*):3.8; mem(*):10383; disk(*):119249; ports(*):[31000-32000];
cpus(arangodb, pri):0.2; mem(arangodb, pri):512; disk(arangodb, pri):512}
trying to make 20151001-105738-2905319616-5050-2640-O1 persistent for
disk(arangodb, pri)[AGENCY_c2bb93ce-46d3-4802-bed5-cf254c6f16df:dataxyz]:512
offer received 20151001-105738-2905319616-5050-2640-O2 with
cpus(*):3.8; mem(*):10383; disk(*):119249; ports(*):[31000-32000];
cpus(arangodb, pri):0.2; mem(arangodb, pri):512;
disk(arangodb, pri)[AGENCY_c2bb93ce-46d3-4802-bed5-cf254c6f16df:dataxyz]:512

26. Deployment
Docker and github

27. Deployment
Docker and github
One container image arangodb/arangodb-mesos used to run
the ArangoDB framework (C++ executable)
all ArangoDB instances in the cluster
the Agency (etcd)

28. Deployment
Docker and github
One container image arangodb/arangodb-mesos used to run
the ArangoDB framework (C++ executable)
all ArangoDB instances in the cluster
the Agency (etcd)
The dcos CLI by Mesosphere is a Python program (virtualenv, pip).

29. Deployment
Docker and github
One container image arangodb/arangodb-mesos used to run
the ArangoDB framework (C++ executable)
all ArangoDB instances in the cluster
the Agency (etcd)
The dcos CLI by Mesosphere is a Python program (virtualenv, pip).
ArangoDB subcommand: a Python program, talks JSON/REST with the
framework, plugs into dcos, deployed from a github repository.

30. Deployment
Docker and github
One container image arangodb/arangodb-mesos used to run
the ArangoDB framework (C++ executable)
all ArangoDB instances in the cluster
the Agency (etcd)
The dcos CLI by Mesosphere is a Python program (virtualenv, pip).
ArangoDB subcommand: a Python program, talks JSON/REST with the
framework, plugs into dcos, deployed from a github repository.
github repository mesosphere/universe has all certified frameworks

31. Scaling ArangoDB
Ultimate aim with a distributed database: horizontal scalability.

32. Scaling ArangoDB
Ultimate aim with a distributed database: horizontal scalability.
Devise a test, . . .
to show linear scaling
use N = 8, 16, 24, 32, 40, 48, 56, 64, 72, 80 nodes with 8 vCPUs each.
run N/2 DBServers, N/2 asynchronous replicas and N/2 Coordinators.

33. Scaling ArangoDB
Ultimate aim with a distributed database: horizontal scalability.
Devise a test, . . .
to show linear scaling
use N = 8, 16, 24, 32, 40, 48, 56, 64, 72, 80 nodes with 8 vCPUs each.
run N/2 DBServers, N/2 asynchronous replicas and N/2 Coordinators.
use single document reads, writes and 50%/50%,
from N/2 load servers in the same Mesosphere cluster

34. Scaling ArangoDB
Ultimate aim with a distributed database: horizontal scalability.
Devise a test, . . .
to show linear scaling
use N = 8, 16, 24, 32, 40, 48, 56, 64, 72, 80 nodes with 8 vCPUs each.
run N/2 DBServers, N/2 asynchronous replicas and N/2 Coordinators.
use single document reads, writes and 50%/50%,
from N/2 load servers in the same Mesosphere cluster
up to 640 vCPUs, want to write as many k docs/(s * vCPU) as possible.

35. Deployment of load servers

36. Deployment of load servers
Docker and ArangoDB

37. Deployment of load servers
Docker and ArangoDB
Use a central ArangoDB instance to
collect results,
evaluate them,
and synchronise load servers.

38. Deployment of load servers
Docker and ArangoDB
Use a central ArangoDB instance to
collect results,
evaluate them,
and synchronise load servers.
Each load server runs the Waiter in a Docker container.

39. Deployment of load servers
Docker and ArangoDB
Use a central ArangoDB instance to
collect results,
evaluate them,
and synchronise load servers.
Each load server runs the Waiter in a Docker container.
The Waiter

40. Deployment of load servers
Docker and ArangoDB
Use a central ArangoDB instance to
collect results,
evaluate them,
and synchronise load servers.
Each load server runs the Waiter in a Docker container.
The Waiter waits,

41. Deployment of load servers
Docker and ArangoDB
Use a central ArangoDB instance to
collect results,
evaluate them,
and synchronise load servers.
Each load server runs the Waiter in a Docker container.
The Waiter waits, most of the time,
observes a collection and notices new "work"documents,
fires up load processes,
reports termination as a "done"document.

42. Deployment of load servers
Docker and ArangoDB
Use a central ArangoDB instance to
collect results,
evaluate them,
and synchronise load servers.
Each load server runs the Waiter in a Docker container.
The Waiter waits, most of the time,
observes a collection and notices new "work"documents,
fires up load processes,
reports termination as a "done"document.
A single JavaScript program directs the whole experiment.

43. Deployment of load servers
Docker and ArangoDB
Use a central ArangoDB instance to
collect results,
evaluate them,
and synchronise load servers.
Each load server runs the Waiter in a Docker container.
The Waiter waits, most of the time,
observes a collection and notices new "work"documents,
fires up load processes,
reports termination as a "done"document.
A single JavaScript program directs the whole experiment.
We deploy the Waiter using Marathon.

44. DEMO?

45. Links
https://www.arangodb.com
https://docs.arangodb.com/cookbook/index.html
https://github.com/ArangoDB/guesser
http://mesos.apache.org/
https://mesosphere.com/
https://mesosphere.github.io/marathon/