How does Mesos Integrate with External Volumes

1. 1
Docker Volume Isolator In Mesos
Jie Yu, Gilbert Song, Guangya Liu

2. 2
Gilbert Song
Distributed Systems Engineer
Mesosphere
Guangya LiuJie Yu
Distributed Systems Engineer
Apache Mesos PMC & Committer
Mesosphere
Senior Software Engineer
IBM

3. 3
If your containers are “cattle”, you have nothing to
lose...
The test: I don’t need persistence beyond the life of
the container instance or host.
But what to do if your containers are pet, if you
containers need to persist data?
Motivation

4. 4
Stateful Service:
A service that needs to persist data/state
A typical stateful service
● Use external volume services
● Stateless to Mesos
Stateful services (like databases) need
durable and scalable state
Motivation

5. 5
There are many docker
images on DockerHub are
databases...
Motivation

6. 6
Current state of storage solution in `MesosContainerizer`
Local
● Persistent Volume
Master
Mesos Agent
Aurora Executor
Tasks
mysqld
datanode
Hadoop Executor
Tasks
Operating System

7. 7
But local is not enough!!
Local
● Availability Risk
○ Migrate container to another host –
your storage is gone
○ Host goes down – your service goes
down
● Scale Limitation
○ Need more storage than the host has?
Sorry…

8. 8
What we would like?
External Distributed Volume for stateful service
Master
Mesos Agent
Aurora Executor
Tasks
mysqld
datanode
Hadoop Executor
Tasks
Operating System

9. 9
The challenges
+ others
Give operator the freedom to choose different external storage platforms.

10. 10
Docker with External Volume
IBM Spectrum Scale AWS GCE OpenStack
Docker Volume Driver

11. 11
Mesos with External Volume
IBM Spectrum Scale AWS GCE OpenStack
Docker Volume Driver

12. Docker Volume Isolator
Docker Volume Isolator
IBM Spectrum Scale AWS GCE OpenStack
Docker Volume Driver
● A driver based model
● Docker Volume Isolator takes care of
business intelligence:
○ Mount with driver name, volume name and
some options.
○ Unmount with driver name and volume name.
12

13. 13
Docker Volume Isolator Architecture
HTTP/JSON UNIX/TCP SOCKETS
for Docker Volume API
Docker Volume Driver Plugin
(Convoy, Rex-ray, Flocker)
Docker Volume
Driver Client
Docker Volume
Isolator
dvdcli
Docker Daemon
(Optional)
Docker Volumes
Mount: dvdcli mount –volumedriver=convoy –volumename=vol1 --volumeopts=size=5 --volumeopts=iops=150
Unmount: dvdcli unmount --volumedriver=convoy --volumename=vol1
DeviceMapper NFS Amazon EBS GlusterFS Ceph

14. 14
Some internal details

15. How does it works (with Rex-ray)?
Mesos Agent docker/volume dvdcli Rex-ray
Container
1) Framework launch a task
2) Provision the container
3) Get the mount point from dvdcli
4) Mount the external volume to mesos container
EBS

16. 16
● Have `dvdcli` installed on each agent
● Volume Lifecycle management (create, delete)
● Use the same Docker volume in both the DockerContainerizer and the
MesosContainerizer simultaneously is strongly discouraged
○ MesosContainerizer has its own reference counting to decide when to unmount a Docker
volume.
○ It would be problematic if a Docker volume is unmounted by MesosContainerizer but the
DockerContainerizer is still using it.
Limitations

17. 17
● Get rid of `dvdcli`
Future Work

18. 18
● Part of DC/OS
● Create Volumes with your normal Marathon App Definition
● Track via Marathon UI
● Reuse the same volume after app restart
External Volumes in Marathon and DC/OS

19. 19
Demo Time
DEMO Time!!

20. 20
Thank you!
Questions?