How to improve the network of Docker? How to integrate with OpenVSwitch? How to apply more fine-grained QoS limitation and monitor the resource usage of containers? How to improve Docker in non-intrusive ways? This slides shared by Li Yulai, the Chief Architect of SpeedyCloud, is to answer questions above. Visit http://www.speedycloud.cn to find out more.

1. Resource Management of Docker
Li Yulai @ SpeedyCloud

2. Docker
• Resources Isolation
– Linux Namespaces
• Limitation of Resources Usage (QoS)
– cgroups
• Resources Monitoring
– cgroups

3. Docker Resources Isolation

4. Docker Resources Isolation
• Linux Namespaces
– CLONE_NEWNS (2.4.19)
– CLONE_NEWUTS (2.6.19)
– CLONE_NEWIPC (2.6.19)
– CLONE_NEWPID (2.6.24)
– CLONE_NEWNET (2.6.24)
– CLONE_NEWUSER (3.8)

5. • /proc/$PID/ns/
– ipc
– mnt
– net
– pid
– user
– uts
Docker Resources Isolation

6. • Namespaces Invoking
– clone
– unshare
– setns
• Namespaces Command Line Tools
– ip netns
– unshare
Docker Resources Isolation

7. • By invoking clone, run new process under a
group of new Namespace
Docker Resources Isolation

8. What can we do about
Namespaces?

9. Docker Network Improvement
• Add NIC to Docker
• Usage ： network.sh docker-test veth0 192.168.1.10/24

10. • Linux Net Namespace: /proc/$PID/ns/net
• ip link add Command creates veth NIC
• ip netns Command adjust netns configuration
by detecting /var/run/netns path
• Use ovs-vsctl (or brctl) to setup the network of
new NIC
Docker Network Improvement

11. Docker Network Improvement

12. Docker Network Improvement

13. QoS of Docker Resources

14. QoS of Docker Resources
• Linux cgroups
• Docker cgroups model
– libcontainer ： /sys/fs/cgroup/<subsystem>/docker/<cont
ainerID>
– Linux Container:
/sys/fs/cgroup/<subsystem>/lxc/<containerID>
• Kernel start parameters
– cgroup_enable=memory swapaccount=1

15. QoS of Docker Resources
• CPU QoS
– cpuset.cpus
– cpu.shares
• Memory QoS
– memory.memsw.limit_in_bytes
– memory.limit_in_bytes
swap = memory.memsw.limit_in_bytes – memory.limit_in_bytes

16. The tangling of Memory, Swap and
OOM

17. The tangling of Memory, Swap and OOM
• Trigger of memory, swap and OOM
– When memory is short, Kernel will exchange
memory pages to swap
– OOM will be triggered if swap and memory are
full.
• How does OOM work
– Find the process that uses most of the memory
( the child process usage will be added to parent
process usage)
– Kill the process that uses most of the memory

18. The tangling of Memory, Swap and OOM
• Imagine that if you are running php-fpm, and the
OOM happens…

19. The tangling of Memory, Swap and OOM
• Save more for swap in your container if
you want to keep alive

20. QoS of Docker Resources
• DISK I/O QoS
– blkio.throttle.read_bps_device
– blkio.throttle.write_bps_device
– blkio.throttle.read_iops_device
– blkio.throttle.write_iops_device
• echo "<major>:<minor> <limit>" > throttle.write_iops_device

21. QoS of Docker Resources
• The major and minor IDs of disk
– cat /proc/partitions
– ls –l /dev/
• Use the major and minor of devices, not those
of partitions

22. QoS of Docker Resources
• Network Bandwidth QoS
– OpenVSwitch
– Linux Bridge
• OpenVSwitch
– ovs-vsctl set interface veth1 ingress_policing_rate=1000
• ebtables + tc
– ebtables -A FORWARD -i veth1 -j mark --mark-set 0x1 --mark-target
ACCEPT
– qdics, class, filter
– tc filter add dev eth0 parent 1:0 protocol ip handle 1 fw flowid 1:1

23. QoS of Docker Resources
• Limitation of Disk Volume
– LVM creates the volume and make it accessible for
Container by using –volume parameter after
mount
– Use btrfs and set up quota
• btrfs qgroup limit –e 100G
/var/lib/docker/btrfs/subvolumes/CONTAINER_ID

24. Docker Resources Monitor

25. Docker Resources Monitor
• CPU ： cpuacct.usage
• Memory ：
– memory.usage_in_bytes
– memory.memsw.usage_in_bytes
• Disk IO ：
– blkio.throttle.io_serviced
– blkio.throttle.io_service_bytes
• Bandwidth ：
– /sys/class/net/<ethname>/statistics/

26. Out of Docker

27. Out of Docker
• Bocker
– https://github.com/p8952/bocker
• Dockerlite
– https://github.com/docker/dockerlite