5. Why would we want to migrate containers?Why would we want to migrate containers?
• It's awesome!
• Load balancing in a cluster
• Kernel upgrade
– Can be done without migration
• Hardware upgrade
5
6. Why wouldn't we want to live migrate containers?Why wouldn't we want to live migrate containers?
6
7. How to avoid live migrating containersHow to avoid live migrating containers
• Incoming traffic load balancing
• Microservices
• Crash-driven upgrades
• Scheduled downtimes
7
8. How to make live migration really live?How to make live migration really live?
• Need to get rid of migrating memory while the container is frozen
• Two ways:
– Pre-copy the memory
– Post-copy the memory
8
9. Live migration in more detailsLive migration in more details
• Pre-copy: collect and transfer the memory (might be iterative)
• Freeze the container
• Save its state
• Copy the state
• Restore
• Unfreeze
• Post-copy: swap in the memory over the network
9
11. What do we need to migrateWhat do we need to migrate
• Virtual Machine
– Environment (i.e. virtual hardware)
– CPU state
– Memory
• Container
– Environment (cgroups, namespaces)
– Processes and stuff
– Memory
11
12. Collect and copy the memoryCollect and copy the memory
• Virtual Machine
– All memory is at hand
• Container
– Memory is spread through the processes
– Different types of memory (shared/private, backed by a file or not)
– Need to collect the processes first
●
Only then collect the memory
12
13. FreezingFreezing
• Virtual Machine
– Suspend all CPUs
• Container
– Walk the tree (/proc), catch the processes and freeze those
– Freeze cgroup helps a bit
13
14. Saving the stateSaving the state
• Virtual Machine
– Hardware state, tree, 300K, ~70 objects
• Container
– State of all objects, graph, 160K, ~1000 objects
– Not all objects have decent API to get the state
14
15. Copying the stateCopying the state
• Virtual Machine
– Can read and copy at once, easy to serialize
• Container
– Not easy to serialize as it's a graph not a tree
15
16. Restoring the stateRestoring the state
• VM: recreate the memory, state of CPUs and virtual hardware
• Containers
– In-kernel: create a myriad of small objects
– In CRIU: same, but there might not be a convenient API
●
Over 1000 syscalls
●
Need to sort it all out
16
17. FreezeFreeze
• VM: resume the virtual CPUs
• Container
– Either SIGCONT through the tree
– Or “unfreeze” the cgroup
– Problem: need to wake processes in the proper order
17
18. Post-memory migration: network swap devicePost-memory migration: network swap device
• Not yet ready for neither VMs nor CTs
• userfaultfd by Andrea Arcangeli of Red Hat
– a file descriptor to inform about page fault and get a memory back
– merged into 4.2 kernel
– work in progress to use it for KVM/QEMU
• Container
– Userfault FD is not sufficient for CRIU case
18
20. CRIU uses beyond the live migrationCRIU uses beyond the live migration
• HPC jobs: periodic checkpoints
• Slow boot services speed up
• That magical SAVE button e.g. in games
• Software testing speed up
• Reverse debugging
20