This document discusses modernizing RancherOS, a micro Linux distribution. It describes replacing System Docker with runC and Containerd to reduce size and improve performance. Through iterative changes like removing unused files, generating container specs, and customizing services, the initrd size was reduced from 245MB to 190MB and boot time from 30 seconds to 12 seconds. The final version can boot and serve HTTP requests within 7 seconds while maintaining compatibility with RancherOS.

1. RancherOS & Linux Kit
Sven Dowideit
Principal Software Engineer RancherOS
19 October 2017

2. Is a micro-Linux distro
• Linux Kernel,
• A custom init written in go,
• Uses Cloud-config to customise instances
• Has a System-Docker and a User-Docker
RancherOS

3. #cloud-init
rancher:
services_include:
http-proxy: true
registry-mirror: true
network:
interfaces:
eth1:
addresses:
- 10.11.11.1/24
...
Cloud-config

4. #cloud-init
rancher:
services:
rancher-agent-starter:
image: rancher/agent:v1.2.5
command:
http://10.10.10.168:8080/v1/scripts/22A906891A04AA99E831:1483142400000:U7MMVCDKpoYa3ngeh
fX6BPBt92k
privileged: true
environment:
- CATTLE_HOST_LABELS='magic=node'
labels:
io.rancher.os.scope: user
io.rancher.os.after: docker
restart: false
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/rancher:/var/lib/rancher
...
Cloud-config continued

5. One cut down System-Docker (v1.10-ish)
And the pluggable User-Docker
• v1.12.16 and up
• Can be selected using the cloud-config or on the
commandline.
Tale of two Docker daemons

6. First up, System-Docker - it’s from Q1 2016.
And worse, we start it 3 times to bootstrap the OS.
Each time we start a Docker daemon, we
docker load < stage/Images.tar
How RancherOS starts

7. Use the moby tool to compose the final runnable image or
ISO
RancherOS has been built by composing Container Images
into an Initrd since 2014
So there’s some legacy build tooling - which LinuxKit also
provides.
U
So… what can we modernise

8. Our 90M OS has turned into a 245M initrd.
[ 3.715450] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[ 3.715450]
[ 3.717145] CPU: 0 PID: 1 Comm: init Not tainted 4.9.26-rancher #1
[ 3.718061] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-1ubuntu1 04/01/2014
[ 3.719478] ffff9fd6c031fe20 ffffffffb632793c ffff9060ed820000 ffffffffb6a8d098
[ 3.720933] ffff9fd6c031fea0 ffffffffb614624d ffff9fd600000010 ffff9fd6c031feb0
[ 3.722387] ffff9fd6c031fe48 ffffffffb6c5eb28 0000000000000100 ffff9060ed830010
[ 3.723840] Call Trace:
[ 3.724382] [<ffffffffb632793c>] dump_stack+0x61/0x7d
[ 3.725189] [<ffffffffb614624d>] panic+0xd8/0x221
[ 3.725943] [<ffffffffb606c67a>] do_exit+0x4d4/0x92c
[ 3.726717] [<ffffffffb608b8f2>] ? wake_up_state+0x10/0x12
[ 3.727539] [<ffffffffb6074485>] ? signal_wake_up_state+0x2a/0x3b
[ 3.728418] [<ffffffffb606cb49>] do_group_exit+0x41/0xa2
Step 1: So we’re huge, and….

9. Let’s replace System-Docker with runC and Containerd,
copying the init code from LinuxKit
Next up, rip out the Eye teeth

10. 519M inittrd
15-20 seconds just to load initrd
boot2docker in 30 seconds
Step 2: even huger-er

11. LinuxKit’s service cfg doesn’t share images, it lays one
down per entry
Also means we need to add overlayfs
Share images between services

12. 383M initrd
14s to starting initrd
boot2docker in 25 seconds
Step 3: baby steps

13. There’s a bunch of files that were needed by System
Docker that we can remove.
Remove cruft

14. 190M initrd
5s to get to starting init
boot2docker in 12seconds
Step 4: ok, this is fine.

15. use the os-config.tmpl to generate the runtime spec
no actual change in size or speed
Generate containerd spec

16. client, err := containerd.New(config.DefaultContainerdSocket)
ctx := namespaces.WithNamespace(context.Background(), "default")
ctr, err := client.NewContainer(ctx, serviceName,
containerd.WithNewSpec(
withService(cfg, serviceSet, service),
removeRunTmpfsMount(),
withDevicesFromSpec(spec),
withOverlay(rootfs, rwDir, workDir),
dumpSpec(serviceName+".new"),
),
)
task, err := ctr.NewTask(ctx, io, WithNoPivotRoot())
err := task.Start(ctx)
Code

17. rancher:
services:
nginx:
image: nginx
command: nginx -g 'daemon off;'
labels:
io.rancher.os.scope: system
io.rancher.os.after: docker
restart: always
ports:
- "80:80"
volumes_from:
volumes:
- /usr/bin/ros:/usr/bin/ros
And finally, customise

18. about 11 seconds to serve the first HTTP request
adding the default nginx image adds ~30M to the initrd
$ docker images nginx
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 1e5ab59102ce 7 days ago 108MB
Step N: wash cycle

19. dhcp is taking 3s - hard-code the IP
start nginx after network
use the os-console image for all os-base services
smaller, simpler LinuxKit kernel image
Lets break all the things

20. boot2your-service in 7 seconds
This is still RancherOS
it still uses cloud-init to load host specific settings.
(Though it can go really badly due to conference wifi :) )
Step N+1: boot2your-service

21. Happy hacking!
@SvenDowideit
Sven@home.org.au