1. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz lazy pulling
2021/11/5
CloudNative Days Tokyo 2021
2. Copyright(c)2021 NTT Corp. All Rights Reserved
GitHub:@ktock / Twitter:@TokunagaKohei
containerd BuildKit
Container Runtime Meetup
3. Copyright(c)2021 NTT Corp. All Rights Reserved
TL;DR
l Pull 3
l eStargz lazy pulling pull
• https://github.com/containerd/stargz-snapshotter
l eStargz Lazy pulling
• Kubernetes k3s KinD containerd nerdctl Podman CRI-O BuildKit Kaniko
go-containerregistry ko buildpacks.io
l eStargz OCI Image Specification
4. Copyright(c)2021 NTT Corp. All Rights Reserved
pull
pulling packages accounts for 76% of container start time,
but only 6.4% of that data is read [Harter et al. 2016]
[Harter et al. 2016] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. "Slacker: Fast Distribution with
Lazy Docker Containers". 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016, Santa Clara, CA, USA
l pull
l cold start pull
l
• e.g. 4
6. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz lazy pulling
Standard Container Registry
eStargz-aware runtimes
BuildKit
Lazy pull
Lazy pull
legacy
ctr-remote
nerdctl
BuildKit
Kaniko
Image builders/convertes
eStargz
creates
eStargz
still runnable on
eStargz-agnostic tools (e.g. Docker)
etc…
etc…
legacy
l OCI lazy pulling
• Pull
• 6
• 6
l Lazy pulling OCI
l BuildKit Kaniko nerdctl eStargz
https://github.com/containerd/stargz-snapshotter
7. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz
eStargz
bin/ls
usr/bin/apt
entrypoint.sh
bin/bash
Range Request
TOC(Table Of Contents) footer
/
gzip
HTTP Range Request
/
stargz
bin/ls
usr/bin/apt
entrypoint.sh
bin/bash
For more details: https://github.com/containerd/stargz-snapshotter/blob/master/docs/stargz-estargz.md
l
l OCI
l CRFS stargz
l 7
8. Copyright(c)2021 NTT Corp. All Rights Reserved
README https://github.com/containerd/stargz-snapshotter
Lower is better
10. Copyright(c)2021 NTT Corp. All Rights Reserved
stargz-snapshotter 1 0
https://github.com/containerd/stargz-snapshotter
Kubernetes
Containerd
Stargz
Snapshotter
external process
gRPC API
kubelet
Lazy pull
Container Registry
eStargz
CRI-O
Stargz Store
external process
exposed on dir
kubelet
l CRI 0 lazy pulling
• Containerd Stargz Snapshotter
• CRI-O Stargz Store
11. Copyright(c)2021 NTT Corp. All Rights Reserved
K3s
l 1 Kubernetes
l containerd CRI
l v1.22.2 Stargz Snapshotter lazy pulling
https://rancher.com/docs/k3s/latest/en/advanced/#enabling-lazy-pulling-of-estargz-experimental
k3s server --snapshotter=stargz
K3s server/agent
Stargz
Snapshotter (library)
k8s
Lazy pull
Container Registry
eStargz
k3d cluster create --image=rancher/k3s:v1.22.2-k3s2 --k3s-arg='--snapshotter=stargz@server:*;agent:*'
l K3d (k3s in Docker) lazy pulling
builtin
https://github.com/k3s-io/k3s
12. Copyright(c)2021 NTT Corp. All Rights Reserved
KinD
l Kubernetes in Docker Kubernetes
l 1CRI containerd
l Stargz Snapshotter lazy pulling
• ghcr.io/stargz-containers/estargz-kind-node:0.9.0
• KinD 21 :https://github.com/kubernetes-
sigs/kind/pull/2076
$ kind create cluster --name estargz-demo --image ghcr.io/stargz-containers/estargz-kind-node:0.9.0
Stargz
Snapshotter
kubelet
Lazy pull
Container Registry
eStargz
https://github.com/kubernetes-sigs/kind
13. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd nerdctl
proc
Stargz
Snapshotter
Fetches files/chunks on demand
Mounts rootfs as FUSE
Node
Lazy pull
Container Registry
eStargz
container
l Containerd Stargz Snapshotter3 lazy pulling
• Containerd remote snapshotter 3
l nerdctl Docker containerd CLI Stargz Snapshotter lazy pulling 1
nerdctl --snapshotter=stargz run ghcr.io/stargz-containers/python:3.9-esgz
nerdctl --snapshotter=stargz compose -f docker-compose.stargz.yaml up
https://github.com/containerd/containerd
https://github.com/containerd/nerdctl
14. Copyright(c)2021 NTT Corp. All Rights Reserved
Podman CRI-O
proc
container
Stargz Store
Provides layers
(mounted as FUSE)
Podman, CRI-O
Uses layers for rootfs
Node
Lazy pull
Container Registry
eStargz
l Stargz Store lazy pulling
l systemd service
yum install -y stargz-snaphsotter
systemctl enable --now stargz-store
podman run ghcr.io/stargz-containers/python:3.9-esgz
l Additional Layer Store Podman >= v3.3.0, CRI-O >= v1.22.0
• Stargz Store Podman/CRI-O FUSE 4 1
https://github.com/cri-o/cri-o
https://github.com/containers/podman
Additional Layer Store
15. Copyright(c)2021 NTT Corp. All Rights Reserved
BuildKit
FROM ghcr.io/stargz-containers/golang:1.15.3-esgz as dev
COPY ./hello.go /hello.go
RUN go build -o hello /hello.go
COPY and RUN without waiting for
the pull completion
• /usr/local/go/bin/go
• /usr/local/go/src/fmt/…
etc...
Fetch files/chunks on demand
Build on node
Lazy pull
Container Registry
eStargz
golang:1.15.3-esgz
l 5 lazy pulling1 v0.8.0
l 5 1pull 5
• 5
$ docker buildx create --use --name lazy-builder ¥
--driver docker-container --driver-opt image=moby/buildkit:master ¥
--buildkitd-flags '--oci-worker-snapshotter=stargz'
$ docker buildx inspect --bootstrap lazy-builder
$ docker buildx build .
https://github.com/moby/buildkit
20. Copyright(c)2021 NTT Corp. All Rights Reserved
https://github.com/opencontainers/image-spec/issues/815
l OCI Image Specification
• gzip 2 application/vnd.oci.image.layer.v1.tar+gz
• 0 Annotation org.opencontainers.image.toc.digest
21. Copyright(c)2021 NTT Corp. All Rights Reserved
Join our discussion!
l Subproject (non-core) of CNCF graduated containerd
l github https://github.com/containerd/stargz-snapshotter
l slack https://slack.containerd.io/ (#containerd-dev channel at CNCF slack)
l Pre-converted images are available on ghcr.io/stargz-containers
l Pull 2
l eStargz 2 lazy pulling2 pull 1
• https://github.com/containerd/stargz-snapshotter
l 1 eStargz Lazy pulling
• Kubernetes k3s KinD containerd nerdctl Podman CRI-O BuildKit Kaniko
go-containerregistry ko buildpacks.io
l eStargz OCI Image Specification 1