A talk given at FOSDEM 2020 in the containers devroom on the current status of the CNCF containerd project as well as a dive into the ways users are extending and embedding containerd in other platforms and projects.

1. Extending and embedding:
containerd project use cases
A 2020 FOSDEM containerd project update
Phil Estes
Distinguished Engineer & CTO, IBM Cloud Platform
CNCF containerd project maintainer

2. State of containerd

3. What is containerd
● A “Container runtime”
○ Below platforms (Docker, Kubernetes)
○ Above lower level runtimes (runc, Kata, Firecracker, gVisor)
● Resource Manager
○ Container processes
○ Image artifacts
○ Filesystem snapshots
○ Metadata and dependencies
● Tightly scoped
○ 100% maintainer approval required to increase scope
○ Built-in CRI plugin only scope increase

4. State of containerd
● 5th project to graduate within the CNCF - February 2019
● Broad support and contribution from across the ecosystem
○ Over 200 individual contributors; represent > 100 companies
○ 13 maintainers represent 9 different companies
● All major cloud providers using containerd
● Supports Linux and Windows platforms, multiple architectures
● Added sub-projects to governance (Rust-based ttrpc; image encryption)

5. containerd 1.3
● Windows support for shim V2 API
● Device mapper snapshotter (Amazon Firecracker team contribution)
● New plugin interface for processing layers (encryption, compression)
● (CRI) Support for per-pod container shim

6. In progress
● Remote snapshotter for sharing snapshots in a cluster
● cgroups v2
● Windows CRI
● Mount and resource management
● Image encryption

7. Who is using containerd?
● Public Clouds
● Kubernetes Infra
● End Users
● DevOps Tools
● Custom Sandboxes

8. How is containerd used?
● Library
○ Go client API
■ oras, BuildKit, Weaveworks Ignite, IBM Cloud Functions, OpenFaaS
“faasd”, Alibaba PouchContainer
○ Extensibility
■ Amazon ECR resolver, Azure Teleport, remote snapshotters [cvmfs, stargz]
○ Imports/Subprojects (cri-o use of containerd/cgroups)
● Kubernetes Runtime
○ CRI-containerd
■ IBM Kubernetes Service, GKE, Ticketmaster, Alibaba, microk8s, KinD, k3s,
AWS Fargate
● Daemon
○ Docker, BuildKit

9. Embedding/Extending Use Cases

10. Architecture

11. API

12. API - CRI
- CRI gRPC API exposed from containerd
- Kubelet can be configured to use containerd as runtime

13. API - containerd
- gRPC API, used by Go client
- Low level access to components
- Mirrors internal component interfaces
- Snapshots, Content, Containers, Task, Events, etc

14. Core

15. Backend

16. Plugins

17. Plugins - Backend

18. Plugins - Backend
- No re-compilation required
- Proxy plugins for content store and snapshotters
- Runtime shims are separate binaries
implementing shim interface

19. Plugins - Client
1. Override services with service
options
2. Customize push and pull with
remote options
type RemoteOpt
func WithImageHandler(h images.Handler) RemoteOpt
func WithImageHandlerWrapper(w func(images.Handler) images.Handler) RemoteOpt
func WithResolver(resolver remotes.Resolver) RemoteOpt
type ServicesOpt
func WithContainerService(containerService containersapi.ContainersClient) ServicesOpt
func WithContentStore(contentStore content.Store) ServicesOpt
func WithDiffService(diffService diff.DiffClient) ServicesOpt
func WithEventService(eventService EventService) ServicesOpt
func WithImageService(imageService imagesapi.ImagesClient) ServicesOpt
func WithLeasesService(leasesService leases.Manager) ServicesOpt
func WithNamespaceService(namespaceService namespacesapi.NamespacesClient) ServicesOpt
func WithSnapshotters(snapshotters map[string]snapshots.Snapshotter) ServicesOpt
func WithTaskService(taskService tasks.TasksClient) ServicesOpt
2
1

20. Plugins - custom containerd binary
● Add a file with import to
cmd/containerd/ in your fork.
● Create your own main.go of
containerd
package main
import (
"fmt"
"os"
"github.com/containerd/containerd/cmd/containerd/command"
// import built-in plugins from cmd/containerd/builtins.go
_ "github.com/mygithub/customplugin"
)
func main() {
app := command.App()
if err := app.Run(os.Args); err != nil {
fmt.Fprintf(os.Stderr, "containerd: %sn", err)
os.Exit(1)
}
}

21. Example Snapshotter Proxy Plugin
// Snapshot service manages snapshots
service Snapshots {
rpc Prepare(PrepareSnapshotRequest) returns (PrepareSnapshotResponse);
rpc View(ViewSnapshotRequest) returns (ViewSnapshotResponse);
rpc Mounts(MountsRequest) returns (MountsResponse);
rpc Commit(CommitSnapshotRequest) returns (google.protobuf.Empty);
rpc Remove(RemoveSnapshotRequest) returns (google.protobuf.Empty);
rpc Stat(StatSnapshotRequest) returns (StatSnapshotResponse);
rpc Update(UpdateSnapshotRequest) returns (UpdateSnapshotResponse);
rpc List(ListSnapshotsRequest) returns (stream ListSnapshotsResponse);
rpc Usage(UsageRequest) returns (UsageResponse);
}
- implement Snapshotter gRPC API
- backend requests are proxied to plugin

22. External snapshotter
● Configure with proxy_plugins
● Build as an external plugin
[proxy_plugins]
[proxy_plugins.customsnapshot]
type = "snapshot"
address = "/var/run/mysnapshotter.sock"
package main
import(
"net"
"log"
"github.com/containerd/containerd/api/services/snapshots/v1"
"github.com/containerd/containerd/contrib/snapshotservice"
)
func main() {
rpc := grpc.NewServer()
sn := CustomSnapshotter()
service := snapshotservice.FromSnapshotter(sn)
snapshots.RegisterSnapshotsServer(rpc, service)
// Listen and serve
l, err := net.Listen("unix", "/var/run/mysnapshotter.sock")
if err != nil {
log.Fatalf("error: %vn", err)
}
if err := rpc.Serve(l); err != nil {
log.Fatalf("error: %vn", err)
}
}

23. Runtime Plugins

24. Runtime shim v2 API
● Minimal and scoped to the execution lifecycle of a container
● Binary naming convention
○ Type io.containerd.runsc.v1 -> Binary containerd-shim-runsc-v1

25. Runtime Plugins - Task Service
service Task {
rpc State(StateRequest) returns (StateResponse);
rpc Create(CreateTaskRequest) returns (CreateTaskResponse);
rpc Start(StartRequest) returns (StartResponse);
rpc Delete(DeleteRequest) returns (DeleteResponse);
rpc Pids(PidsRequest) returns (PidsResponse);
rpc Pause(PauseRequest) returns (google.protobuf.Empty);
rpc Resume(ResumeRequest) returns (google.protobuf.Empty);
rpc Checkpoint(CheckpointTaskRequest) returns (google.protobuf.Empty);
rpc Kill(KillRequest) returns (google.protobuf.Empty);
rpc Exec(ExecProcessRequest) returns (google.protobuf.Empty);
rpc ResizePty(ResizePtyRequest) returns (google.protobuf.Empty);
rpc CloseIO(CloseIORequest) returns (google.protobuf.Empty);
rpc Update(UpdateTaskRequest) returns (google.protobuf.Empty);
rpc Wait(WaitRequest) returns (WaitResponse);
rpc Stats(StatsRequest) returns (StatsResponse);
rpc Connect(ConnectRequest) returns (ConnectResponse);
rpc Shutdown(ShutdownRequest) returns (google.protobuf.Empty);
}

26. How is containerd used?
● Library
○ Go client API
■ oras, BuildKit, Weaveworks Ignite, IBM Cloud Functions, OpenFaaS
“faasd”, Alibaba PouchContainer
○ Extensibility
■ Amazon ECR resolver, Azure Teleport, remote snapshotters [cvmfs, stargz]
○ Imports/Subprojects (cri-o use of containerd/cgroups)
● Kubernetes Runtime
○ CRI-containerd
■ IBM Kubernetes Service, GKE, Ticketmaster, Alibaba, microk8s, KinD, k3s,
AWS Fargate
● Daemon
○ Docker, BuildKit

27. Thank You!