Kubernetes is designed to be an extensible system. But what is the vision for Kubernetes Extensibility? Do you know the difference between webhooks and cloud providers, or between CRI, CSI, and CNI? In this talk we will explore what extension points exist, how they have evolved, and how to use them to make the system do new and interesting things. We’ll give our vision for how they will probably evolve in the future, and talk about the sorts of things we expect the broader Kubernetes ecosystem to build with them.
4. What is Kubernetes?
...an abstraction layer over
infrastructure
...a framework for declarative
APIs and distributed control
Infrastructure
Extensibility
API
Extensibility
6. A major focus of the last 2 years of development
From infrastructure to APIs, we have over a dozen
extension points
We have WAY more material than time!
https://goo.gl/2qz8jW
Kubernetes & Extensibility
8. Networks are like snowflakes
There is no “one size fits all” for almost
anything networking related
We needed a way for users to customize how
Kubernetes consumes networking infra
Network Plugins
9. Old: built-in “plugins” (aka “send Tim a PR”)
New: CNI - Container Network Interface
● Started by CoreOS, now CNCF with community
● “exec” interface with stdin/stdout/env API
Widely used, also by other projects (e.g. Mesos)
Underpins the default impl in Kubernetes
Network Plugins (present)
10. Proposal open for a gRPC based API which
covers more than just interfaces and IPAM
Tighter coupling with Service API seems valuable
Proposals open for multi-IP and multi-network
Network Plugins (future)
11. Many storage technologies - physical and
virtual, block and file
● Cloud block devices, FC, iSCSI, NFS, Ceph, Gluster, ...
Many vendors want their products to support
Kubernetes
Storage Plugins
12. Old: built-in “plugins” (aka “send Tim a PR”)
Old: Volume “flex” plugins via “exec”
New: CSI - Container Storage Interface
● Collaboration: Google, Mesosphere, Docker, Cloud Foundry
● gRPC spec, with Kubernetes-specific adaptors
● In development now, alpha in Kubernetes 1.10
Plan to transition most in-tree plugins to CSI
Storage Plugins (present)
13. GPUs and other “accelerator” hardware is
becoming very common
Part of the larger resource model in Kubernetes
gRPC based plugins
Beta in Kubernetes 1.10
Device Plugins
14. Docker was baked-in, but people wanted to try
new and interesting ideas
● rkt, Containerd, CRI-O
● Kata containers, Hyper.sh, gVisor
Making it a plugin made the code better: win-win!
CRI - gRPC based plugins
Container Runtimes
17. Controllers
THE fundamental design pattern in Kubernetes
Examples: scheduler, kubelet, deployments,
kube-proxy, cloud providers, load balancers,
volume provisioners, auto-scalers, ...
Allows automation & extension of almost any
existing API
19. Kubernetes is designed to leverage clouds
Built-in cloud-provider API (i.e. send me a PR) is
hooked into many core control loops
Now 8 implementations (and huge LOC count),
so moving out-of-tree
Cloud Providers
20. The API is a VIP (more or less) and virtual LB
We ship a default implementation (kube-proxy),
but that can be replaced
Controller: watch the API server for Services
and Endpoints, program $NETWORK
Services
21. But Wait, There’s More!
● Secret management (KMS)
● HTTP load-balancing (Ingress)
● NetworkPolicy
● DNS
● Scheduler extenders & whole schedulers
● ...and that’s JUST the infrastructure (i.e. boring) parts
23. ● Add new types of resources to your cluster
● Add custom policy hooks
○ to custom and built-in APIs
● "APIs that add and modify APIs"
API Extensibility
24. ● In Mac Edge, Windows Edge, and EE 2.0
● Supports API Extensions.
● Certified Kubernetes
● Docker Stacks uses API Extensions
Kubernetes for Docker
27. Exploring Stacks
https://goo.gl/JT7v8Z
$ docker stack deploy --compose-file docker-compose.yml stackdemo
Waiting for the stack to be stable and running...
- Service redis has one container running
Stack stackdemo is stable and running
36. Exploring Stacks API
https://goo.gl/JT7v8Z
# last time...
$ docker stack deploy --compose-file docker-compose.yml stackdemo
Waiting for the stack to be stable and running...
- Service web has one container running
- Service redis has one container running
Stack stackdemo is stable and running
37. Exploring Stacks API
https://goo.gl/JT7v8Z
№ last time...
$ docker stack deploy --compose-file docker-compose.yml stackdemo
Waiting for the stack to be stable and running...
- Service web has one container running
- Service redis has one container running
Stack stackdemo is stable and running
$ kubectl get stacks
NAME AGE
stackdemo 39s
41. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl proxy -v 5
Starting to serve on 127.0.0.1:8001
I0613 10:13:27.322416 82905
proxy_server.go:138] Filter accepting
GET
/apis/compose.docker.com/v1beta2/name
spaces/default/stacks localhost
$ kubectl get stacks -s localhost:8001
NAME AGE
stackdemo 1m
43. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl get apiservices.apiregistration.k8s.io
NAME AGE
v1. 29d
v1.apps 29d
...
v1beta2.compose.docker.com 29d
v2beta1.autoscaling 29d
46. Exploring Stacks API
https://goo.gl/JT7v8Z
$ kubectl get services -n docker
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d
$ kubectl get deployments -n docker
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
compose 1 1 1 1 29d
compose-api 1 1 1 1 29d
49. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API
50. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API
51. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API
Compose
52. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API
Compose
redis
docker
CLI
53. Kubernetes APIs
Service API
API Registration
API
Compose.
docker.com
Kubernetes Cluster
Deployment API
Compose-API
compose.docker.com
API
Compose
redis
redis
redis
redis
54. ● Users
○ Already have a client installed
○ Already know how to find, trust it (TLS) and auth to the API
● Controllers
○ Can efficiently watch your resources
● Admins
○ Can separate your resources by Namespace
○ Can authorize and audit log access to your resources
Why Use an API Extension?
57. Extension API
Server (EAS)
API Aggregation &
Extension API Servers (EAS)
Extension API
Server (EAS)Extension API
Server (EAS)
API
resource
Controller
58. Extension API
Server (EAS)
API Aggregation &
Extension API Servers (EAS)
Extension API
Server (EAS)Extension API
Server (EAS)
API
resource
Controller
61. EAS
Forked LoC: 0
Storage: provided
Components: 1
Popularity: 100s
Multiversioning: not yet
Customizability: good
CRD
Forked LoC: 5000*
Storage: you manage
Components: 3
Popularity: 10s
Multiversioning: yes
Customizability: better
* http://github.com/sample-apiserver
62. Extension Ecosystem
Devices 5 public plugins
Storage 10 public plugins
Networking >20 public plugins
Custom APIs >400 Github Projects with
custom APIs
63. Extension Ecosystem
● 4 Serverless frameworks
● 6 PaaSes
● 10 CI/CD systems
● 14 different database controllers
● 4 popular ML toolkits
64. Adding Types to the API
● Extension API Servers
● Custom Resource Definitions
Adding Policy to the API
● ValidatingAdmissionWebhooks
● MutatingAdmissionWebhooks
API Extensions
65. Admission:
After authn/z but before storing the change.
Affects mutations, not reads.
Webhooks:
The API Server calls your URL, synchronously
Run in cluster via service or outside, e.g.
serverless.
Admission Webhooks
66. Old thinking:
Better to make narrow specific interfaces, like
ImagePolicyWebhook, for specific use cases.
Can make easier to use. Overly general
extensions may limit future optimization.
Admission Webhooks
67. New thinking:
Many custom resoures. Cluster owners need to
write policy for core resources and for custom
resources written by 3rd parties. Need to
compose policies written by different parties.
Admission Webhooks
68. Composability.
Make all the changes before doing all the checks.
MutatingAdmissionWebhooks
- then-
ValidatingAdmissionWebhooks
Admission Webhooks
70. Istio:
inject sidecar into all the pods
Service Catalog:
inject credentials into
Mutating Admission Webhooks
71. - Mutate the pod template of a deployment
- Install a flaky webhook matching all resources.
Bad Ideas
72. •Kubernetes for Docker:
• Super easy way to try Kubernetes
•API Extensions:
• Use them. Author them. On Docker. For Kubernetes.
•Try it:
• https://goo.gl/JT7v8Z
Conclusion