Extending kubernetes with CustomResourceDefinitions

Extending Kubernetes with
CustomResourceDefinitions
Stefan Schimanski
sttts@redhat.com / sttts @ GitHub / @the_sttts
CONTAINERdays 2018

Restful
http API
/
/version
/api
/api/v1/pods
/api/v1/pods/<name>
/api/v1/pods/<name>/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alpha1/jobs
/apis/batch/v2alpha1/cronjobs
/apis/batch/v1
/apis/batch/v1/jobs
kube-apiserver
kubeletproxy
Node:
$ kubectl create -f foo.yaml
User:
scheduler
controller
manager
apiserver
ingress
controller
Master:
cloud native apps
Pods:

Restful
http API
/
/version
/api
/api/v1/pods
/api/v1/pods/<name>
/api/v1/pods/<name>/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alpha1/cronjobs
/apis/batch/v1
/apis/batch/v1/jobs
/apis/<our-group>/v1/<your-resource>
kube-apiserver
kubeletproxy
Node:
$ kubectl create -f foo.yaml
User:
scheduler
controller
manager
apiserver
ingress
controller
Master:
cloud native
apps
Pods:
your
controller

$ kube-apiserver
--secure-port 0
--etcd-servers http://127.0.0.1:2379
--service-cluster-ip-range 10.0.0.0/16
--storage-backend etcd2
--storage-media-type application/json
$ etcd
$ kubectl config set-cluster local --server=http://127.0.0.1:8080
$ kubectl config set-context local --cluster=local
$ kubectl config use-context local
$ kubectl get namespaces --v=7
$ kubectl get namespace default -o json
$ kubectl annotate namespace default meetup=hello
$ curl http://127.0.0.1:8080/api/v1/namespaces/default
$ etcdctl ls --recursive
$ etcdctl get /registry/namespaces/default
$ etcdctl -o extended get /registry/namespaces/default

$ http GET http://127.0.0.1:8080/api/v1/namespaces/default
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"annotations": {
"meetup": “hallo"
},
"creationTimestamp": "2017-03-10T07:51:39Z",
"name": "default",
"resourceVersion": “73",
},
"spec": { "finalizers": ["kubernetes“] },
"status": { "phase": "Active“ }
}
$ http GET http://127.0.0.1:8080/api/v1/namespaces/default |
jq ".metadata.annotations["meetup"] = "$(date)"" |
http PUT http://127.0.0.1:8080/api/v1/namespaces/default
HTTP/1.1 200 OK
Content-Length: 341
Content-Type: application/json
Date: Fri, 10 Mar 2017 08:28:01 GMT

group version resource
HTTP paths:
{
“apiVersion“: “v2alpha1“,
“kind“: “Job“,
“metadata“: {
“name“: “backup“
},
“spec“: { ... }
}
http POST
as JSON
or Proto
/namespaces/<name>/
Possible (logical) verbs:
• GET
• CREATE (= POST)
• UPDATE (= PUT)
• LIST (GET on root path)
• DELETE
• PATCH
• WATCH (?watch=true)

Custom Resources (CRs)
&
Custom Resource Definitions (CRDs)
apiextensions/v1beta1

Custom Resource
hamburg-sensor.yaml
apiVersion: containerdays.io/v1
kind: Sensor
metadata:
name: hamburg
namespace: germany
spec:
unit: Celcius
status:
lastUpdated: 2010-03-26T15:13:42.05Z
temperature: 28
We want to store these

apiextensions-apiserver inside kube-apiserver
https://github.com/kubernetes/apiextensions-apiserver
In every Kubernetes 1.7+ cluster.
kube-apiserver
kube-aggregator kube resources apiextensions-
apiserver
404
etcd
"delegation"
"aggregation"

Custom Resource Definition (CRD)
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: sensors.containerdays.io
spec:
group: containerdays.io
version: v1
names:
kind: Sensor
plural: sensors
scope: Namespaced mustmatch
Defines how CRs are stored

Custom Resource Definition (CRD)
metadata:
spec:
version: v1
names:
kind: Sensor
plural: sensors
scope: Namespaced
$ kubectl create –f sensors-crd.yaml
a moment
...
status:
acceptedNames:
kind: Sensor
listKind: SensorList
plural: sensors
singular: sensor
conditions:
- type: NamesAccepted
message: no conflicts found
reason: NoConflicts
status: "True“
- type: Established
message: the initial names have
been accepted
reason: InitialNamesAccepted
status: "True“
mustmatch

$ kubectl get sensors –v=7
• I0429 21:17:53.042783 66743 round_trippers.go:383] GET https://localhost:6443/apis
• I0429 21:17:53.135811 66743 round_trippers.go:383] GET
https://localhost:6443/apis/containerdays.io/v1
• I0429 21:17:53.138353 66743 round_trippers.go:383] GET
https://localhost:6443/apis/containerdays.io/v1/namespaces/default/sensors
No resources found.
sensors → kind Sensor
resource sensors
discovery
LIST
API group containerdays.io/v1
We call this "REST mapping"

$ http localhost:8080/apis/
{
"groups": [{
"name": "containerdays.io",
"preferredVersion": {"groupVersion": "containerdays.io/v1", "version": "v1“},
"versions": [{"groupVersion": "containerdays.io/v1", "version": "v1"}]
}, ...]
}
$ http localhost:8080/apis/containerdays.io/v1
{
"apiVersion": "v1",
"groupVersion": "containerdays.io/v1",
"kind": "APIResourceList",
"resources": [{
"kind": "Sensor",
"name": "sensors",
"namespaced": true,
"verbs": ["create", "delete", "deletecollection",
"get", "list", "patch", "update", "watch“
]
}, ...]
}
resource name ⇒ /apis/containerdays.io/v1/sensors
/namespaces/<name>/

$ kubectl create –f sensors-crd.yaml $ kubectl create –f hamburg-sensor.yaml
kind: Sensor
metadata:
name: hamburg
namespace: germany
spec:
unit: Celcius
status:
lastUpdate: 2010-03-26T15:13:42.05Z
temperature: 28
metadata:
spec:
version: v1
names:
kind: Sensor
plural: sensors
scope: Namespaced
1 2
Create the CRD Create CustomResources

$ kubectl create –f sensors-crd.yaml $ kubectl create –f hamburg-sensor.yaml
kind: Sensor
metadata:
name: hamburg
namespace: germany
spec:
unit: Celcius
status:
lastUpdate: 2010-03-26T15:13:42.05Z
temperature: 28
metadata:
spec:
version: v1
names:
kind: Sensor
plural: sensors
scope: Namespaced
1 2
Create the CRD Create CustomResources
Follow spec+status pattern.

$ kubectl get sensors –w --no-headers
brussels <none> {"apiVersion":"containerdays.io/v1","kind":"Sensor",...
hamburg <none> {"apiVersion":"containerdays.io/v1","kind":"Sensor",...
$ curl -f 'http://127.0.0.1:8080/apis/containerdays.io/v1/namespaces/default/sensors?watch=true&resourceVersion=434‘
{"type":"DELETED","object":{"apiVersion":"containerdays.io/v1","kind":"Sensor","metadata":{"name":"brussels","namespace
":"hamburg","selfLink":"/apis/containerdays.io/v1/namespaces/belgium/sensors/brussels","uid":"8f5312c0-29c8-11e7-88f9-
4c3275978b79","resourceVersion":"435","creationTimestamp":"2017-04-25T15:05:03Z"},"spec":{...}}}
{"type":"ADDED","object":{"apiVersion":"containerdays.io/v1","kind":"Sensor","metadata":{"name":"hamburg","namespace
":"germany","selfLink":"/apis/containerdays.io/v1/namespaces/germany/sensors/berlin","uid":"b4318cb5-29c8-11e7-88f9-
4c3275978b79","resourceVersion":"436","creationTimestamp":"2017-04-25T15:06:05Z"},"spec":{...}}}
Watch, i.e. event stream

Validation
• The standard: OpenAPI v3 schema
• based on JSON Schema
metadata:
spec:
version: v1
names: ...
validation:
openAPIV3Schema: <see next slide>

spec:
unit: Celclius
status:
lastUpdate: 2010-03-26T...
temperatur: true
properties:
spec:
properties:
unit:
anyOf: [{"pattern": "^Celclius$"}, …]
default: "Celcius"}
required: ["unit"]
status:
properties:
lastUpdate: {"type": "string", "pattern": "^[0-9]+-...$"},
temperature: {"type": "number", "minimum": -273.15},
required: []
OpenAPIv3Schema
a quantor (anyOf, oneOf, allOf exist)
note: enum is forbidden (why?)
regular expression
probably in 1.12+
Custom Resource
Helpful tools:
kubernetes/kube-openapi#37
tamalsaha/kube-openapi-generator
Some other tool from prometheus-operator?
Rancher has another one, speak to @lemonjet

Controllers
where is the business logic?

$ while true; do
http GET http://127.0.0.1:8080/api/v1/namespaces/default |
http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break
done
⟲
1. read object (preferably event driven with watches)
2. change object, update the world
3. update object on apiserver
4. repeat

$ while true; do
{jq ".metadata.annotations["meetup"] = "$RANDOM""; sleep 1;} |
done
HTTP/1.1 409 Conflict
Content-Length: 310
Content-Type: application/json
Date: Fri, 10 Mar 2017 08:27:58 GMT
{
"apiVersion": "v1",
"code": 409,
"details": {
"kind": "namespaces",
"name": "default“
},
"kind": "Status",
"message": "Operation cannot be fulfilled on namespaces "default": the object has been
modified; please apply your changes to the latest version and try again",
"metadata": {},
"reason": "Conflict",
"status": "Failure“
}
„optimistic concurrency“

$ while true; do
done
⟲
1. read object (preferably event driven with watches)
2. change object, update the world
3. update object on apiserver
4. repeat
expect version conflicts,
remember: optimistic concurrency

$ kubectl get namespaces --watch --no-headers |
while read NS STATUS TIME ; do
# do whatever you like here, e.g. change the namespace
echo "$NS changed“
done
⟲
W
atch=nohot looping
$ curl -f 'http://127.0.0.1:8080/api/v1/namespaces?watch=true&resourceVersion=4711‘
{"type":"ADDED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
{"type":“MODIFIED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
{"type":“DELETED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...

"CRDs are limited"
• no version conversion (only one version possible per CRD)
• in 1.11 multiple versions, but no conversion
• in 1.12+ conversions come, but slowly
• no defaulting (PR exists)
• no validation (beta in 1.9, Google Summer of Code project)
• no subresources (scale, status) (beta in 1.11)
• no admission (since 1.7: admission webhooks + initializers)
• alpha CRDs are beta
• no custom printing in kubectl (in 1.11: custom printer columns)

Outlook – Custom Resources
• Kubernetes 1.11
• Server Side Printing Columns – “kubectl get” customization – #60991
• β : Multiple versions without conversion – design proposal
• β: Subresources – ⍺ since 1.10 – #62786
• OpenAPI additionalProperties allowed now
(mutually exclusive with properties)
• Kubernetes 1.12+
• ⍺: Multiple versions with declarative field renames
• ⍺: Graceful Deletion – being discussed – #63162
• ⍺: Pruning – in validation spec unspecified fields are removed – blocker for GA
• ⍺: Defaulting – defaults from OpenAPI validation schema are applied
• Strict create mode? Discuss: #5889 – my favorite CRD UX issue
Related: CRD OpenAPI validation spec not served by kube-apiserver

Zoom into apiextensions-apiserver
kube-apiserver
kube-
aggregator
kube
resources
apiextensions-apiserver
404
etcd
"delegation"
"aggregation"
authn/z
CR handlers
CR handlers
CR handlers
⟲Naming Controller
⟲CRD Finalizer
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
encode
GET
CREATE
LIST
UPDATE
DELETE
WATCH
mutating
webhooks
validating
webhooks NoOps
json.Unmarshal

/apis/extensions/v1alpha1/jobs/backup
/apis/batch/v2alpha1/jobs/backup
/apis/batch/v1/jobs/backup
{
“metadata“: {
},
“spec“: { ... }
}
{
“metadata“: {
},
“spec“: { ... }
}
{
“apiVersion“: “v1“,
“metadata“: {
},
“spec“: { ... }
}
lossless
conversion
* I omitted the namespace in /apis/batch/v1/jobs/namespaces/default/backup

{
“metadata“: {
},
“spec“: { ... }
}
{
“metadata“: {
},
“spec“: { ... }
}
{
“apiVersion“: “v1 “,
“metadata“: {
},
“spec“: { ... }
}
lossless
conversion
etcd
{
“apiVersion“: “v1 “,
“metadata“: {
},
“spec“: { ... }
}
Protobuf
or
JSON
storage
version
encoding

Extending kubernetes with CustomResourceDefinitions

Extending kubernetes with CustomResourceDefinitions

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Extending kubernetes with CustomResourceDefinitions

Ähnlich wie Extending kubernetes with CustomResourceDefinitions (20)

Mehr von Stefan Schimanski

Mehr von Stefan Schimanski (11)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Extending kubernetes with CustomResourceDefinitions