Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Extend and Build on	
Kubernetes
Dr.	Stefan	Schimanski
sttts@redhat.com
@the1stein
Kubernetes Meetup Frankfurt,	Mar	10	2017
Nune Isabekyan,	https://x-team.com/blog/introduction-kubernetes-architecture/
Disclaimer:
Kubernetes happens to be able to launch pods.	It‘s even quite good at	it.
We will	not	launch pods today.
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alp...
$	kube-apiserver
--secure-port 0	
--etcd-servers	http://127.0.0.1:2379
--service-cluster-ip-range	10.0.0.0/16
--storage-ba...
$	http	GET http://127.0.0.1:8080/api/v1/namespaces/default
{
"apiVersion":	"v1",				
"kind": "Namespace",
"metadata": {
"a...
$	while true;	do
http	GET	http://127.0.0.1:8080/api/v1/namespaces/default |	
jq ".metadata.annotations["meetup"]	=	"$(date...
/apis/batch/v2alpha1/jobs
Apigroup Version			Resource
HTTP	paths:
{
“apiVersion“:	“v2alpha1“,
“kind“:	“Job“,
“metadata“:	{...
{				
"apiVersion":	"v1",				
"kind":	"Status",				
"metadata":	{},				
"code":	409,			
"message":	“...",				
"status":	"F...
/apis/extensions/v1alpha1/jobs
/apis/batch/v2alpha1/jobs
/apis/batch/v1/jobs {
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“m...
/apis/extensions/v1alpha1/jobs
/apis/batch/v2alpha1/jobs
/apis/batch/v1/jobs {
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“m...
/apis/extensions/v1alpha1/jobs/nightly
/apis/batch/v2alpha1/jobs/nightly
/apis/batch/v1/jobs/nightly
{
“apiVersion“:	“v1al...
{
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v2alpha1...
I	want my own kinds
and store them in	the apiserver and use kubectl.
Discovery
how kubectl knows which kinds/resources exist
$	http	127.0.0.1:8080/apis/	
{
"groups":	[{
"name": "batch",
"preferredVersion": {"groupVersion": "batch/v1", "version": "...
Third	Party	Resources
Third	Party	Resources
*	they are marked as beta in	the API,	but	became beta before we even had alpha.
apiVersion:	extensions/v1beta1
kind:	ThirdPartyResource
metadata:		
name:	databases.example.com
description:	"A	specificat...
apiVersion:	extensions/v1beta1
kind:	ThirdPartyResource
metadata:		
name:	databases.example.com
description:	"A	specificat...
apiVersion:	extensions/v1beta1
kind:	ThirdPartyResource
metadata:		
name:	database.example.com
description:	"A	specificati...
$ kubectl get databases –w --no-headers
wordpress <none>				{"apiVersion":"example.com/v1","kind":"Databases",...
wordpres...
https://github.com/kubernetes/features/issues/95
ThirdPartyResources are limited
• no version conversion
• no defaulting
• no validation
• no subresources (scale,	status)
...
Controllers
where is the business logic?
$	while true;	do
http	GET	http://127.0.0.1:8080/api/v1/namespaces/default |	
jq ".metadata.annotations["meetup"]	=	"$(date...
$	kubectl get namespaces -w --no-headers	|	
while read NS	STATUS	TIME	;	do
#	do	whatever you like	here,	e.g.	change the na...
API	Aggregation
Why
• ThirdPartyResources are limited
• no version conversion
• no defaulting
• no validation
• no subresources (scale,	st...
Alpha	in	v1.6:	k8s.io/apiserver
• generic apiserver library in	Go
• today used inside
• kube-apiserver
• federation apiser...
Nune Isabekyan,	https://x-team.com/blog/introduction-kubernetes-architecture/
kube-apiserver service catalog PaaS
kube-aggregator
kube-apiserver service catalog PaaS
kube-aggregator
API API API
API
Pods Jobs ... announcement ... build test project
kube-apiserver kube-apiserver kube-apiserver
federation
apiserver
API API API
API
Pods replicasets Pods replicasets Pods r...
kube-apiserver service catalog PaaS
kube-aggregator
discoverydiscovery
kube-apiserver service catalog PaaS
kube-aggregator
GET
GET	/apis/servicecatalog/subscription/database-prod-wordpress
kube-apiserver service catalog PaaS
kube-aggregator
GET	/apis/servicecatalog/subscription/database-prod-wordpress
GET
RBAC...
kube-apiserver service catalog PaaS
kube-aggregator
GET
Vision:	
$ helm install service-catalog
$ kubectl create service-a...
Status
• will	be part of Kubernetes 1.6	as an	alpha
• https://github.com/kubernetes/sample-apiserver
• potentially kube-ag...
Links
• https://docs.google.com/document/d/1y16jKL2hMjQO0trYBJJSczPA
Wj8vAgNFrdTZeCincmI/
Two Ways to Extend the K8s	API -...
https://github.com/kubernetes/community/blob/master/contributors/devel/client-libraries.md
Backup
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alp...
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alp...
Restful
http	API
mux
k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST
AddSupportedResourcesWebService – /apis/ba...
Restful
http	API
mux
pkg/apis/batch
type Jobs	struct
pkg/apis/batch/v2alp
ha1
type Jobs	struct
api.Sche
me api.Scheme.Conv...
type Scheme struct
• AddKnownTypes(gv, obj Object)
• Default(src Object)
• Copy(src Object) Object
• Convert(in, out inter...
Extend and build on Kubernetes
Extend and build on Kubernetes
Nächste SlideShare
Wird geladen in …5
×

Extend and build on Kubernetes

1.730 Aufrufe

Veröffentlicht am

API basics, kinds, resource, apigroups, etcd storage, watches, discovery, ThirdPartyResources, custom apiservers, API aggregation

Veröffentlicht in: Software

Extend and build on Kubernetes

  1. 1. Extend and Build on Kubernetes Dr. Stefan Schimanski sttts@redhat.com @the1stein Kubernetes Meetup Frankfurt, Mar 10 2017
  2. 2. Nune Isabekyan, https://x-team.com/blog/introduction-kubernetes-architecture/
  3. 3. Disclaimer: Kubernetes happens to be able to launch pods. It‘s even quite good at it. We will not launch pods today.
  4. 4. Restful http API / /version /api /api/v1/pods /api/v1/pods/status /apis /apis/batch /apis/batch/v2alpha1 /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjobs /apis/batch/v1 /apis/batch/v1/jobs kube-apiserver kubeletproxy Node: $ kubectl create -f foo.yaml User: scheduler controller manager apiserver ingress controller Master: cloud native apps Pods:
  5. 5. $ kube-apiserver --secure-port 0 --etcd-servers http://127.0.0.1:2379 --service-cluster-ip-range 10.0.0.0/16 --storage-backend etcd2 --storage-media-type application/json $ etcd $ kubectl config set-cluster local --server=http://127.0.0.1:8080 $ kubectl config set-context local --cluster=local $ kubectl config use-context local $ kubectl get namespaces –v=7 $ kubectl get namespace default -o json $ kubectl annotate namespace default meetup=hello $ curl http://127.0.0.1:8080/api/v1/namespaces/default $ etcdctl get / --recursive $ etcdctl get /registry/namespaces/default $ etcdctl -o extended get /registry/namespaces/default
  6. 6. $ http GET http://127.0.0.1:8080/api/v1/namespaces/default { "apiVersion": "v1", "kind": "Namespace", "metadata": { "annotations": { "meetup": “hallo" }, "creationTimestamp": "2017-03-10T07:51:39Z", "name": "default", "resourceVersion": “73", }, "spec": { "finalizers": ["kubernetes“] }, "status": { "phase": "Active“ } } $ http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations["meetup"] = "$(date)"" | http PUT http://127.0.0.1:8080/api/v1/namespaces/default HTTP/1.1 200 OK Content-Length: 341 Content-Type: application/json Date: Fri, 10 Mar 2017 08:28:01 GMT
  7. 7. $ while true; do http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations["meetup"] = "$(date)"" | http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break done HTTP/1.1 409 Conflict Content-Length: 310 Content-Type: application/json Date: Fri, 10 Mar 2017 08:27:58 GMT { "apiVersion": "v1", "code": 409, "details": { "kind": "namespaces", "name": "default“ }, "kind": "Status", "message": "Operation cannot be fulfilled on namespaces "default": the object has been modified; please apply your changes to the latest version and try again", "metadata": {}, "reason": "Conflict", "status": "Failure“
  8. 8. /apis/batch/v2alpha1/jobs Apigroup Version Resource HTTP paths: { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } }
  9. 9. { "apiVersion": "v1", "kind": "Status", "metadata": {}, "code": 409, "message": “...", "status": "Failure“ } /apis/batch/v2alpha1/jobs { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } Resource vs. Kind http path vs. logical object
  10. 10. /apis/extensions/v1alpha1/jobs /apis/batch/v2alpha1/jobs /apis/batch/v1/jobs { “apiVersion“: “v1alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } }
  11. 11. /apis/extensions/v1alpha1/jobs /apis/batch/v2alpha1/jobs /apis/batch/v1/jobs { “apiVersion“: “v1alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } }
  12. 12. /apis/extensions/v1alpha1/jobs/nightly /apis/batch/v2alpha1/jobs/nightly /apis/batch/v1/jobs/nightly { “apiVersion“: “v1alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v1beta1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } * I omitted the namespace in /apis/batch/v1/jobs/namespaces/default/nightly
  13. 13. { “apiVersion“: “v1alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v1 “, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } etcd { “apiVersion“: “v1 “, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } Protobuf or JSON storage version encoding
  14. 14. I want my own kinds and store them in the apiserver and use kubectl.
  15. 15. Discovery how kubectl knows which kinds/resources exist
  16. 16. $ http 127.0.0.1:8080/apis/ { "groups": [{ "name": "batch", "preferredVersion": {"groupVersion": "batch/v1", "version": "v1“}, "versions": [{"groupVersion": "batch/v1", "version": "v1"}] }, ...] } $ http 127.0.0.1:8080/apis/batch/v1 { "apiVersion": "v1", "groupVersion": "batch/v1", "kind": "APIResourceList", "resources": [{ "kind": "Job", "name": "jobs", "namespaced": true, "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch“ ] }, ...] } resource name ⇒ /apis/batch/v1/jobs
  17. 17. Third Party Resources
  18. 18. Third Party Resources * they are marked as beta in the API, but became beta before we even had alpha.
  19. 19. apiVersion: extensions/v1beta1 kind: ThirdPartyResource metadata: name: databases.example.com description: "A specification of a SQL database.“ versions: - name: v1 $ kubectl create –f databases-tpr.yaml
  20. 20. apiVersion: extensions/v1beta1 kind: ThirdPartyResource metadata: name: databases.example.com description: "A specification of a SQL database.“ versions: - name: v1 $ kubectl create –f databases-tpr.yaml $ kubectl create –f wordpress-databases.yaml apiVersion: example.com/v1 kind: Databases metadata: name: wordpress spec: user: wp password: secret encoding: unicode
  21. 21. apiVersion: extensions/v1beta1 kind: ThirdPartyResource metadata: name: database.example.com description: "A specification of a SQL database.“ versions: - name: v1 $ kubectl create –f databases-tpr.yaml $ kubectl create –f wordpress-database.yaml apiVersion: example.com/v1 kind: Database metadata: name: wordpress spec: user: wp password: secret encoding: unicode
  22. 22. $ kubectl get databases –w --no-headers wordpress <none> {"apiVersion":"example.com/v1","kind":"Databases",... wordpress <none> {"apiVersion":"example.com/v1","kind":"Databases",...
  23. 23. https://github.com/kubernetes/features/issues/95
  24. 24. ThirdPartyResources are limited • no version conversion • no defaulting • no validation • no subresources (scale, status) • no admission • alpha ⇒ API might change • but: demand is high, expect improvements in 1.7+ • Today‘s users of TPRs: https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c
  25. 25. Controllers where is the business logic?
  26. 26. $ while true; do http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations["meetup"] = "$(date)"" | http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break done ⟲
  27. 27. $ kubectl get namespaces -w --no-headers | while read NS STATUS TIME ; do # do whatever you like here, e.g. change the namespace echo "$NS changed“ done ⟲ $ curl -f 'http://127.0.0.1:8080/api/v1/namespaces?watch=true&resourceVersion=4711‘ {"type":"ADDED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ... {"type":“MODIFIED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ... {"type":“DELETED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
  28. 28. API Aggregation
  29. 29. Why • ThirdPartyResources are limited • no version conversion • no defaulting • no validation • no subresources (scale, status) • no admission • Some things need full power of Go • Service catalog • OpenShift PaaS • other powerful APIs ≫ Goal: allow powerful extensions without modifying Kubernetes itself
  30. 30. Alpha in v1.6: k8s.io/apiserver • generic apiserver library in Go • today used inside • kube-apiserver • federation apiserver • service catalog • allows creation of custom apiservers in a couple hundred lines of code • each custom apiserver is its own process, communicating via HTTPS • delegates authentication/authorization to kube-apiserver • uses etcd storage (possibly shared with kube)
  31. 31. Nune Isabekyan, https://x-team.com/blog/introduction-kubernetes-architecture/
  32. 32. kube-apiserver service catalog PaaS kube-aggregator
  33. 33. kube-apiserver service catalog PaaS kube-aggregator API API API API Pods Jobs ... announcement ... build test project
  34. 34. kube-apiserver kube-apiserver kube-apiserver federation apiserver API API API API Pods replicasets Pods replicasets Pods replicasets deployment service Europe US Asia Not this: federation controllers⟲ federated resources: availability zones + regions:
  35. 35. kube-apiserver service catalog PaaS kube-aggregator discoverydiscovery
  36. 36. kube-apiserver service catalog PaaS kube-aggregator GET GET /apis/servicecatalog/subscription/database-prod-wordpress
  37. 37. kube-apiserver service catalog PaaS kube-aggregator GET /apis/servicecatalog/subscription/database-prod-wordpress GET RBAC Namespace
  38. 38. kube-apiserver service catalog PaaS kube-aggregator GET Vision: $ helm install service-catalog $ kubectl create service-announcement .... GET /apis/servicecatalog/subscription/database-prod-wordpress
  39. 39. Status • will be part of Kubernetes 1.6 as an alpha • https://github.com/kubernetes/sample-apiserver • potentially kube-aggregator integrated into kube-apiserver in 1.7
  40. 40. Links • https://docs.google.com/document/d/1y16jKL2hMjQO0trYBJJSczPA Wj8vAgNFrdTZeCincmI/ Two Ways to Extend the K8s API - Add resources to a Kubernetes API with TPR or AA • https://github.com/kubernetes/community/blob/master/contributor s/design-proposals/aggregated-api-servers.md • https://gist.github.com/philips/a97a143546c87b86b870a82a753db1 4c - Kubernetes Third-Party Resource Users
  41. 41. https://github.com/kubernetes/community/blob/master/contributors/devel/client-libraries.md
  42. 42. Backup
  43. 43. Restful http API / /version /api /api/v1/pods /api/v1/pods/status /apis /apis/batch /apis/batch/v2alpha1 /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjo /apis/batch/v1beta1 /apis/batch/v1beta1/jobs /apis/batch/v2alpha1/jobs Group Version ResourceHTTP paths: In Go: gvk := schema.GroupVersionKind{Group: “batch“, Version: “v2alpha1“, Kind: “Job“} obj := api.Scheme.New(gvk) codec := api.Codecs.LegacyCodec(gvk.GroupVersion()) codec.Decode(reqBody, gvk, obj) type Job struct { metav1.TypeMeta metav1.ObjectMeta Spec JobSpec Status JobStatus } pkg/apis/batch/v2alpha1/types.go type TypeMeta struct { Kind string APIVersion string } type ObjectMeta struct { Name string ... }
  44. 44. Restful http API / /version /api /api/v1/pods /api/v1/pods/status /apis /apis/batch /apis/batch/v2alpha1 /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjo /apis/batch/v1beta1 /apis/batch/v1beta1/jobs MaxInFlightLimit TimeoutForNonLongRunningRequests Panic Recovery CORS Authentication Audit Impersonation Authorization k8s.io/apiserver/pkg/server. DefaultBuildHandlerChain „Filters“ k8s.io/apiserver/pkg/server/routes/index.go – / k8s.io/apiserver/pkg/server/routes/version.go – /version k8s.io/apiserver/pkg/server/routes/swagger.go – /swaggerapi k8s.io/apiserver/pkg/server/routes/openapi.go – /swagger.json „Routes“ mux k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST AddSupportedResourcesWebService – /apis/batch/v2alpha1 k8s.io/apiserver/pkg/endpoints.APIInstaller.Install /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjobs ... WithRequestInfo ctx.RequestInfo
  45. 45. Restful http API mux k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST AddSupportedResourcesWebService – /apis/batch/v2alpha1 k8s.io/apiserver/pkg/endpoints.APIInstaller.Install /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjobs ... pkg/apis/batch type Jobs struct pkg/apis/batch/v2alpha1 type Jobs struct api.Scheme k8s.io/apiserver pkg/api api.Scheme.Convert(&internalJob, &v2alohaJob) /apis/batch/v2alpha1/jobs GET PUT POST DELETE ... /status /scale /proxy ... subresources
  46. 46. Restful http API mux pkg/apis/batch type Jobs struct pkg/apis/batch/v2alp ha1 type Jobs struct api.Sche me api.Scheme.Convert(&job, &v1job) POST /apis/batch/v2alpha1/jobs k8s.io/apiserver pkg/endpoints/handlers.CreateNamedResource binary JSON payload Go struct v2alpha1.Job HTTP Request Go struct internal.Job Store k8s.io/apiserver pkg/registry/generic Storage k8s.io/apiserver pkg/storage/etcd3 ProtoBuf Job Go struct v2alpha1.Job etcd
  47. 47. type Scheme struct • AddKnownTypes(gv, obj Object) • Default(src Object) • Copy(src Object) Object • Convert(in, out interface{}) • New(gvk) Object ApiGroup pkg/apis/batch pkg/apis/batch/v1 pkg/apis/batch/v2alpha1 pkg/apis/batch/register.go pgk/apis/batch/install Group Version Kind Resource type Object interface • GetObjectKind() string client-go/pkg/api.Scheme client-go/pkg/api.Codecs Discovery type APIGroupList struct type APIVersions struct type APIResourceList struct GroupVersionKind „gvk“ GroupVersionResource Unversioned types Unstructured List Registry / Storage type Storage interface type Lister interface type Updater interface type Getter interface type Deleter interface .... deepcopy-gen conversion-gen defaulting-gen Code Generation type OwnerReference struct type ObjectReference struct type TypeMeta struct type ObjectMeta struct Meta api.Scheme api.Codecs api.Registry api.GroupFactoryRegistry Globals

×