Cloud Native Night May 2017, Mainz: Talk by Giant Swarm (@giantswarm). Join our Meetup: www.meetup.com/cloud-native-night Abstract: At Giant Swarm, we manage Kubernetes clusters for customers 24/7, both on-premises and in the cloud. That means we do not just set something up and hand it over, but we actually take care that it’s operational and up-to-date at all times. This presentation demonstrates how Giant Swarm are using Operators to codify all operational tasks of managing Kubernetes cluster and distributed applications on top. The operators manage PKI infrastructures, networks, VMs and storage both on-premises and in the cloud.

1. Running Kubernetes in Kubernetes
by Timo Derstappen

2. Overview
1. Background /
Motivation
2. Architecture
3. Operators
4. Demo
5. Outlook

3. Background
We manage Kubernetes clusters for customers 24/7.
Both on-premises and in the cloud.

4. Kubernetes as a Service (KaaS)
DC in Frankfurt,
Germany
Full access to clusters

5. On-premises KaaS
Most of the time we run multiple clusters at our customer
datacenter as well.

6. Why not RBAC, NetworkPolicies etc?
"Soft" multi-tenancy not enough in enterprise context
Different service classi cations
Different environments: separation of dev, test,
prod
Test tooling on new K8s versions

7. People must come to things in their
own time, in their own way, for their
own reasons, or they never truly come
at all. - Dee Hock

8. RBAC, NetworkPolicies FTW
We encourage to use these features and help with
integration.
Small clusters make little sense
Support while processes need to be
adapted
Kubernetes matures - trust over time

9. Introducing:
Giantnetes

10. Motivation
Rather obvious. If you provide others with Kubernetes you
would like to use the same concepts within your product as
well.

11. De nition
G8s: Giantnetes
K8s: Kubernetes == Guest cluster

13. Overview
1. Background /
Motivation
2. Architecture
3. Operators
4. Demo
5. Outlook

16. VMs
Docker container with QEMU tooling
CoreOS started in a KVM
Parameterized via CloudCon g
Certs are handed to the VMs via a
volume

17. Master
Ingress in G8s for the
API
KubeDNS
Calico Policy Controller

18. Networking
Host cluster: Calico
Guest cluster: Flannel /
Calico

19. Networking
One Flannel server per host
One Flannel client per host per
cluster
Flannel VXLAN backend

24. Certi cates
Each cluster has its own root ca
Using PKI backend in Vault
Each service of a cluster has a role in
Vault

25. Iterations of the platform
1. K8s clusters via systemd units over eet
2. K8s manifests to create K8s clusters used as
templates
3. Writing operators

26. Overview
1. Background /
Motivation
2. Architecture
3. Operators
4. Demo
5. Outlook

27. Operators
Codify all operational tasks

28. Third-Party Resources
Easily extend the Kubernetes API

29. KVM operator
Create k8s clusters on bare-metal via KVM based on a TPR

30. AWS operator
Create k8s clusters on AWS based on a TPR

37. The Cluster TPR de nes
Kubernetes
con guration/version
Ingress con guration/version
Flannel con guration
Master/Node con gurations

38. The Cert TPR de nes
TTL
Common Name
SANs
Cluster /
Component

39. Overview
1. Background /
Motivation
2. Architecture
3. Operators
4. Demo
5. Outlook

40. Overview
1. Background /
Motivation
2. Architecture
3. Operators
4. Demo
5. Outlook

41. Micro Operators
Operators should have single
responsibilities.
Keep them simple and maintainable

42. Operatorkit
Our services are based upon microkit. We would like to
reduce boilerplate in our operators and collect them in a
library called operatorkit.

43. Self-hosted
Having Giantnetes and all guest clusters running self-hosted
will further ease the lifecycle management of the clusters.

44. Thank you.
Timo Derstappen
@teemow