Thanks to tools like kubeadm, Terraform or Ansible setting up a Kubernetes cluster on a dedicated environment is getting reachable, but what’s about setting up a bunch of cluster in multiple clouds in automatic way? This is still a challenge. Also if you want to do same in your own datacenter. In this talk we will take a look to the approach to orchestrate and manage a whole set of k8s cluster by the Cluster API project of kubernetes (a subproject of sig-cluster-lifecycle). The main idea behind it is to use the Kubernetes API itself to manage multiple clusters with there master and worker nodes in same way you would manage your PODs - define the needed resources and the responsible controller will take care for providing it. After an overview about the concepts of cluster API, I will show what’s needed to implement a cluster API conform machine class/deployment. There I will see that adding your own provider isn’t that hard as you may aspect. At the end of the day it just requires a simple interface to implement. The corresponding kubermatic controllers we implemented at Loodse are available as open source, so its possible to play around with it. A live demo will show how easy it is to spin up and maintain multiple Kubernetes cluster at different public and on-premise cloud providers over one managing cluster. A final wrap up will summarize the current state of the Cluster API project and the advantages of managing clusters as cattles instead of pets.

1. K8s ClusterAPICreating Kubernetes multi clusters
with ClusterAPI
Stuttgart

2. K8s Cluster API Folie 2
Who I am? Tobi
Who are we working for? Loodse GmbH
Kubernetes?! Short recap
Kubermatic?! What’s that?
Node Deployment Solutions The old way
Cluster API! Make K8s even greater
Hetzner Machine! Implementation of Machine Provider
Demo time! Praise the demo gods!
Review! State, Pro and Cons
Agenda

3. toschneck
@toschneck
Tobias Schneck

4. Founders
Awarded by
Partners

5. K8s Cluster API Folie 5
Kubernetes

6. K8s Cluster API Folie 6
Kubernetes - a short recap

7. K8s Cluster API Folie 7
Kubernetes SIG - Cluster API
● Subproject of sig-cluster-lifecycle
○ Focus cluster deployment and upgrades
○ Other Projects: kops, kubeadm, minikube
● Repos
○ (old, in migration) https://github.com/kubernetes/kube-deploy
○ (new) https://github.com/kubernetes-sigs/cluster-api
● Main participants:
○ Google, Redhat, Heptio, VMware, Loodse …
● Open Development: Get involved!

8. K8s Cluster API Folie 8

9. K8s Cluster API Folie 9
Currently Supported Providers
Bare Metal

10. Node Deployment Solution

11. Why we
implemented
Cluster API?

12. K8s Cluster API Folie 12
• Need Ops to scale the cluster
• No generic approach
• Each implementation needs
specific knowledge
• Scaling of new nodes requires
external dependencies
• No generic node auto scaling
• Huge effort to deploy k8s on a
different platform
Node Node Node
Master
Node
Current Approach

13. K8s Cluster API Folie 13
Kubernetes Cluster BKubernetes Cluster B
Kubernetes Cluster A
Cluster API
Infrastructure Providers
Cloud, On-Prem, Bare Metal
Potential Providers =>
bootstraping CLI
upgrade trigger
Provider Specifc Config
autoscaling
Bootstrapping Machine/Cluster
etcd
Cluster
Machine
MachineSet
Cluster API Server
(ext-apiserver)
group: cluster.,k8s.io
Controller Manager
Machine Controler
Prov. Config
Provider Machine
Actuator
Master Machine(s) NodeMachine(s)
kubeadm kubeadm
API
Server etcd
Kubernetes Cluster B
Master Machine(s) NodeMachine(s)
Provisions
Manage Cluster
and Machines
- Creates Machines
- Bootstraps Cluster
- Upgrades Cluster
API Server + Controllers can be
provided after bootstrapping
Kubermatic
Machine Controller

14. K8s Cluster API Folie 14
kubectl create machines
• Generic cluster scaling
• Pets vs cattle for nodes
• Implementation of generic auto
scaling possible
• Very similar setup for different
provider
Node Node Node
Master
Node
Kubernetes Native Integration
Cluster API

15. K8s Cluster API Folie 15
• A "Machine" is the declarative spec for a Node,
as represented in Kubernetes core.
• After provisioning a new Node matching the
Machine spec is registered.
Machines/Nodes

16. Let’s take a look
at the Code!

17. K8s Cluster API Folie 17
How does a machine look like?
apiVersion: "machine.k8s.io/v1alpha1"
kind: Machine
metadata:
name: machine1
spec:
metadata:
name: node1
providerConfig:
sshPublicKeys: []
cloudProvider: "hetzner"
cloudProviderSpec:
token: MY-HZ-TOKEN
serverType: "cx11"
datacenter: ""
location: "fsn1"
operatingSystem: "ubuntu"

18. K8s Cluster API Folie 18
MachineController
Watches for new Machine resources
Provision the machine using provider-specific drivers
• Machine joins the cluster
• Kubelet creates the Node resource
Machine
Resources

19. K8s Cluster API Folie 19
How does it work?

20. K8s Cluster API Folie 20
How does it work?
● Render Userdata
○ CoreOS, Ubuntu & CentOS
● Create instance at Cloudprovider
○ Install Container Runtime, Kubelet, Kubeadm
○ Use Kubeadm + Token to join node
● Match Node to Machine, set ownerRef

21. K8s Cluster API Folie 21
How to implement a new provider?
// Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}

22. K8s Cluster API Folie 22
How to implement a new provider?
// Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}
github.com/kubermatic/machine-controller/../hetzner/provider.go

23. K8s Cluster API Folie 23
Where to get it and how to use it?
github.com/kubermatic/machine-controller
● Apache2 license
● Contains Machine Controllers for:
AWS, Azure, DigitalOcean, Hetzner, OpenStack, vSphere
kubectl apply -f examples/machine-controller.yaml
kubectl apply -f examples/machine-${my_provider}.yaml

24. K8s Cluster API Folie 24
New -> MachineSet
Set definitions for Machines
apiVersion: "cluster.k8s.io/v1alpha1"
kind: MachineSet
metadata:
name: << MACHINE_NAME >>
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
name: << MACHINE_NAME >>
template:
metadata:
labels:
name: << MACHINE_NAME >>
spec:
providerConfig:
value:
sshPublicKeys:
- "<< YOUR_PUBLIC_KEY >>"
cloudProvider: "vsphere"
cloudProviderSpec:
templateVMName: '<< OS_NAME >>-template'
username: '<< VSPHERE_USERNAME >>'
vsphereURL: '<< VSPHERE_ADDRESS >>'
datacenter: 'Datacenter'
folder: '/Datacenter/vm/e2e-tests'
password: << VSPHERE_PASSWORD >>
cluster: '<< VSPHERE_CLUSTER >>'
datastore: datastore1
allowInsecure: true
cpus: 2
MemoryMB: 2048
operatingSystem: "<< OS_NAME >>"
operatingSystemSpec:
distUpgradeOnBoot: false
containerRuntimeInfo:
name: "<< CONTAINER_RUNTIME >>"
versions:
kubelet: "<< KUBERNETES_VERSION >>"

25. K8s Cluster API Folie 25
New -> MachineDeployment
Set definitions for Machines
apiVersion: "cluster.k8s.io/v1alpha1"
kind: MachineDeployment
metadata:
name: << MACHINE_DEPLOYMENT_NAME >>
namespace: kube-system
spec:
paused: false
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
minReadySeconds: 0
selector:
matchLabels:
name: << MACHINE_NAME >>r
template:
metadata:
labels:
foo: << MACHINE_NAME >>
spec:
providerConfig:
value:
sshPublicKeys:
- "<< YOUR_PUBLIC_KEY >>"
cloudProvider: "vsphere"
cloudProviderSpec:
templateVMName: '<< OS_NAME >>-template'
username: '<< VSPHERE_USERNAME >>'
vsphereURL: '<< VSPHERE_ADDRESS >>'
datacenter: 'Datacenter'
folder: '/Datacenter/vm/e2e-tests'
password: << VSPHERE_PASSWORD >>
cluster: '<< VSPHERE_CLUSTER >>'
datastore: datastore1
allowInsecure: true
cpus: 2
MemoryMB: 2048
operatingSystem: "<< OS_NAME >>"
operatingSystemSpec:
distUpgradeOnBoot: false
containerRuntimeInfo:
name: "<< CONTAINER_RUNTIME >>"
versions:
kubelet: "<< KUBERNETES_VERSION >>"

26. DEMO TIME

27. K8s Cluster API Folie 27
ClusterAPI - Quo Vadis?
● Currently in Alpha
● Types do change in a backwards-incompatible way
● Some questions do not have a final answer yet
● Does not solve all problems, e.g.
○ Bootsraping of master cluster
○ Backup & Restore
● Already allows easy decoupling of general applicable from provider-specific code
● Generic cluster-autoscaler nearly finished!
● Participation from Google & RedHat

28. K8s Cluster API Folie 28
Cluster API <> Kubermatic
Kubermatic is based on Cluster API and Machine Controllers and adds:
● Kubermatic Dashboard
● Backup & Restore Concept
● Cluster Upgrade Mechanism
● Secure Communication between Bootstrapping Cluster and User Cluster
● Control of different on-prem/public Cloud Provider over one Master Plane
● Prometheus Meta Monitoring and Alerting
● ELK Stack for Infrastructure
● User Management (v2.8)
Try it yourself!
(Only AWS, Digital Ocean enabled)
https://cloud.kubermatic.io

29. Questions?

30. Our Conferences in 2019

31. We are hiring!
Hamburg, Berlin, Munich, San Francisco, Gdańsk, …
www.loodse.com
@Loodse