Kubernetes Immutable Boot via Digital Rebar w/ KubeAdm

1. Zero-Configuration Pattern
Provisioning Kubernetes on Unmanaged Infrastructure
Rob @zehicle Hirschfeld, RackN
November, 2017

2. Hang on to your Hats!
Krazy New Stuff
● Immutable Bootstrap
(demo!)
● Node Admission (v1.7)
● Dynamic Kubelet (v1.8)

3. 3
#KubeCon - @zehicle
Rob Hirschfeld (@zehicle)
Involved in Kubernetes since launch
Co-chair of Cluster Ops SIG
Co-Founder of RackN & Digital Rebar Project
We focus on operations automation for bare metal

4. 4
#KubeCon - @zehicle
We’ve been using Kubespray since Kubernetes v1.2
But first… Kubespray
● Very Solid Ansible Playbook
● Strong Community
● Amazing Features like HA & Upgrade
HTTP://bit.ly/SYDkubespray
But….

5. 5
#KubeCon - @zehicle
Why not Kubespray?
We’d like to do better!
● No Centralized Orchestration
● No Inventory Building
● No SSH
● Immutable Booting
● and, much FASTER
I don’t always Ansible, but when I do Ansible, I use Kubespray.

6. 6
#KubeCon - @zehicle
Let’s get Immutable!
What?
● Create, Destroy & Repeat
● Machines recreated, not updated
● Typically “Pre-Baked” images
Why?
● Very repeatable and predictable installation
● Simpler node configuration
● Faster deploy time

7. 7
#KubeCon - @zehicle
Community converging to single install utility!
Leveraging Kubeadm
Basic Three Step Cluster Initialization:
1. Initialize Master
2. Retrieve Token from Initialize
3. Join Nodes with Token
Still requires coordination / synchronization

8. 8
#KubeCon - @zehicle
We need to build underlay infrastructure
But First, Kubeadm Prereqs
Basic Three Step Underlay:
1. install operating system
with network access
2. attach disks (optional?!)
3. install Docker on the machine
Oh, and we need to have some control
mechanism on the nodes too.

9. 9
#KubeCon - @zehicle
A bootstrapping illustration
node01 node02 node03 node04

10. 10
#KubeCon - @zehicle
A bootstrapping illustration
Install
O/S
Install
Docker
Install
O/S
Install
Docker
1
2
Install
O/S
Install
Docker
Install
O/S
Install
Docker
node01 node02 node03 node04
Install Stage

11. 11
#KubeCon - @zehicle
A bootstrapping illustration
Install
O/S
Install
Docker
Kubeadm
-- Init
Cluster
Token
Install
O/S
Install
Docker
1
2
3
4
Install
O/S
Install
Docker
Install
O/S
Install
Docker
node01 node02 node03 node04
master node
(random or selected)

12. 12
#KubeCon - @zehicle
A bootstrapping illustration
Install
O/S
Install
Docker
Kubeadm
-- Init
Cluster
Token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
1
2
3
4 5
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
node01 node02 node03 node04

13. 13
#KubeCon - @zehicle
Later...
A bootstrapping illustration
Install
O/S
Install
Docker
Kubeadm
-- Init
Cluster
Token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
1
2
3
4 5
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
node01 node02 node03 node04 nodeN

14. 14
#KubeCon - @zehicle
A bootstrapping illustration
Install
O/S
Install
Docker
Kubeadm
-- Init
Cluster
Token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
1
2
3
4 5
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
node01 node02 node03 node04 nodeN

15. PSA: THIS IS NOT A NEW INSTALLER
At RackN, we push back against the distro
installer wars (ala OpenStack).
We believe that Kubernetes install tooling
should be a shared community investment.

16. Demo!
Kubeadm
Rebar
Immutable
Bootstrap

17. Pretty Cool! But...
There is more to do
● Adding Nodes requires Token
● Adding Kubelet requires Configuration
● Cluster API (Orchestrating Update)

18. 18
#KubeCon - @zehicle
Benefits!
1. Immutable Configuration
2. Auto Scaling
3. Faster Node Install
4. Centralized Configuration of Cluster
5. Coordinated Upgrades
Still requires coordination / synchronization
https://kubernetes.io/docs/admin/admission-controllers/
Node Admission

19. 19
#KubeCon - @zehicle
NOT Node specific!
Admission control provides an API
mechanism to block creation of new
objects.
In this case, Admission would allow an
external system to validate that new
nodes are known and trusted.
HSM: Hardware Signing Module
Node Admission
Kubelet
API Server
External
Node
Validation
1 Install
2Create
3 Confirm
5Allow
4 Verify

20. 20
#KubeCon - @zehicle
HSM: Hardware Signing Module
Node Admission with HSM
Kubelet
API Server
External
Node
Validation
1 Install
2Create
3 Confirm
5Allow
4 Verify
PKI
A Pass Token B Encrypt Token
Encrypted
Token
E Encrypted
Token
FEncrypted
Token
GVerfiy
HSM ensures unique
identy of machine by
signing secret token.
Only token creater
(PKI) and machine know
the secret. API Server
cannot read or validate
internally.
C Public Key

21. 21
#KubeCon - @zehicle
Frankly, RackN is on the fence.
If injecting a join cluster token
then the external system has
already verified the new node.
Is Node Admission Needed?

22. 22
#KubeCon - @zehicle
https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/
Kubelet Dynamic Configuration
We want to eliminate external configuration tools.
Kubernetes is already a system configuration database!
Can’t we just use that capability to bootstrap the system?
Then we have fewer tools to learn and managed!
(IMHO, this is known as a the bootstrap fallacy)

23. 23
#KubeCon - @zehicle
Ideally, it would be like this...
1. Centrally Configurate
2. Install Kubelet
3. Allow Kubelet to Register
4. Kubelet Configures itself
Kubelet2 Install
3Reg
4Config
API Server1 Config

24. 24
#KubeCon - @zehicle
https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/
Kubelet Dynamic Configuration
1. Install Node and Kubelet
2. Configure Kubelet
3. Allow Kubelet to Register
4. Register Configuration in API
5. Reconfigure Kubelet to use
configuration from API
6. Manage configuration from API
Kubelet
1 Install
3Reg
6Config
API Server
2 Config
4 Config
5 ReConfig

25. 25
#KubeCon - @zehicle
Frankly, RackN is on the fence.
Since we have to boostrap a node
with some configuration, there is
not much difference between
some and all configuration.
We have not eliminated
configuration.
Is Dynamic Configuration Needed?

26. We’re Making Great Progress!
We can automatically
bootstrap a cluster
using open community
tools with minimal
configuration.
And we have room to improve.

27. Thank you
Join In! http://rebar.digital
Follow:
● Rob Hirschfeld > @zehicle
● RackN > @rackngo
● Digital Rebar > @digitalrebar
● Cluster Ops SIG > http://bit.ly/k8sclops