We recently moved our workloads onto a self Hosted K8s environment starting from Bare Metal. In this talk I would like to explain why and how and share our lessons learned.
2. 2Self Hosted Bare Metal Kubernetes for SMEs
Who We Are
● n-fuse GmbH
● Since 2001
● ~25 Engineers
● Branches in Stuttgart, Germany and Riga, Latvia
● Accompanying our clients from idea to production
● In hard- and software projects
● Customers ranging from startups to top 10 DAX
www.n-fuse.co
3. 3Self Hosted Bare Metal Kubernetes for SMEs
Outline of this Talk
●
What is Kubernetes and why to use it?
●
Installation process and caveats
●
Architecture of our environment
●
Monitoring, backup
●
How we use it, CI/CD
●
Outlook
...and a lot of real world experiences sprinkled over it
4. 4Self Hosted Bare Metal Kubernetes for SMEs
What is Kubernetes?
● Open source container orchestrator that automates scale out of container
operations
– Docker in most cases
● Focuses completely on containers and is itself built as a set of containers
● Now maintained by the CNCF which is in-turn a member of the Linux
Foundation
5. 5Self Hosted Bare Metal Kubernetes for SMEs
Why to Use K8s?
● All the benfits of containers
● Very good fit for µ-services
● Low overhead
● The ecosystem
6. 6Self Hosted Bare Metal Kubernetes for SMEs
How to set up K8s? Just some Ways
●
Developer desktop
– Minikube
●
Turnkey Hosted Kubernetes
– Google Kubernetes Engine
– AWS EKS
– Azure Container Kubernetes Service
●
Custom
– Using IaaS (AWS, Google Computing Engine, Azure)
●
Using Tools like Kubespray, kops
– On Premise/ Bare Metal
●
Using Tectonic, Rancher, SuSE CaaS etc.
●
From Scratch on CoreOS our pick
7. 7Self Hosted Bare Metal Kubernetes for SMEs
CoreOS
●
Aka Container Linux
●
Very minimal Linux distro optimized
– Uses Gentoo’s build system :)
●
Heavy user of Systemd
●
3 release channels (alpha, beta, stable)
●
Automatic updates built-in
– Using two partition strategy like ChromeOS for fault resilience
●
Redhat’s takeover of CoreOS should not interfere as they announced to
keep it as is
– If you don’t trust them, try flatcar Linux, a “friendly fork”
8. 8Self Hosted Bare Metal Kubernetes for SMEs
Bare Metal Provisioning of CoreOS with Matchbox
●
Using Matchbox provision the rest of the CoreOS fleet
– Toolbelt to install a whole cluster through network boot (using PXE etc.)
– Running one a “bootstrapping machine” to serve DHCP, images and configs
●
All provisioning configs are described in JSON
9. 9Self Hosted Bare Metal Kubernetes for SMEs
Our K8s Environment – Logical View
10. 10Self Hosted Bare Metal Kubernetes for SMEs
Our K8s Environment – Physical View
11. 11Self Hosted Bare Metal Kubernetes for SMEs
Physical Facts
●
Located in DC of a housing provider in Frankfurt
– Latency matters!
●
Redundant 10 GbE Fibre Network
●
4-Node Server á
– 12 Cores
– 64 GB RAM
– 1x 800 GB DC grade SSD
– 2x 4 TB spinner’s
12. 12Self Hosted Bare Metal Kubernetes for SMEs
Networking
●
VLANs and interface bonding on physical level
●
Flannel for K8s/container subnets
●
KubeDNS for service discovery
●
Planned upgrades
– Networking with Project Calico
– Service discovery with CoreDNS
13. 13Self Hosted Bare Metal Kubernetes for SMEs
Support of K8s
●
Documentation available online, but doesn’t cover specific configuration
●
Bare metal installation docs aren’t up-to-date
– CoreOS had a installation guide for bare metal, but from one day to the other it was replaced
by a Tectonic installation guide
●
Support channels
– Slack: too crowded; hard to find what you need; nobody really answers when you ask
question
– IRC: land of bots (mostly); no reaction to questions at all; no history at all
– Mailing list: haven’t tried to write ourselves; at least searchable
– GitHub: here you can find proof that you’re not the only one with your problem; some issues
just change milestones, but don’t get resolved
– StackOverflow: by now the best choice; best used together with documentation
– Various blogs: some good recipes, but a lot of them are outdated due to fast evolution of
K8s
– Conference videos: one of good sources of information, but you need to pay attention to
outdated topics
14. 14Self Hosted Bare Metal Kubernetes for SMEs
Versions, Versions, Versions...
●
Most annoying: K8s (even the latest version) doesn’t support the latest
Docker version and lags behind for months
●
Not all components (like ETCD, flannel etc.) versions are compatible with
each other and with K8s version of your choice
– Docker
– ETCD
– Flannel
●
CHANGELOG.md of K8s is good source of version compatibility with
external dependencies
15. 15Self Hosted Bare Metal Kubernetes for SMEs
Docker Version Support Matrix
●
Currently K8s supports just following Docker versions
●
All supported Docker versions are End-of-Life
●
We’re stuck with older CoreOS Linux releases because they have too
recent Docker versions for K8s
– Watch out for CoreOS auto update!
Docker version 1.10.3 1.11.2 1.12.6 17.03.x 18.05.x
K8s version (latest)
1.6 x x x
1.7 x x x
1.8 x x x x
1.9 x x x
1.10 x x x -
1.11 ? ? ? -
16. 16Self Hosted Bare Metal Kubernetes for SMEs
ETCD Version
●
Pay attention to maximum ETCD version supported by K8s release of your
choice
●
Biggest issue: you cannot downgrade ETCD
17. 17Self Hosted Bare Metal Kubernetes for SMEs
TLS
●
All communication within ETCD and K8s cluster is secured by TLS
●
For all intra-communication you can and should use self generated CA and
certs
– CFSSL is a good tool to generate CA, certs etc.
●
Only external services – like ingresses – have Let’s Encrypt or commercial
certs
18. 18Self Hosted Bare Metal Kubernetes for SMEs
Security and K8s
●
Most important: protect access to the API on master and nodes (via proxy)
– We use a basic IPTables firewall to disabled access from the net
– We use client certificate based authentication
●
RBAC (by default on from K8s v1.8) for resource access control
●
NetworkPolicy resources to control intra-pod and external communication
– Doesn’t work with Flannel, but works with Project Calico
19. 19Self Hosted Bare Metal Kubernetes for SMEs
Installation Hints’n’tips
● Use UTC as your timezone everywhere
– Doing so will help later get in sync when reviewing logs etc.
● By default older CoreOS releases uses older flannel version
– That you can easily upgrade by creating respective systemd unit
20. 20Self Hosted Bare Metal Kubernetes for SMEs
Workload partitioning
●
No-brainer: cordoning (disabling scheduling) on some nodes
●
Simplest, yet most effective in small clusters: node selection using labels
●
Advanced: affinity and anti-affinity that still uses node labels, but in regex
way
●
More advanced: using taints and tolerations that ensures pods aren’t
scheduled onto inappropriate nodes
21. 21Self Hosted Bare Metal Kubernetes for SMEs
CI/CD with Gitlab
●
Running on the K8s cluster
– Deployed via https://github.com/sameersbn/docker-gitlab
– Maybe helm chart in the future (currently in Alpha/Beta)
●
Setting up Docker registry part of it allows to store private Docker images
and use them within K8s clusters
●
Gitlab has K8s integration for CI/CD
– requires some tinkering to get it running on a non-remote cluster
●
For CI/CD to work Gitlab installs the Gitlab-Runner
– Builds/tests your projects or Docker images
– Push them to registry
– Deploy to K8s cluster
22. 22Self Hosted Bare Metal Kubernetes for SMEs
Developer Access to K8s
●
Per developer SSH-tunnel
●
kubectl access through tunnel via developer certificate
23. 23Self Hosted Bare Metal Kubernetes for SMEs
Backup Strategy
●
Contents of master node as well host-dir based persistent volumes are
rsynced to another host on nightly basis
24. 24Self Hosted Bare Metal Kubernetes for SMEs
Monitoring
●
Low-key system tools
– Docker stats
– Linux top
●
Kubernetes Dashboard (relying on data from built-in Heapster)
– No time travel
– No alerting
●
Planned: Prometheus
25. 25Self Hosted Bare Metal Kubernetes for SMEs
Updates
●
CoreOS
– In general everything happens automatically out-of-the-box except you can’t use it by
default due to version conflicts
– Instead use CoreRoller tool to run your own image store
– Updates can be scheduled to off-hours and various nodes
●
K8s
– Quite easy within minor releases: just change version numbers in systemd unit as well K8s
system manifest files and restart systemd unit
– Between major releases, carefully study the CHANGELOG to find what has changed in
kubelet startup options (depreciations, additions) and change systemd unit accordingly
26. 26Self Hosted Bare Metal Kubernetes for SMEs
Hints’n’tips for Running Stuff on K8s
●
kubectl is your friend
●
Use Deployments instead of ReplicaSets
– Remember to check manifests made by others before deploying them
●
There’s no restarting for pods like “docker restart”
●
Rolling-update is a great thing. Scale up, change deployment settings or
container image and K8s will do the rest
●
Don’t use pure Kube-registry
– It has issues while setting it up as well using it within cluster
– Instead use Gitlab with enabled Docker registry
27. 27Self Hosted Bare Metal Kubernetes for SMEs
Outlook
●
Persistent volumes featuring network transparent remote storage – ROOK
– We will migrate all host-dir based PVs to ROOK based PVs
●
Highly available master/ ingress
– Two dedicated master nodes
– HA cluster based pacemaker