Kubernetes deployment on bare metal with container linux

Kubernetes Deployment on
Bare Metal with Container
Linux
資訊與通訊研究所
Mac Chiang (蔣是文)

Copyright 2017 ITRI 工業技術研究院
Agenda
• Why bare metal?
• Why Container Linux?
• How to deployment?
• Conclusion
2

Why bare metal?
• Hardware can’t support virtualization
▪ CPU Model: Dual Core AMD Opteron(tm) Processor 270
• Better performance
▪ Bare metal vs. VM
3

Why Container Linux (CoreOS)?
• Lightweight Linux
• Container optimized OS
• Security focused
• Auto update
• Integrated well with Kubernetes
4

Agenda
• Why bare metal?
• Why Container Linux?
• How to deployment?
• Conclusion
5

Deployment Approach
• Manual Installation (Step by Step)
▪ https://coreos.com/kubernetes/docs/latest/getting-started.html
• Matchbox + ignition (Recommended)
▪ https://github.com/coreos/matchbox/tree/master/examples/groups/k8s-
install
6

Environment
Deployment Server
Node1:Controller,Etcd Node2: Worker Node3: Worker
7

CoreOS + Kubernetes Steps
• Install CoreOS
• Setup an etcd cluster
• Generate the certificates for Kubernetes components
• Deploy a controller (master) node
• Deploy worker nodes
• Configure kubectl to work with our cluster
• Deploy the add-ons
▪ DNS
▪ Dashboard
8

Install CoreOS
• PXE and iPXE
▪ Booting with iPXE
▪ Booting with PXE
▪ Required RAM :1024M+
• Disk
▪ Installing to Disk
Boot from
PXE or iPXE
Boot from ISO
Install to Disk
coreos-install -d /dev/sda -c cloud-config.yaml
9

What’s etcd?
• Distributed key, value store
• Used for configuration and monitoring store
• Used for Service discovery
• JSON/REST API
10

Deploy etcd Cluster
Single-Node/Development Multi-Node/Production
https://coreos.com/os/docs/latest/cluster-architectures.html 11

What’s flannel?
• A virtual network
that gives a
subnet to each
host for use with
container
runtimes
12

Deployment Options
• MASTER_HOST
▪ Publicly routable IP of master node.
a. Worker nodes must be able to reach the
master node(s) via this address on port
443
▪ Multiple master nodes
a. Network load balancer
b. DNS configure
• ETCD_ENDPOINTS
▪ List of etcd machines
(http://ip1:port,http://ip2:port,http://ip3:p
ort)
• POD_NETWORK=10.2.0.0/16
▪ The flannel overlay network will provide
routing to this network.
• SERVICE_IP_RANGE=10.3.0.0/24
▪ The CIDR network to use for service
cluster VIPs (Virtual IPs)
▪ Handled by a local kube-proxy service to
each host
• K8S_SERVICE_IP=10.3.0.1
▪ The VIP (Virtual IP) address of the
Kubernetes API Service.
• DNS_SERVICE_IP=10.3.0.10
▪ The VIP (Virtual IP) address of the cluster
DNS service.
13

Generate Kubernetes TLS Assets
• Root CA Public Key
▪ ca.pem
• API Server Public & Private Keys
▪ apiserver.pem
▪ apiserver-key.pem
• Worker Node Public & Private Keys
▪ ${WORKER_FQDN}-worker.pem
▪ ${WORKER_FQDN}-worker-key.pem
• Cluster Admin Public & Private Keys
▪ admin.pem
▪ admin-key.pem
https://coreos.com/kubernetes/docs/latest/openssl.html
14

Deploy Kubernetes Master Node
• Configure Service Components
▪ TLS Assets
▪ Network Configuration
▪ Docker Configuration
▪ Create the kubelet Unit
▪ Set Up the kube-* Pod
a. kube-apiserver
b. kube-proxy
c. kube-controller-manager
d. kube-scheduler
• Start Services
▪ Load Changed Units
▪ Configure flannel Network
▪ Start kubelet
▪ Basic Health Checks
15

Master TLS Assets
• /etc/kubernetes/ssl/ca.pem
• /etc/kubernetes/ssl/apiserver.pem
• /etc/kubernetes/ssl/apiserver-key.pem
16

Network & Docker Configuration
/etc/flannel/options.env
FLANNELD_ETCD_ENDPOINTS=${ETCD_ENDPOINTS}
17

Kubelet Unit and Kube-* PODs
/etc/systemd/system/
kubelet.service
/usr/lib/coreos/kubelet-wrapper
--pod-manifest-path=/etc/kubernetes/manifests
Hyperkube
/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-proxy.yaml
/etc/kubernetes/manifests/kube-controller-manager.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml
An all-in-one binary for the
Kubernetes server
components
18

Start Services
• Load Changed Units
• Configure flannel Network
• Start kubelet
• Basic Health Checks
curl http://127.0.0.1:8080/version
19

Deploy Kubernetes Worker Node
• Configure Service Components
▪ TLS Assets
▪ Networking Configuration
▪ Docker Configuration
▪ Create the kubelet Unit
▪ Set Up the kube-proxy Pod
▪ Set Up kubeconfig
• Start Services
▪ Load Changed Units
▪ Start kubelet, and flannel
20

Worker TLS Assets
• /etc/kubernetes/ssl/ca.pem
• /etc/kubernetes/ssl/${WORKER_FQDN}-worker.pem
• /etc/kubernetes/ssl/${WORKER_FQDN}-worker-
key.pem
21

Kubelet Unit and kube-proxy/kubeconfig
/etc/systemd/system/
kubelet.service
/usr/lib/coreos/kubelet-wrapper
--api-servers=https://${MASTER_HOST}
--kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
--pod-manifest-path=/etc/kubernetes/manifests
Hyperkube
/etc/kubernetes/manifests/kube-proxy.yaml
An all-in-one binary for the
Kubernetes server
components
22

Start Services
• Load Changed Units
• Start kubelet, and flannel
23

Configure kubectl
• Download the kubectl Executable
• Configure kubectl
▪ Master server host
▪ Root CA public key
▪ Cluster admin public & private Keys
• Verify kubectl Configuration and Connection
kubectl get nodes
NAME LABELS STATUS
X.X.X.X kubernetes.io/hostname=X.X.X.X Ready
• Enabling shell autocompletion
echo "source <(kubectl completion bash)" >> ~/.bashrc
24

Deploy the Add-ons
• DNS
• Dashboard
kubectl port-forward kubernetes-dashboard-xxxx 9090 --namespace=kube-
system
Then visit http://127.0.0.1:9090 in your browser.
25

Kube Dashboard
namespace=kube-system
26

What’s MatchBox?
• HTTP and gRPC service that renders signed Ignition
configs, cloud-configs, network boot configs, and
metadata to machines to create CoreOS clusters
27

Machbox workflow
https://github.com/coreos/matchbox/blob/master/Documentation/matchbox.md
28

Matchbox Steps
• Get CoreOS
• Generate TLS assets
• Prepare groups, profiles and ignition files
• Setup dnsmasq and matchbox container
• Start deployment
• Configure kubectl to work with our cluster
• Check all PODs and Services
29

Get CoreOS
./scripts/get-coreos channel version
examples/assets/
└── coreos
└── 1298.6.0
├── CoreOS_Image_Signing_Key.asc
├── coreos_production_image.bin.bz2
├── coreos_production_image.bin.bz2.sig
├── coreos_production_pxe_image.cpio.gz
├── coreos_production_pxe_image.cpio.gz.sig
├── coreos_production_pxe.vmlinuz
└── coreos_production_pxe.vmlinuz.sig
https://github.com/coreos/matchbox/tree/master/scripts
30

Generate TLS Assets
./scripts/tls/k8s-certgen -h
Usage: k8s-certgen
Options:
-d DEST Destination for generated files (default: .examples/assets/tls)
-s SERVER Reachable Server IP for kubeconfig (e.g. node1.example.com)
-m MASTERS Controller Node Names/Addresses in SAN format
(e.g. IP.1=10.3.0.1,DNS.1=node1.example.com)
-w WORKERS Worker Node Names/Addresses in SAN format
(e.g. DNS.1=node2.example.com,DNS.2=node3.example.com)
-h Show help
31

Prepare groups, profiles and ignition
examples/
├── assets
│ ├── coreos
│ │ ├── 1298.6.0
│ │ └── tls
├── groups
│ ├── install.json
│ ├── node1.json
│ ├── node2.json
│ └── node3.json
├── profiles
│ ├── install-reboot.json
│ ├── k8s-controller.json
│ └── k8s-worker.json
└──ignition
├── install-reboot.yaml
├── k8s-controller.yaml
└── k8s-worker.yaml
https://github.com/coreos/matchbox/tree/master/examples/groups/k8s-install
32

Installation Flow
install.json
install-
reboot.json
install-
reboot.yaml
curl
"{{.ignition_endpoint}}?{{.request.r
aw_query}}&os=installed" -o
ignition.json
node1.json
k8s-
controller.json
"selector": {
"os": "installed",
"mac": "00:26:2d:06:ff:bc"
},
k8s-
controller.yaml
"coreos_channel": "stable",
"coreos_version": “1298.6.0",
33

Setup dnsmasq and matchbox
• Dnsmasq
docker run --name dnsmasq --cap-add=NET_ADMIN --network="host" -v
$PWD/dnsmasq.conf:/etc/dnsmasq.conf:z quay.io/coreos/dnsmasq -d
• Matchbox
docker run -p 8080:8080 --rm -v $PWD/example:/var/lib/matchbox:Z
quay.io/coreos/matchbox:latest -address=0.0.0.0:8080 -log-level=debug
Notice:
Don’t forget to open firewall port for matchbox(8080), dns, tftp and dhcp
34

PXE boot
time="2017-04-05T07:31:13Z" level=info msg="Starting matchbox HTTP server on 0.0.0.0:8080"
time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /boot.ipxe"
time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /ipxe?uuid=03000200-0400-0500-0006-000700080009&mac=00-
26-2d-07-00-78&domain=k8s.itri&hostname=WR1-43&serial=To%20Be%20Filled%20By%20O.E.M."
time="2017-04-05T07:34:03Z" level=debug msg="Matched an iPXE config" labels=map[uuid:03000200-0400-0500-0006-
000700080009 mac:00:26:2d:07:00:78 domain:k8s.itri hostname:WR1-43 serial:To Be Filled By O.E.M.] profile=install-reboot
time="2017-04-05T07:34:03Z" level=info msg="HTTP GET /assets/coreos/current/coreos_production_pxe.vmlinuz"
time="2017-04-05T07:34:04Z" level=info msg="HTTP GET /assets/coreos/current/coreos_production_pxe_image.cpio.gz"
time="2017-04-05T07:36:29Z" level=info msg="HTTP GET /ignition?uuid=03000200-0400-0500-0006-
000700080009&mac=00-26-2d-07-00-78&os=installed"
time="2017-04-05T07:36:29Z" level=debug msg="Matched an Ignition or Fuze template" group=node3
labels=map[uuid:03000200-0400-0500-0006-000700080009 mac:00:26:2d:07:00:78 os:installed] profile=k8s-controller
matchbox logs
Demo: https://youtu.be/z9eYOuWLc8k
35

Configure kubectl
• Use the generated kubeconfig directly
KUBECONFIG=examples/assets/tls/kubeconfig
• Overwrite kubeconfig
cp examples/assets/tls/kubeconfig ~/.kube/config
36

Check all PODs and Services
[root@centos7 matchbox]# kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-v1.2.0-4088228293-7vwxd 2/2 Running 0 15h
kube-system kube-apiserver-10.201.3.44 1/1 Running 0 15h
kube-system kube-controller-manager-10.201.3.44 1/1 Running 0 15h
kube-system kube-dns-782804071-j52dv 4/4 Running 0 15h
kube-system kube-dns-autoscaler-2715466192-krz0p 1/1 Running 0 15h
kube-system kube-proxy-10.201.3.42 1/1 Running 0 15h
kube-system kube-scheduler-10.201.3.44 1/1 Running 0 15h
kube-system kubernetes-dashboard-3543765157-xj185 1/1 Running 0 15h
[root@centos7 matchbox]# kubectl get svc --all-namespaces
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 10.3.0.1 <none> 443/TCP 15h
kube-system heapster 10.3.0.95 <none> 80/TCP 15h
kube-system kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 15h
kube-system kubernetes-dashboard 10.3.0.66 <none> 80/TCP 15h
37

Conclusion
• Container Linux (CoreOS) is a good choice for bare
metal & production
• Manual installation vs. Matchbox+ignition
• What’s next?
▪ Try it
▪ Join Kubernetes Taiwan User Group
▪ Kubernetes Training Courses and Playground
a. https://www.katacoda.com/courses/kubernetes
b. https://www.katacoda.com/courses/kubernetes/playground
38

Thank you!
macchiang@itri.org.tw
Kubernetes Taiwan User Group

Kubernetes deployment on bare metal with container linux

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Kubernetes deployment on bare metal with container linux

Ähnlich wie Kubernetes deployment on bare metal with container linux (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Kubernetes deployment on bare metal with container linux