In this talk, we walk through using Kubespray to deploy a Kubernetes cluster on GCP. We also discuss customization and common operations.

1. June 12, 2018 | 5:00 pm Silicon Valley, CA, USA
Deploying
Kubernetes with
kubespraySergey Matykevich
Cloud Foundry Engineer
Altoros

2. Kubernetes automation
● kops
● kubeadm
● kubo
● kubespray

3. kops
Pros:
● Automates both infrastructure creation
and cluster startup
● Uses cloud providers native features
● Support updates and upgrades
● Is maintained by kubernetes team
Cons:
● Limited infrastructure support (AWS, GCP)
● Still in alpha

4. kubeadm
Pros:
● The most flexible tool
● Suitable for bare metal installations
Cons:
● Don’t provision infrastructure.
● Require a lot of manual work

5. kubo
Pros:
● Uses BOSH
● Support multiple cloud providers
● Monitors health of your cluster
Cons:
● Uses BOSH
● Require BOSH specific knowledge
● Might be difficult to customize

6. Kubespray
Pros:
● Supports multiple deployment
● Reasonable security by default
● Automatic HA
● Supports upgrades
● Under the hood uses kubeadm
● If you know ansible you know Kubespray
● Easy to customize for exposed items
Cons:
● If you pull straight from master you might
be in trouble fast
● Uses Ansible
● Can be slow
● Not everything is idempotent (cert
changes)
● Still some lacking items, like l7 load
balancers in gcp
● Can get complicated for customizations
not exposed

7. Kubespray
A couple of recommendations:
Use a release version to base your work off of
KubeSpray uses ansible so one thing I would recommend if you are experimenting or just deploying to
different environments is use --flush-cache
What am I showing today?
● Deploy to GCP
● Use terraform to follow infra as code
● Cookie cutter templates for multiple environments

8. Prerequisites
GCP Service account

9. Step 1: Create ssh keys
# Create new rsa keys
if [ ! -f cust_id_rsa ] ; then
ssh-keygen -t rsa -b 2048 -f cust_id_rsa -q -N ""
fi
# Load keys
eval $(ssh-agent)
ssh-add -D
ssh-add cust_id_rsa

10. Step 2: Run terraform
terraform init
terraform plan -out cust.plan
terraform apply cust.plan

11. Private version
GCP network
Nat
gateway
Load
balancer
Private subnet
jumpbox
Master instance
group
Worker instance
group

12. Public version
GCP network
Load
balancer
Public subnet
jumpbox
Master instance
group
Worker instance
group

13. Step 3: Get cluster ips
bastion_ip=$(terraform output bastion_ip)
lb_ip=$(terraform output lb_ip)
master_ips=$(gcloud compute instances list --filter="${res_prefix}-masters" --format=json |jq -r
'.[].networkInterfaces[].networkIP'|tr "n" " "| sed -e "s/ {1,}$//")
node_ips=$(gcloud compute instances list --filter="${res_prefix}-workers" --format=json |jq -r
'.[].networkInterfaces[].networkIP'|tr "n" " "| sed -e "s/ {1,}$//")

14. Step 4: Clone kubespay repo
git clone https://github.com/kubernetes-incubator/kubespray.git
cd kubespray && git checkout ${kubespray_version}

15. Step 5: Apply customizations
# bring in modifications, in particular security changes
rsync -avz $git_root/mods/roles/* roles/

16. Step 5: Generate inventory
[all]
master1 ansible_ssh_host=10.0.0.8 ip=10.0.0.8
...
[kube-master]
master1
master2
master3
[etcd]
master1
master2
master3
...

17. Step 6: Run the playbook
ansible-playbook -vvv -i inventory/hosts.ini cluster.yml --flush-cache

18. Step 7: Connect to the cluster
#change the admin.conf file to the current lb_ip
sed -i -e "s/lb-apiserver.kubernetes.local/$lb_ip/g" inventory/artifacts/admin.conf
## echo out set KUBECONFIG
echo "Set your KUBECONFIG"
echo "export KUBECONFIG=$(pwd)/inventory/artifacts/admin.conf"

19. Additional playbooks
upgrade-cluster.yml
scale.yml

20. Questions?