Expert Tips for Successful Kubernetes Deployment - AWS Summit Sydney 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mitch Beaumont
Solutions Architect, Amazon Web Services
Expert tips for successful Kubernetes
deployments on AWS

We Give You The Power To Choose:
ECS EKS
EC2 Fargate EC2 Fargate
1. Choose your
orchestration tool
2. Choose your
launch type

• Started at Google
• Influenced by Google Borg
• Container Orchestrator
• Contributed to the CNCF
Kubernetes 101

Operations as Code
Annotated Documentation
Frequent, small, reversible changes
Refine operation procedures frequently
Anticipate failure
Learn from operational failures
What does operational excellence look like on AWS?

$ kubectl get tips –n aws-summit

GET api/v1/namespaces/aws-summit/tips/{1}
Never build a Kubernetes Cluster the hard way!

Options for Kubernetes cluster setup
Community
• Kops – kubernetes-aws.io
• Kubeadm – toolkit for bootstrapping a cluster
• Kubespray – set of tools for deploying K8s clusters.
Enterprise
• Elastic Container Service for Kubernetes (EKS)
• Red Hat OpenShift
Other options
• CloudFormation, Terraform, Ansible, Puppet

Kops – Kubernetes Operations
Community supported
• SIG AWS
• Kops office hours and Slack channel
Generate CloudFormation or Terraform scripts
kops create cluster
--name cluster1.kubernetes-aws.io
--zones ap-southeast-2a, ap-southeast-2b, ap-southeast-2c
--master-count 3
--master-size m4.large
--node-count 5
--state s3://kubernetes-aws-io
--yes

Op er a t i o n al
Exc el len ce
Sec u r i t y
R el i a b i l i tyP er f o r m an ce
Ef f i c i en cy
C o s t
O p t i m i s a t i o n
M a s t er s
et c d

Elastic Container Service for Kubernetes
mycluster.eks.amazonaws.com
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
Kubectl

Elastic Container Service for Kubernetes
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
etcd
Master
etcd
Master
etcd
Master
AWS Managed
Customer
Managed

aws eks create-cluster --cluster-name summit-cluster
--role-arn arn:aws:iam::1123581321:role/eks-vpc-
EksServiceRole-21345589144
--vpc-id vpc-21345589144
--region us-west-2

In Summary
• Community tools like Kops, kubeadm and
kubespray help remove some of the
effort.

In Summary
effort.
• Kops provisions infrastructure, as well as
the Kubernetes cluster components.

In Summary
effort.
• Use CloudFormation or Terraform to
manage Operations as Code.

In Summary
effort.
• Use CloudFormation or Terraform to
manage Operations as Code.
• EKS provides a fully managed masters
and etcd. (control plane)

GET api/v1/namespaces/aws-summit /tips/{2}
Consider your networking options

“Every pod should have it’s own IP
address, and all pods should be able to
talk to one and other”
Node Node
Pod Pod
Networking With Kubernetes

What Is CNI
Network
Plugin
Runtime
Network
• A way for Kubernetes to tell an
underlying SDN that it wants to connect a
container to a network.
• Standards based pluggable architecture
for container networking.
• API for writing plugins to configure
network interfaces for containers.
• CNCF Project

Popular Solutions For Kubernetes Networking
1 2 3 4 5 6 7 8
StarsonGithub

ENI
VPC Subnet – 172.16.18.0/24
Bridge
Pod CIDR
10.244.10.0/24
Destination Via
10.244.10.0/24 172.16.18.101
10.244.11.0/24 172.16.18.102
… …
Node IP
172.16.18.101
“50 Route Limit”
AWS Route Table
Networking With Kubenet

Overlay Networks
• Can't get enough IP space? (subnets not
sized correctly)
• Your existing network cannot handle the
number of routes required (VPC route
tables have a limit of 50 routes).
• You want to tap in to additional
capabilities that a specific overlay
network provides – network policies

src: node
dst: node
ENI ENI
VPC Subnet – 10.0.0.0/24
Instance 2
Bridge Bridge
Flannel0
src: pod
dst: pod
Flannel0
Flannel Pod Cidr
10.244.0.0/16
Flannel Pod Cidr
10.244.0.0/16
Networking With Flannel

Do I Need An Overlay Network?

Nginx Pod
Java Pod
ENI
Secondary IPs:
10.0.0.1
10.0.0.2
Veth IP: 10.0.0.1
Veth IP: 10.0.0.2
Nginx Pod
Java Pod
ENI
Veth IP: 10.0.0.20
Veth IP: 10.0.0.22
Secondary IPs:
10.0.0.20
10.0.0.22
ec2.associateaddress()
VPC Subnet – 10.0.0.0/24
Instance 1 Instance 2
Networking With Amazon VPC CNI

How Do I Achieve Segmentation?
Frontend
Cats Dogs
kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
name: default-deny
spec:
podSelector:
matchLabels: {}
prod-namespace

Frontend
Cats Dogs
prod-namespace
kind: NetworkPolicy
metadata:
name: public-to-frontend
spec:
podSelector:
matchLabels:
role: frontend
ingress:
- from:
- ipBlock:
cidr: "0.0.0.0/0"
ports:
- protocol: TCP
port: 80

Frontend
Cats Dogs
prod-namespace
kind: NetworkPolicy
metadata:
name: frontend-to-cats
spec:
podSelector:
matchLabels:
role: cats
ingress:
- from:
- podSelector:
matchLabels:
role: “frontend”
ports:
- protocol: TCP
port: 80

In Summary
• All pods need a real IP and must be able
to communicate with each other.

In Summary
• Kops defaults to Kubenet.

In Summary
• Kubernetes adopted the CNI container
networking standard.

In Summary
• Amazon EKS supports the “amazon-vpc-
cni” plugin.

In Summary
• Amazon EKS supports the “amazon-vpc-
cni” plugin.
• Segmentation can be achieved using
network policies.

Monitor all the things!

Visibility About Your Kubernetes Cluster
ApplicationContainerNodeCluster

Building A Log Aggregator
An open source data collector providing a unified logging layer,
supported by 500+ plugins connecting to many types of systems.
A distributed, RESTful search and analytics engine.
(Amazon Elasticsearch)
Lets you visualise your Elasticsearch data.

WorkerWorkerMaster
WorkerWorkerMaster
ASG
AZ1
Region
AZ2
ASG
Amazon
CloudWatch
Logs
Amazon
Elasticsearch
Service
Kibana
Fluentd
DaemonSet
Kubectl logs
Elasticsearch (index),
Fluentd (store), and
Kibana (visualise)
Logs

In Summary
• Monitoring tools should compliment
dynamic nature of containers.

In Summary
• Application and cluster log data help us
Learn from operational failures.

In Summary
• It is essential that you Refine operation
procedures frequently.

In Summary
• It is essential that you Refine operation
procedures frequently.
• Partners like DataDog offer great
solutions.

Build, Ship, Run …

Continuous Deployment to Kubernetes
Developer
AWS CodePipeline
AWS CodeCommit AWS CodeBuild AWS Lambda
Amazon ECR Kubernetes

Developer
AWS CodePipeline
• Code is committed to AWS CodeCommit.
• PR created for review of changes.
• Changes merged to master branch.
• AWS CodePipeline detects changes and starts
moving changes through the pipeline.

Developer
AWS CodePipeline
• AWS CodeBuild packages code changes
and dependencies and builds a Docker
image.
• Other pipeline stages can be included to
test code and the package, also using
AWS CodeBuild.
• The Docker Image is pushed to Amazon
ECR.

Developer
AWS CodePipeline
• AWS CodePipeline invokes an AWS
Lambda function which updates the
Kubernetes deployment file with the
image tag.
• AWS Lambda invokes Kubernetes API
(Python SDK) to update application
deployment.
• A rolling update is performed of the pods
to match the Docker image that was
created.

mport yaml, boto3, botocore, json, zipfile
from os import path
from kubernetes import client, config
s3 = boto3.resource('s3')
code_pipeline = boto3.client('codepipeline')
ssm = boto3.client('ssm')
def lambda_handler(event, context):
cplJobId = event['CodePipeline.job']['id']
cplKey = event['CodePipeline.job']['data']['inputArtifacts'][0]['location']['s3Location']['objectKey']
cplBucket = event['CodePipeline.job']['data']['inputArtifacts'][0]['location']['s3Location']['bucketName']
s3.meta.client.download_file(cplBucket,cplKey,'/tmp/build.zip')
zip_ref = zipfile.ZipFile('/tmp/build.zip', 'r')
zip_ref.extractall('/tmp/')
zip_ref.close()
with open('/tmp/build.json') as json_data:
d = json.load(json_data)
s3.meta.client.download_file(d["template-bucket"], 'web-server-deployment.yml', '/tmp/web-server-deployment.yml')
s3.meta.client.download_file(d["template-bucket"], 'config', '/tmp/config')
print(d["repository-uri"], d["tag"], d["deployment-name"])
inplace_change("/tmp/web-server-deployment.yml", "$REPOSITORY_URI", d["repository-uri"])
inplace_change("/tmp/web-server-deployment.yml", "$TAG", d["tag"])
# Build config file from template and secrets in SSM
CA = ssm.get_parameter(Name='CA', WithDecryption=True)["Parameter"]["Value"]
CLIENT_CERT = ssm.get_parameter(Name='ClientCert', WithDecryption=True)["Parameter"]["Value"]
CLIENT_KEY = ssm.get_parameter(Name='ClientKey', WithDecryption=True)["Parameter"]["Value"]
inplace_change("/tmp/config", "$ENDPOINT", d["cluster-endpoint"])
inplace_change("/tmp/config", "$CA", CA)
inplace_change("/tmp/config", "$CLIENT_CERT", CLIENT_CERT)
inplace_change("/tmp/config", "$CLIENT_KEY", CLIENT_KEY)
config.load_kube_config('/tmp/config')
try:
with open(path.join(path.dirname(__file__), "/tmp/web-server-deployment.yml")) as f:
dep = yaml.load(f)
k8s_beta = client.ExtensionsV1beta1Api()
resp = k8s_beta.patch_namespaced_deployment(name=d["deployment-name"],
body=dep, namespace="default")
print("Deployment created. status='%s'" % str(resp.status))
code_pipeline.put_job_success_result(jobId=cplJobId)
return 'Success'
except Exception as e:
code_pipeline.put_job_failure_result(jobId=cplJobId, failureDetails={'message': 'Job Failed', 'type': 'JobFailed'})
print(e)
raise e
def inplace_change(filename, old_string, new_string):
with open(filename) as f:
s = f.read()
if old_string not in s:
# print '"{old_string}" not found in {filename}.'.format(**locals())
return
with open(filename, 'w') as f:
# print 'Changing "{old_string}" to "{new_string}" in {filename}'.format(**locals())
s = s.replace(old_string, new_string)
f.write(s)
config.load_kube_config('/tmp/config')
try:
with open(path.join(path.dirname(__file__), "/tmp/web-server-deployment.yml")) as
f:
dep = yaml.load(f)
k8s_beta = client.ExtensionsV1beta1Api()
resp = k8s_beta.patch_namespaced_deployment(name=d["deployment-name"],
body=dep, namespace="default")
print("Deployment created. status='%s'" % str(resp.status))
def inplace_change(filename, old_string, new_string):
with open(filename) as f:
s = f.read()
if old_string not in s:
# print '"{old_string}" not found in {filename}.'.format(**locals())
return
with open(filename, 'w') as f:
# print 'Changing "{old_string}" to "{new_string}" in
{filename}'.format(**locals())
s = s.replace(old_string, new_string)
f.write(s)

In Summary
• Deploy Frequently, and deploy small,
reversible changes.

In Summary
reversible changes.
• Kubernetes Deployments resources
support roll-out histories.

In Summary
reversible changes.
• Kubernetes Deployments resources
support roll-out histories.
• Use liveness and readiness probes to
support reliable deployments.

GET api/v1/namespaces/aws-summit/tips/{summary}
• Understand what your networking
requirements are.
• There are lots of options available
for deploying clusters.
• It’s hard to know where you’re going
without know where you’ve come from,
logging and monitoring is critical!
• Strive for Operational Excellence.

Some Stuff For You …
Kubernetes on AWS Workshop
https://github.com/aws-samples/kubernetes-aws-workshop
Networking with Amazon EKS
https://aws.amazon.com/blogs/opensource/networking-foundation-
eks-aws-cni-calico/

Alexa, deploy a Kubernetes cluster for me!

Thank you!
beaumonm@amazon.com

Expert Tips for Successful Kubernetes Deployment - AWS Summit Sydney 2018

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Expert Tips for Successful Kubernetes Deployment - AWS Summit Sydney 2018

Ähnlich wie Expert Tips for Successful Kubernetes Deployment - AWS Summit Sydney 2018 (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

Expert Tips for Successful Kubernetes Deployment - AWS Summit Sydney 2018