Weitere ähnliche Inhalte Ähnlich wie K8s, Amazon EKS - 유재석, AWS 솔루션즈 아키텍트 (20) Mehr von Amazon Web Services Korea (20) Kürzlich hochgeladen (20) K8s, Amazon EKS - 유재석, AWS 솔루션즈 아키텍트1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jaeseok Yoo
K8s, Amazon EKS
2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Time
9:30 - 10:30 Docker & Container Orchestration, k8s
10:30 – 10:45 Beak
10:45 - 12:00 K8s, Amazon EKS
HoL: Launch EKS Cluster
12:00 – 13:00 Launch
13:00 – 13:40 HoL: Launch microservices
13:40 – 14:20 HoL: Helm
15:15 – 16:00 HoL: Monitoring with Prometheus and Grafana
3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Docker
4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
애플리케이션의 구성
런 타임 엔진 코드
디펜던시 구성
5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 다른 애플리케이션 스택
• 다른 하드웨어 배포 환경
• 다른 환경에서 애플리케이션을
실행하는 효율적인 방법은?
• 다른 환경으로 쉽게
마이그레이션하는 방법은?
문제점
6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
솔루션 - 도커
이식성 : 이미지 기반 배포
유연성 : 마이크로 서비스 모듈화
신속성 : 가벼운 도커 이미지
효율성 : OS kernel 공유
7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
VM과 컨테이너 비교
Server (Host)
Host OS
Hypervisor
App 2
Guest OS Guest OS Guest OS
Bins/Libs Bins/Libs Bins/Libs
App 1 App 3
VM
Server (Host)
Host OS
Docker
Bins/Libs Bins/Libs Bins/Libs
App 1 App 2 App 3
Container
Hypervisor
Guest OS
8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Docker 이미지 구성
bootfs
kernel
Base image
Image
Image
W
ritable
Container
add
nginx
add
nodejs
U
buntu
References
parent
image
Base Image : 템플릿으로 사용되는
읽기 전용 이미지
Base Image에서 시작해서 커스텀
Image 추가하는 방식
Dockerfile 활용하여 손쉽게 배포 관련
구성 설정 및 재배포에 용이함
9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Docker 엔진 구조 및 Docker CLI 예
• docker build # Build an image from a Dockerfile
• docker info # Display system-wide information
• docker images # List all images on a Docker host
• docker run # Run an image
• docker ps # List all running and stopped instances
• docker stop # Stop a running instances
• docker rm # Remove an instance
• docker rmi # Remove an image
• docker pull # Download an image from registry
• docker push # Upload an image to the registry
10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dockerfile은 이미지를 빌드하기 위한
일련의 명령어 모음
11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dockerfile
# our base image
FROM alpine:3.5
# Install python and pip
RUN apk add --update py2-pip
# install Python modules needed by the Python app
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r
/usr/src/app/requirements.txt
# copy files required for the app to run
COPY app.py /usr/src/app/
COPY templates/index.html /usr/src/app/templates/
# tell the port number the container should expose
EXPOSE 5000
# run the application
CMD ["python", "/usr/src/app/app.py"]
$ docker build -t <YOUR_USERNAME>/myfirstapp .
Sending build context to Docker daemon 9.728 kB
Step 1 : FROM alpine:latest
---> 0d81fc72e790
Step 2 : RUN apk add --update py-pip
---> 976a232ac4ad
Removing intermediate container 8abd4091b5f5
Step 3 : COPY requirements.txt /usr/src/app/
---> 65b4be05340c
Step 4 : RUN pip install --no-cache-dir -r
/usr/src/app/requirements.txt
---> 8de73b0730c2
Step 5 : COPY app.py /usr/src/app/
…
Dockerfile은 컨테이너 내부 이미지 환경 및 구성 정의
12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dockerfile best practice - 딱 필요한 Base 파일 선택
From the stock ubuntu image:
ubuntu latest 2b1dc137b502 52 seconds ago 458 MB
From python:2.7-alpine:
alpine latest d3145c9ba1fa 2 minutes ago 86.8 MB
13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FROM ubuntu:latest
RUN apt-get update -y && apt-get install -y python-pip python-dev build-essential
LABEL maintainer changsul@amazon.com
COPY . /app
WORKDIR /app
RUN pip install r requirements.txt
EXPOSE 5000
ENTRYPOINT ["python"]
CMD ["application.py"]
Dockerfile best practice - 딱 필요한 Base 파일 선택
14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FROM python:2.7-alpine
LABEL maintainer changsul@amazon.com
COPY . /app
WORKDIR /app
RUN pip install r requirements.txt
EXPOSE 5000
ENTRYPOINT ["python"]
CMD ["application.py"]
Dockerfile best practice - 딱 필요한 Base 파일 선택
15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FROM python:2.7-alpine
LABEL maintainer abbyfull@amazon.com
COPY requirements.txt /app
RUN pip install r /app/requirements.txt
COPY . /app
WORKDIR /app
EXPOSE 5000
ENTRYPOINT ["python"]
CMD ["application.py"]
Dockerfile best practice - 캐쉬 무효화 최소화
16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dockerfile best practice
빌드 이미지 크기 및 레이어 수 최소화
런타임시 필요한 것만 선택
각 빌드별 태깅
Semantic version (i.e. “1.3.2-9”)
Build Number (i.e., “127”)
Build Id (i.e. “511d5e51-b415-4cb2-b229-b3c8a46b7a2f”)
템프 파일 제거
RUN apt-get update && apt-get install -y
bzr
cvs
git
mercurial
subversion
&& rm rf /var/lib/apt/lists/*
17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
고객사례 - Nextdoor
Base OS version
Apt packages:
OpenSSL
libpq
syslog-ng
Datadog
Python runtime
PyPI packages:
Boto
Django
Mapnik
SendGrid
Source code
Static assets
Images
JS
CSS
18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Layer 별 각기 다른 업데이트 주기
Quarterly
Weekly/
monthly
Continuous
19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AMI에서 Docker Container로 변경
Base OS layer
System packages
Python packages
Nextdoor source
20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Docker 이전에는 빌드 20분 소요
chroot
sudo apt-get install
sudo pip install
git clone
make install
dpkg create
21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Base image , system deps 추가
FROM hub.corp.nextdoor.com/nextdoor/nd_base:precise
ADD app/docker/scripts/apt-fast
app/docker/scripts/system-deps.sh
/deps/
RUN /deps/system-deps.sh
22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Python virtualenv 설정 업데이트
ADD app/docker/scripts/venv-deps.sh
app/apps/nextdoor/etc/requirements*.txt
app/apps/nextdoor/etc/nextdoor.yml
app/services/scheduler/etc/scheduler.yml
app/services/supervisor/etc/supervisor.yml
/deps/
RUN /deps/venv-deps.sh
23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
App 소스 업데이트
ADD app/static/nextdoorv2/images /app/static/nextdoorv2/images
ADD app/thrift /deps/thrift
ADD app/nd /deps/nd
ADD app /app
24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
빌드 시간 20분 -> 평균 2분
ECS에 최종 배포까지 평균 5분
25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://docs.docker.com/
https://en.wikipedia.org/wiki/Docker_(software)
https://en.wikipedia.org/wiki/LXC
https://en.wikipedia.org/wiki/Linux_namespaces
https://en.wikipedia.org/wiki/Cgroups
https://en.wikipedia.org/wiki/Chroot
https://www.slideshare.net/Docker/creating-effective-images-abby-fuller-aws
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
https://github.com/docker/labs/blob/master/beginner/chapters/webapps.md
http://crosbymichael.com/dockerfile-best-practices.html
References
26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Common Questions
• How do I deploy my containers to hosts?
• How do I do zero downtime or blue green deployments?
• How do I keep my containers alive?
• How can my containers talk to each other?
• Linking? Service Discovery?
• How can I configure my containers at runtime?
• What about secrets?
• How do I best optimize my "pool of compute”?
27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How do we make this work at scale?
28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
We need to
• start, stop, and monitor lots of containers running on
lots of hosts
• decide when and where to start or stop containers
• control our hosts and monitor their status
• manage rollouts of new code (containers) to our hosts
• manage how traffic flows to containers and how
requests are routed
29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Orchestration
Instance Instance Instance
OS OS OS
Container Runtime Container Runtime Container Runtime
App Service App App Service Service
Container Orchestration
31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Orchestration
myJob: {
Cpu: 10
Mem: 256
}
Orchestrator
Schedule
Run “myJob”
32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Orchestration
Instance/OS Instance/OS Instance/OS
App Service App App Service Service
Service Management
Scheduling
Resource Management
OrchestrationService Management
§Availability
§Lifecycle
§Discovery
33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Orchestration
Instance/OS Instance/OS Instance/OS
App Service App App Service Service
Service Management
Scheduling
Resource Management
Orchestration
Scheduling
§Placement
§Scaling
§Upgrades
§Rollbacks
34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Orchestration
Instance/OS Instance/OS Instance/OS
App Service App App Service Service
Service Management
Scheduling
Resource Management
Orchestration
Resource Management
§ Memory
§ CPU
§ Ports
35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What are container orchestration tools?
36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Container Services Landscape
MANAGEMENT
Deployment, Scheduling,
Scaling & Management of
containerized applications
HOSTING
Where the containers run
Amazon Elastic
Container Service
Amazon Elastic
Container Service
for Kubernetes
Amazon EC2 AWS Fargate
IMAGE REGISTRY
Container Image
Repository
Amazon Elastic
Container Registry
37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Run a (managed) container on AWS
AMAZON CONTAINER SERVICES
Choose your orchestration tool1
Choose your launch type2
ECS EKS
EC2 Fargate EC2 Fargate
38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Kubernetes?
Open source container
management platform
Helps you run
containers at scale
Gives you primitives
for building
modern applications
40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes(K8s) Components
Control Plane (Controller)
Etcd Lightweight, open source Key-Value store containing the cluster
API Server Serves the APIs required to manage the cluster
Scheduler Determines where (on which nodes) pods will run in the cluster
Controller Manager
The “worker on the controller” that actually manages the cluster
(e.g. replication)
Kubernetes Node
kubelet Runs the node, starts and stops containers
kube-proxy
Acts as a network proxy – routes traffic based upon IP and Port.
Each service is assigned a unique port on the nodes it runs across,
kube-proxy allows that port to be mapped to whatever the service
expects.
cAdvisor Agent that monitors node health and statistics
41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes(K8s) Architecture
42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes(K8s) Objects
• kubectl
• Pods
• Labels
• Deployments
• Replication Controllers
• Services
43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
kubectl
• Command line interface for
running commands against the
k8s API
• Intuitive familiar commands
(get, create, describe, delete,
etc.) that are simple to learn and
easy to use
~/.kube/config
k8s master
kube-api
scheduler
44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pods
• A group of one or more
containers
• Shared:
• Data volumes
• cgroup
• Namespace – network, IPC, etc. node
pod1 pod2
45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Labels
• Key/Value Pairs
• Used to query specific resources
within your cluster
pod1
pod2
dev
prod
app001
app001
46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ReplicaSets
• Ensure that a specified number
of pod “replicas” exist in the
cluster
23
47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployments
• Declarative updates for Pods
and ReplicaSets
23
48. Containers on Hosts
Host 1
Host 2
Host 3
A host is a server – e.g. EC2 virtual machine.
We run these hosts together as a cluster.
Web App
To start let’s run a 3 copies of our web
app across our cluster of EC2 hosts.
3x
Our simple example web application is
already containerized.
Cluster
49. Run n containers
Host 1
Host 2
Host 3
We define a deployment and set the replicas
to 3 for our container.
deploymentkubectl
rep = 3
50. Scale up!
Host 1
Host 2
Host 3
Need more containers?
Update the replication set!
deploymentkubectl
rep = 5
The new containers are started on the cluster.
51. Untimely termination
Host 1
Host 2
Host 3
Oh no! Our host has died!
Replication
set
rep = 5
Kubernetes notices only 3 of the 5
containers are running and starts 2
additional containers on the remaining
hosts.
52. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services
• A Kubernetes Service is an abstraction which defines a logical set
of Pods and a policy by which to access them - sometimes called a
micro-service. The set of Pods targeted by a Service is (usually)
determined by a Label Selector.
• Let’s talk about what are the differences between LoadBalancer,
NodePort and Ingress
53. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services : ClusterIP
• Exposes the service on a cluster-
internal IP
• Only reachable from within the
cluster
• Access possible via kube-proxy
• Useful for debugging services,
connecting from your laptop or
displaying internal dashboards
54. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services : NodePort
• Exposes the service on each Node’s IP
at a static port.
• Routes to a ClusterIP service, which is
automatically created.
• from outside the cluster:
<NodeIP>:<NodePort>
• 1 service per port
• Uses ports 30000-32767
55. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services : LoadBalancer
• Exposes the service externally using a
cloud provider’s load balancer.
• NodePort and ClusterIP services (to
which LB will route) automatically
created.
• Each service exposed with a
LoadBalancer (ELB or NLB) will get its
own IP address
• Exposes L4 (TCP) or L7 (HTTP)
services
56. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service : LoadBalancer - Sample
apiVersion: v1
kind: Service
metadata:
name: my-nginx-lb
labels:
app: nginx-lb
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: nginx-lb
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx-lb
spec:
replicas: 3
template:
metadata:
labels:
app: nginx-lb
spec:
containers:
- name: nginx-lb
image: nginx:1.7.9
ports:
- containerPort: 80
57. Services
One of the ways traffic gets to your containers.
• Internal IP addresses are assigned to each container
• Services are connected to containers
and use labels to reference which containers
to route requests to
IP
IP
IP
Service
IP
60. Deployments – Rolling Update
IP
IP
IP
Service
IPReplication set
version = 1
count = 3
Only after the new pod returns a healthy
status to the service do we add more new
pods and scale down the old.
Replication set
version = 2
count = 1
IP
Deployment
Replication set
version = 1
count = 0
Replication set
version = 2
count = 3
62. Deployments – Canary
Service
app=nginx
Version=1
IP
Replication set
app=nginx
version=1
count=3
Deployment
Replication set
app=nginx
version=2
count=1
Deployment
Service
app=nginx
Replication set
app=nginx
version=2
count=2
Replication set
app=nginx
version=1
count=2
Replication set
app=nginx
version=1
count=1
Replication set
app=nginx
version=2
count=3
63. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services : Ingress
64. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services : Ingress
• exposes HTTP/HTTPS
routes to services within
the cluster
• Many implementations:
ALB, Nginx, F5, HAProxy
etc
• Default Service Type:
ClusterIP
65. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ALB Ingress Controller
AWS Resources
Kubernetes Cluster
Node Node
Kubernetes
API Server ALB Ingress
Controller
Node
HTTP ListenerHTTPS Listener
Rule: /cheesesRule: /charcuterie
TargetGroup:
Green (IP Mode)
TargetGroup:
Blue (Instance
Mode)
NodePort NodePort
66. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ConfigMap and Secret
67. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ConfigMap & Secret
• ConfigMap and Secret allow you to decouple
configuration artifacts from image content to keep
containerized applications portable.
• You can pass the ConfigMap or Secret to the pod by environment
variable or volume mount.
• Secret uses Base64 encoding.
68. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
StatefulSet
69. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Statefulset Properties
• Network identifiers
• Persistent Storage
• Ordered graceful deployment and scaling
• Ordered graceful termination
• Ordered rolling updates
• If none of these fit your portfolio, use Deployment or Replicaset
70. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
StatefulSet
1) Define headless
service, statefulset
and PVC
2) Control loop allocates
PV based on PVC
request
StorageClass
gp2 io1 sc1 encrypted
io1
st1
3) Kubernetes creates
statefulset
MySQL Pods
mysql-0 mysql-1 mysql-2 mysql-3
Network
Identifiers
Ordered
Deployment
71. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
StatefulSet
1) Define headless
service, statefulset
and PVC
2) Control loop allocates
PV based on PVC
request
3) Kubernetes creates
statefulset
MySQL Pods
mysql-0 mysql-1 mysql-2 mysql-3
Ordered
Scaling
mysql-4
72. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage
73. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lifecycle of a storage volume
Provisioning Binding Using Reclaiming
• Static
• Dynamic*
• Control loop watches
for PVC requests and
satisfies if PV is
available.
• For Dynamic, PVC
will provision PV
• PVC to PV binding is
one-to-one mapping
• Cluster mounts
volume based on
PVC
• Retain (default)
• Recycle
• Delete
74. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage Class Persistent Volume Persistent Volume
Claim
Pod
75. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What if I need specific volume type?
StorageClass
gp2 io1 sc1 encrypted
io1
st1
1) Admin pre-provisions
StorageClass based
on workload needs
2) End user requests for
specific volume types
(For ex, encrypted
io1 volume)
3) Control loop watches
PVC request and
allocates volume if
PV exists
MySQL Pods
4) End user creates
stateful workload
76. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
77. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
57%of Kubernetes workloads
run on AWS today
— Cloud Native Computing Foundation
79. Containers options on AWS – over time
Amazon ECS
EC2 Container
Instances
Auto Scaling group
2015
ECS API
Docker
Host
AWS Cloud
AWSmanagedCustomermanaged
80. Containers options on AWS – over time
AWS Fargate
Amazon ECS
EC2 Container
Instances
Auto Scaling group
2017
ECS API
Docker
Host
AWS Cloud
AWSmanagedCustomermanaged
81. Containers options on AWS – over time
AWS Fargate
Amazon ECS
EC2 Container
Instances
Auto Scaling group
Worker
nodes
Auto Scaling groupDIY K8S
ECS API
K8s API
Docker
Host
AWS Cloud
AWSmanagedCustomermanaged
82. Containers options on AWS – over time
AWS Fargate
Amazon ECSAmazon EKS
EC2 Container
Instances
Auto Scaling group
Worker
nodes
Auto Scaling groupDIY K8S
2018
K8s API ECS API
K8s API
Docker
Host
AWS Cloud
AWSmanagedCustomermanaged
85. Containers options on AWS – over time
AWS Fargate
Amazon ECSAmazon EKS
EC2 Container
Instances
Auto Scaling group
Managed
Node Groups
Auto Scaling group
Worker
nodes
Auto Scaling groupDIY K8S
2019
K8s API ECS API
K8s API
Docker
Host
AWS Cloud
AWSmanagedCustomermanaged
86. Containers options on AWS – over time
AWS Fargate
Amazon ECSAmazon EKS
EC2 Container
Instances
K8s API ECS API
AWS Cloud
Auto Scaling group
Managed
Node Groups
Auto Scaling group
Worker
nodes
Auto Scaling groupDIY K8S
NEW
Docker
Host
K8s API
AWSmanagedCustomermanaged
87. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes on AWS
Managed Kubernetes on
AWS
Highly
available
Automated
version
upgrades
Integration
with other
AWS services
Etcd
Master
Managed
Kubernetes
control
plane CloudTrail,
CloudWatch, ELB,
IAM, VPC, PrivateLink
88. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes on AWS
3x Kubernetes masters for HA
89. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Availability
Zone 1
Master Master
Availability
Zone 2
Availability
Zone 3
Master
Workers Workers Workers
Customer Account
AWS Managed
90. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Control Plane
91. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Architecture
EKS VPCCustomer VPC
Worker Nodes
EKS-Owned
ENI
Kubernetes
API calls
Exec, Logs,
Proxy
Internet
92. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Architecture
EKS VPCCustomer VPC
Worker Nodes
EKS-Owned
ENI
Kubernetes
API calls
Exec, Logs,
Proxy
Internet
93. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What happens when I run ‘kubectl create –f pods.yaml’?
94. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IAM Authentication
Kubectl
3) Authorizes AWS Identity with RBAC
K8s API
1) Passes AWS Identity
2) Verifies AWS Identity
4) K8s action
allowed/denied
AWS Auth
95. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes Control Plane
Master Node
Scheduler
Controller
Manager
Cloud
Controller
Manager
API Server
etcd
Kubectl
96. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cluster Authentication and Authorization
• User or IAM role who creates EKS cluster gains Admin privileges
• This {“super”} user/role can then add additional users or IAM roles
and configure RBAC permissions
• To add, configure aws-auth Configmap
kubectl edit -n kube-system configmap/aws-auth
97. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws-auth configuration
apiVersion: v1
data:
mapRoles: |
- rolearn: arn:aws:iam::555555555555:role/devel-worker-nodes-NodeInstanceRole-74RF4UBDUKL6
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
mapUsers: |
- userarn: arn:aws:iam::555555555555:user/admin
username: admin
groups:
- system:masters
- userarn: arn:aws:iam::555555555555:user/john
username: john
groups:
- pod-admin # k8s RBAC group
98. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Data Plane
99. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Architecture
EKS VPCCustomer VPC
Worker Nodes
EKS-Owned
ENI
Kubernetes
API calls
Exec, Logs,
Proxy
Internet
100. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes Data Plane
Worker Node
kube-dnsKubelet
aws-
node
Container runtime
Control Plane
API
kube-
proxy
101. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service
[Service]
ExecStartPre=/sbin/iptables -P FORWARD ACCEPT
ExecStart=/usr/bin/kubelet --cloud-provider aws
--config /etc/kubernetes/kubelet/kubelet-config.json
--allow-privileged=true
--kubeconfig /var/lib/kubelet/kubeconfig
--container-runtime docker
--network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS
Restart=on-failure
RestartForceExitStatus=SIGPIPE
RestartSec=5
KillMode=process
[Install]
WantedBy=multi-user.target
102. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS AMI Build Scripts
https://github.com/awslabs/amazon-eks-ami
Source of truth for EKS Optimized AMI
Easily build your own EKS AMI
Build assets for EKS AMI for each supported Kubernetes version
103. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Optimized AMI with GPU Support
Easily run Tensorflow/Kubeflow on Amazon EKS
Includes NVIDIA packages to support Amazon P2 and P3 instances
Available on AWS Marketplace
104. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Managed Node Group
• You can create, update, or terminate nodes for your cluster with a
single operation.
• Nodes run using the latest Amazon EKS-optimized AMIs in your
AWS account while node updates and terminations gracefully
drain nodes to ensure that your applications stay available.
• All managed nodes are provisioned as part of an Amazon EC2
Auto Scaling group that is managed for you by Amazon EKS.
105. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Worker Node Setup – Bootstrapping
/etc/eks/bootstrap.sh <cluster-name> [options]
Uses UserData for configuring System resources and extra Kubelet
config
Reserve compute resources for System Daemons (Kubelet, Container
runtime) and Pod eviction thresholds
106. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Upgrades
107. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes Version
Versions supported: 1.12.10, 1.13.12, 1.14.9
EKS will support up to 3 versions of Kubernetes at once
”Deprecation” will prevent new cluster creation on old version
108. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Platform Version
Platform Version revisions represent API server configuration
changes or Kubernetes patches
Platform Versions increment within a Kubernetes version only
109. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EKS Kubernetes Version Updates
New UpdateClusterVersion API –
supports in place updates of Kubernetes
version
Introduces an ”update” EKS API object
ListUpdates and DescribeUpdate APIs to
provide visibility into the status of a
given update
110. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Updating Worker Nodes
Two options:
1) Create new node group with latest EKS AMI >> taint old nodes >>
drain old nodes >> terminate old CFN template
2) Simply update AMI in CFN template; “rolling” replacement policy
terminates nodes
(Downsides: un-graceful termination of applications)
111. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Services Roadmap
https://github.com/aws/containers-roadmap
112. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Get Started
https://eksworkshop.com
Modules:
• Health Checks
• Logging with Elasticsearch, Fluentd, and
Kibana (EFK)
• Monitoring using Prometheus and Grafana
• Servicemesh with Istio
• Stateful Containers using StatefulSets
113. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!