Learning Objectives: - What is Amazon EKS - How Amazon EKS helps you run Kubernetes on AWS - Timelines for availability and next steps

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Nishi Davidson, Product Management, Amazon EKS
Dan Wilson, Principal Development Architect, SAP Concur
March 2018
Running Kubernetes on AWS
Managing workloads on-prem vs Amazon EKS

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
63%
of Kubernetes workloads run
on AWS today
— CNCF survey

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Cloud agnostic design: Common design in AWS and on-prem in your data center.
• Self-healing hence Stable: Applications once created, are kept alive even if they failure.
• Standardized hence Consistent: Inconsistencies isolated in the container while rest of the infrastructure tools
are standardized.
• Flexible hence Fast: Containerized applications can be deployed in seconds on any hardware.
• Integrated with monitoring/logging/alerting: Frees engineering to innovate on application logic.
• Scalable: Design scales easily to 10s of 1000s of machines.
Why k8s helps your environment?

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kubernetes @ SAP Concur
So how have some AWS customers
adopted Kubernetes in production?

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Polyglot – golang, javascript, java, .net, etc.
• Mostly private data centers, some AWS
• Global – US, Europe, Japan, China, etc.
• Public Sector, Enterprise & Consumer Apps
• Started w/ production clusters on k8s v1.1
• Need k8s multi-cluster day 1 for HA & blue/green cluster upgrades
k8s @ SAP Concur – on-prem and in AWS Cloud

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
K8s Cluster
Master
Mastermaster
CoreOS
Master
Masteretcd
CoreOS
worker1
CoreOS
worker2
CoreOS
workerN
CoreOS
flannel
newRelic & prometheus
Logstash forwarder
svc
ELK
Monitoring
Load Balancers
API
svc svc
svc
k8s @ SAP Concur – v1 multi-cluster

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8s @ SAP Concur – v1 multi-cluster
K8s 1P
LBaaS
API
Load Balancers
K8s 2 K8s 3
pool
VIP
P P
deployment
service
deployment
service
deployment
service

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
N clusters
CI
k8s @ SAP Concur

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Skipper
Kubernetes Deployment API
Curlable API to deploy microservice apps
to multiple k8s clusters
Open Source - Apache 2.0
https://github.com/concur/skipper

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
deployment API
CI
Deploy petshop:v1
to prod
US14a, b & c
EMEA14a, b & c
APAC14a, b & c
K8s API calls for service,
deployment, etc.

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
K8s 1
Ingress Controller
DNS
replicas sets & pods
Service
Ingress
internal
k8s @ SAP Concur – v2 clusters

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8s @ SAP Concur - auth
K8s 1
Clients/Skipper
Token per namespace and jurisdiction
K8s 2 K8s 3

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8s @ SAP Concur - oauth2 w/ dex
K8s 1
Clients/Skipper
Token per namespace and cluster
token 1 token 2 token 3
K8s 2 K8s 3

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8s @ SAP Concur – dns & routing
K8s 1Ingress Controller
DNS
replicas sets & pods
Service 1 & 2
Ingress
K8s 1
Ingress Controller
replicas sets & pods
Service 2 only
Ingress
?
Cluster 1 Cluster 2

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8s @ SAP Concur – dns & routing
K8s 1
Ingress Controller
DNS
replicas sets & pods
Service 1 & 2
Ingress
K8s 1
Ingress Controller
replicas sets & pods
Service 2 only
Ingress
CNAME service1.domain.com
Cluster1.domain.com
CNAME service2.domain.com
Cluster1-2.domain.com
Cluster 1 Cluster 2

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8s @ SAP Concur – dns & routing - redis
K8s 1
Ingress Controller
DNS
replicas sets & pods
redis1
Ingress
CNAME redis1.domain.com -> Cluster1.domain.com
CNAME redis2.domain.com -> Cluster2.domain.com
CNAME redis3.domain.com -> Cluster3.domain.com
Cluster 1
K8s 1
Ingress Controller
replicas sets & pods
redis2
Ingress
Cluster 2
K8s 1
Ingress Controller
replicas sets & pods
redis3
Ingress
Cluster 3

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8s @ SAP Concur – upgrades
Location US EUROPE
Zone A B C A B C
Cluster v.1.7 us17a us17b us17c eur17a eur17b eur17c

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8s @ SAP Concur – upgrades
Location US EUROPE
Zone A B C A B C
Cluster v.1.7 us17a us17b us17c eur17a eur17b eur17c
Cluster v.1.8 us18a us18b us18c eur18a eur18b eur18c

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
In place upgrades

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
k8S @ SAP Concur – monitoring and logging
• Add monitors for etcd and ingress
• Tune Prometheus
• monitor cluster capacity from the perspective of allocated resources instead of actual
resource usage
• configure alertmanager to route alerts appropriately
• tuning of some alert thresholds to get rid of the noise
• just added a monitor for error rate of kube2cnqrf5
• Many custom overrides on top of the 3rd party Prometheus operator

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How will AWS make k8s adoption
easier?
Intro to Amazon EKS

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo – basic flow
Subnet
EKS
Master
EKS Node
Group 2
EKS Node
Group 1
aws clikubectl
Pre-requisites:
• Configure VPC and
EKSServiceRole
• Configure kubectl and
awscli

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IAM ROLE User
X
IAM ROLE
Service Account Y
kubectl → K8s APIs → CRUD Operations on K8s
aws-cli → EKS Service APIs → CRUD Operations on Infra
K8s Master Nodes
K8s Master Nodes
K8s Master Nodes
API Server
Controller Mgr
kubelet
etcd
Cloud Controller Mgr.
Scheduler
Authentication
Webhook Tokens
Authorization
RBAC Mode
Admission Control
NamespaceLifecyle,LimitRanger
ServiceAccount,DefaultStorageClass,Reso
urceQuota
Amazon EKS – access and auth
AWS STS
client side
Heptio-aws-authenticator
server side

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
172.16.0.0/16
User X
Service
Account Y
Kubectl
K8s Node 2K8s Node 1
kubelet
kube-proxy
kubelet
kube-proxy
VPC Subnet per AZ
172.16.0.1/24
ENI ENIPrimary Private IP:
172.16.1.118
Secondary IPs:
172.16.1.147, 172.16.1.224….
Service: Front end
POD 2 POD 3
eth0
Service: Back end
POD 1 POD 4
eth0
ec2.associateaddress()
L3 RouteTable
veth0 Bveth0 A
eth0
172.16.1.147/32
eth0
172.16.1.224/32
CNI
K8s Master NodesK8s Master NodesK8s Master Nodes
API Server
Controller Manager
kubelet
etcd
Scheduler
kube-proxy
Cloud Controller Mgr.
Amazon EKS – networking with CNI plugin

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EKS – networking with CNI plugin
1. Simplify networking options for customers
2. Support high throughput, high availability, low latency and minimal jitter.
3. Allow customers to reuse AWS VPC networking and security best practices such as use of:
• VPC flow logs for troubleshooting and compliance auditing,
• VPC routing polices for traffic engineering,
• Security groups for isolation and regulatory requirements
4. Setup Pod networking within seconds.
5. Support cluster scale to a min. of 5000+ (K8s scale).

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S T A G E
S E P A R A T I O N
“ T E N A N T ”
S E P A R A T I O N
F I N E - G R A I N E D
F I R E W A L L S
C O M P L I A N C E
E.g., typically use namespaces
for different teams within
a company—but without
network policy, they are
not network isolated
Reduce attack surface within
microservice-based applications
Isolate dev, test, and prod E.g., PCI, HIPAA

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
172.16.0.0/16
User X
Service
Account Y
K8s Node 2K8s Node 1
kubelet
kube-proxy
kubelet
kube-proxy
VPC Subnet per AZ - 172.16.0.1/24
ENI ENI
Service: Front end
POD 2 POD 3
Service: Back end
POD 1 POD 4
CNI
K8s Master NodesK8s Master NodesK8s Master Nodes
API Server
Controller Manager
kubelet
etcd
Scheduler
kube-proxy
Cloud Controller Mgr.
Amazon EKS – dns, services and elb
DNS
kubedns
dnsmasq
healthz
DNS Service – Static IP
POD 2 POD 2
kind: Service
type: LoadBalancer

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EKS – monitoring and logging
• Cluster-wide Cloudwatch metrics and Cloudtrail logs for Control plane – EKS GA
• Cloudwatch metrics and Cloudtrail logs for Node groups – EKS Roadmap
Infra. metrics agents.– collectd,
cloudwatch
Logs Sink – ElasticSearch,
Cloudwatch, Cloudtrail
Log Text / Graphs –
Cloudwatch, Kibana
Metrics Sink – Prometheus,
InfluxDB, Graphite, Cloudwatch
Infra. log agents or exporters –
cloudwatch, fluentd, logstash
Metric Graphs – Grafana,
Cloudwatch
• Application container metrics / logs – Customer configured
K8sOptions

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EKS – cluster management
• Control Plane high-availability and scale
• Control Plane auto-upgrades with version testing
• In-built Cluster DNS and Service Discovery
• Cluster-wide metrics and logs for control plane
• AWS AMIs bundled with necessary tools
• Packer scripts to run K8s with distros of your choice on AWS

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Node Group
management
Master Auto-scale
Fargate for EKS
Add-Ons: UI, Helm,
Plugins, Brokers….
Amazon EKS – key roadmap and open source elements
Roadmap Items
Open Source @ AWS • Heptio aws-authenticator • CNI Plugin
https://github.com/heptiolabs/k
ubernetes-aws-authenticator
https://github.com/aws/amazon-vpc-cni-
k8s/blob/master/proposals/cni-
proposal.md
Privatelink
support

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you
d-nishi on github, dnishi@amazon.com k8s slack & gmail
danwilson on github, k8s slack & gmail, https://github.com/concur/skipper