Just because Containerized, Kubernetized and Cloudified doesnt mean you application is production grade and disruption free. You will have to utilize all the features provided by Kubernetes to really make it produciton ready. In these slides, I will try to explain what could be possible disruptions that can happen in your Kubernetes cluster that can impact your application or workloads. And then will try to explain features or configuration of Kubernetes that will help in making your application production grade.

1. Chennai, India
755 02 01 268
Narayanan.kmu@gmail.com
bribebybytes.github.io/landing-page
NARAYANAN
KRISHNAMURTHY
Technical Architect, ADP India
Cloud Architect with 15 Years in IT
CLOUD DEVOPS TELECOM
Skills
Languages
English
Tamil
Hindi
Being Social
Bribe By Bytes
Hobby
1

3. Have some
Bad News &
Good News!

4. Just because you containerized, Kubernetized and Cloudified your
application, doesn’t mean its Reliable, Scalable and Secured automatically

5. Bad News First!
• Your Hardware will fail
• Your Enterprise grade application will fail
• Your Cloud will fail
• Your Kubernetes cluster will fail
Embrace it!

6. QoSLayer Application Layer Cluster Layer Infrastructure Layer
Reliable
Scalable
Available
Secured
Performance

7. INVOLUNTARY
DISRUPTIONS
VOLUNTARY
DISRUPTIONS

8. INVOLUNTARY
DISRUPTIONS
VOLUNTARY
DISRUPTIONS
Master
Worker 1 Worker 2
login Emp Emp

9. INVOLUNTARY DISRUPTIONS
Master
Worker 1 Worker 2
login Emp Emp
CLUSTER ADMIN DELETES A POD BY MISTAKE ☹

10. INVOLUNTARY DISRUPTIONS
Master
Worker 1 Worker 2
login Emp Emp
A HARDWARE FAILURE OF THE PHYSICAL MACHINE or VM

11. INVOLUNTARY DISRUPTIONS
Master
Worker 1 Worker 2
login Emp Emp
CLUSTER ADMIN DELETES A NODE BY MISTAKE ☹

12. INVOLUNTARY DISRUPTIONS
Master
Worker 1
New Pod
Worker 2
login Emp Emp
POD GETS EVICTED FROM NODE DUE TO RESOURCE CONSTRAINTS
New Pod New Pod
New Pod
New Pod New Pod New Pod

13. VOLUNTARY DISRUPTIONS
Master
Worker 1 Worker 2
login Emp Emp

14. Master
Worker 1 Worker 2
login Emp Emp
DRAINING A NODE FOR REPAIR OR UPGRADE OR TO SCALE DOWN
VOLUNTARY DISRUPTIONS

15. Master
Worker 1 Worker 2
login Emp
Emp
DRAINING A NODE FOR REPAIR OR UPGRADE OR TO SCALE DOWN
VOLUNTARY DISRUPTIONS

16. Master
Worker 1 Worker 2
login Emp
Emp
DRAINING A NODE FOR REPAIR OR UPGRADE OR TO SCALE DOWN
VOLUNTARY DISRUPTIONS
PENDING QUEUE!

17. Cluster admin deletes a pod
by mistake
A hardware failure of the
physical machine or Virtual
Machine
Cluster admin deletes a node
by mistake
Pod gets evicted from node
due to resource constraints
Draining a node for repair or
upgrade or to scale down
Application Upgrade

18. Good news -
Solution?

19. https://www.plectica.com/maps/I7WZTGITU/edit/RAKHSLAXT

20. Choose Right Controller/Storage Req
Pod Replicas
Application Upgrade Strategy
https://www.youtube.com/watch?v=c7ytxiddImw
spec:
replicas: 1
deployment.spec.replicas
deployment.spec.strategy statefulset.spec.updateStrategy
Recreate – deletes all
RollingUpdate – one pod upgrade at a time
OnDelete – only on Delete | Partition(canary)
RollingUpdate
daemonset.spec. updateStrategy
onDelete
RollingUpdate
https://www.youtube.com/watch?v=GQJP9QdHHs8
deployment daemonset statefulset job ephemeral persistent
https://www.youtube.com/watch?v=c7ytxiddImw
Pod eviction during
resource constraints
Node disk or mem
pressures

21. liveness
Liveness and Readiness Probes
readiness
pods/probe/exec-liveness.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
test: liveness
name: liveness-exec
spec:
containers:
- name: liveness
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
livenessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
pods/probe/tcp-liveness-readiness.yaml
apiVersion: v1
kind: Pod
metadata:
name: goproxy
labels:
app: goproxy
spec:
containers:
- name: goproxy
image: k8s.gcr.io/goproxy:0.1
ports:
- containerPort: 8080
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
https://www.youtube.com/watch?v=u7sbDPmezAo

22. node-selectors
Affinity and Anti Affinity
nodeAffinity podAffinity and podAntiAffinity
pods/pod-nginx.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
nodeSelector:
disktype: ssd
pods/pod-with-node-affinity.yaml
apiVersion: v1
kind: Pod
metadata:
name: with-node-affinity
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/e2e-az-name
operator: In
values:
- e2e-az1
- e2e-az2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: another-node-label-key
operator: In
values:
- another-node-label-value
containers:
- name: with-node-affinity
image: k8s.gcr.io/pause:2.0
pods/pod-with-pod-affinity.yaml
apiVersion: v1
kind: Pod
metadata:
name: with-pod-affinity
spec:
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: security
operator: In
values:
- S1
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: security
operator: In
values:
- S2
containers:
- name: with-pod-affinity
image: k8s.gcr.io/pause:2.0
Pod eviction during
resource constraints
Node disk or mem
pressures
Not all similar Pods
folks together

23. naive-dep-login
Affinity and Anti Affinity
self-relialized-dep-login
01-affinity-antiaffinity1-naive-dep-login.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: login-deployment
labels:
app: login
spec:
replicas: 1
selector:
matchLabels:
app: login
template:
metadata:
labels:
app: login
spec:
containers:
- name: login
image: "busybox:1"
command:
- sleep
- "7200"
01-affinity-antiaffinity2-self-relialized-dep-login.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: login-deployment
labels:
app: login
spec:
replicas: 1
selector:
matchLabels:
app: login
template:
metadata:
labels:
app: login
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: color
operator: In
values:
- blue
weight: 1
containers:
- name: login
image: "busybox:1"
command:
- sleep
- "7200"

24. taint
Taints and Tolerations
Node affinity, is a property of Pods that attracts them to a set of nodes (either as a preference or a
hard requirement). Taints are the opposite -- they allow a node to repel a set of pods.
toleration
kubectl taint nodes node1 key=value:NoSchedule pods/pod-with-toleration.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
tolerations:
- key: "example-key"
operator: "Exists"
effect: "NoSchedule"
One or more taints are applied to
a node; this marks that the node
should not accept any pods that
do not tolerate the taints.

25. Kube-api-server Service controllers
or kube-proxy
kubelet in Node Container Runtime
(e.g., Docker)
Containers
kubectl delete pod login-abcdf-
123adfc
Pod Set to
‘Terminating’
state
Pod removed
from Endpoints
pre-stop
hook trigged
SIGTERM
signal is sent
to each
container
kill <process>
pre-stop hook
executed
Pod no more
considered as
valid replica
SIGKILL signal
is sent to each
container
kill -9
<process>
Remove
Pod from
API
Server
Initiates
SIGTERM
Initiate
SIGKILL
Pods garbage
collected
30
secs
Grace
Period
Remove Pods
and Cleans-up
Deleting a Pod! - #ClaGIFied
Controllers will
start panicking

26. Kubectl Kube-api-server Service controllers
or kube-proxy
kubelet in Node Container Runtime
(e.g., Docker)
Containers
Kubectl drain
node1
Pod Set to
‘Terminating’
state
Pod removed
from Endpoints
pre-stop
hook trigged
SIGTERM
signal is sent
to each
container
kill <process>
pre-stop hook
executed
Pod no more
considered as
valid replica
SIGKILL signal
is sent to each
container
kill -9
<process>
Remove
Pod from
API
Server
Initiates
SIGTERM
Initiate
SIGKILL
Pods garbage
collected
30
secs
Grace
Period
Remove Pods
and Cleans-up
For Every Node –
Cordon it
For Every
POD
Cordon it –
Mark
Unschedulable
Is
PDB
met?
Retry
Draining
a Node!
-
#ClaGIFied

27. POD
DISRUPTION
BUDGET

28. POD DISRUPTION BUDGET
A PDB limits the number of pods of a replicated
application that are down simultaneously from
voluntary disruptions.

29. How
Does that
Work?

30. Your Deployment
Pod Disruption Budget
PDB
e001/pdb.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: loginBudget
spec:
minAvailable: 1
selector:
matchLabels:
app: login
e001/dep-login.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: login-deployment
labels:
app: login
spec:
replicas: 1
selector:
matchLabels:
app: login
.
.
.
.
.
.
Admin calls kubectl drain
Programmatically using
Eviction API
https://www.youtube.com/watch?v=pNbkZMEDevs

31. VOLUNTARY DISRUPTIONS
Disclaimer: Not all disruptions will be protected by
PDB
Some examples includes:
1.Deleting a deployment directly
2.Deleting a pod directly

32. How to determine
right value for my
PDB?

33. How to determine right value for my PDB?
• There is no single rule for this
Few Examples will be:
1. You are running a Consul cluster with K8S and you want to maintain a quorum of minimum 3 server components for fault
tolerance. In this case we can specify PDB’s minAvailable as 3.
2. You are running a statefulset for your database with K8S. And here you can specify PDB to avoid disruption in that DB, may
be you need respective team to take DB backups and then confirm that you can perform the disruption.
3. For stateless microservice, you might say I need minimum 1 replica running all the time and set PDB accordingly. Like we
saw in our demo sometime back.
4. And the list goes on.
So it means for every workload you are running in your cluster the setup of PDB can differ.
https://www.youtube.com/watch?v=pNbkZMEDevs

34. https://github.com/mikkeloscar/pdb-controller/
The controller simply gets all Pod Disruption Budgets for each namespace and
compares them to Deployments and StatefulSets. For any resource with
more than 1 replica and no matching Pod Disruption Budget, a default PDB
will be created
Cool tip on PDB controller

35. resources.requests(limits).cpu
Resource Constraints and PriorityClass
PriorityClass – Non-Namespaced object
containers:
- name: login
image: "busybox:1"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
command:
- sleep
- "7200"
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for
High Priority service pods only."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: low-priority
value: 5000
globalDefault: false
description: "This priority class should be used for
Low Priority service pods only."
resources.requests(limits).memory
spec:
priorityClassName: high-priority
Pod Spec with reference to PriorityClassName
Pod eviction during
resource constraints

36. failure-domain.beta.kubernetes.io/zone(< 1.17)
topology.kubernetes.io/zone (>= 1.17)
Topology Spread – Hosts/Zones/Regions
pods/pod-with-pod-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: topo-emp-deployment
labels:
app: emp
spec:
replicas: 2
selector:
matchLabels:
app: emp
template:
metadata:
labels:
app: emp
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: failure-
domain.beta.kubernetes.io/zone
containers:
- name: with-pod-affinity
image: k8s.gcr.io/pause:2.0
topologyKey: kubernetes.io/hostname
kubernetes.io/hostname
topologyKey: failure-domain.beta.kubernetes.io/region
failure-domain.beta.kubernetes.io/region(< 1.17)
topology.kubernetes.io/region (>= 1.17)
Not all similar Pods
folks together
Pods are HA during zonal
or Region Failure
https://cloud.google.com/compute/docs/regions-zones
https://docs.microsoft.com/en-us/azure/availability-zones/az-overview
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-
regions-availability-zones.html#concepts-availability-zones
AZ’s are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.

37. Cluster admin deletes a pod
by mistake
A hardware failure of the
physical machine or Virtual
Machine
Cluster admin deletes a node
by mistake
Pod gets evicted from node
due to resource constraints
Draining a node for repair or
upgrade or to scale down
Application Upgrade
Choose Right
Controller
Pod Replicas
Application
Upgrade Strategy
PDB
Affinity and Anti
Affinity/Taints and
Tolerations
Taints and
Tolerations
Topology Spread –
Hosts/Zones/
Regions
Resource
Constraints and
PriorityClass

38. QoSLayer Application Layer Cluster Layer Infrastructure Layer
Reliable
Scalable
Available
Secured
Performance

39. Chennai, India
755 02 01 268
Narayanan.kmu@gmail.com
bribebybytes.github.io/landing-page
NARAYANAN
KRISHNAMURTHY
Technical Architect, ADP India
Cloud Architect with 15 Years in IT
CLOUD DEVOPS TELECOM
Skills
Languages
English
Tamil
Hindi
Being Social
Bribe By Bytes
Hobby
39