Consolidating Infrastructure with Azure Kubernetes Service

Keine Notizen für die Folie

In this slide we are showing what items the traditional azure resources that make up Tailwind traders we have
* Postgres
* SQL
* Multiple VM's
* ACI (Azure container instances)
* Blob storage

Kubernetes is complex, running Kubernetes yourself is not the easiest of tasks without that skillset in your business.
In this slide we are showing the high level architecture of Kubernetes and all the moving parts, we want to call out the control plane (master node) and the worker nodes but we also want the audience to understand the flow from the API server to Kubelet to the worker nodes.

This slide the main points that we want to make is that, leading in off the last slide.
The Kubernetes control plane is managed by us as Azure, so the customer does not need to worry about that (so greater reduced complexity to allow your business to concentrate on your application not cluster administration)
The end user is responsible for the Kubernetes worker nodes (This means operational task like OS patching)

In this slide we want to tie together the architecture of Kubernetes with tailwind traders application.
We want to articulate how the traffic flows the cluster to the pod. Also tieing in the two slides before that link the control plane (master nodes) and the worker nodes.
For more information around Kubernetes ingress https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
This is the first slide in which we will introduce Calcio network policies https://azure.microsoft.com/en-au/blog/integrating-azure-cni-and-calico-a-technical-deep-dive/ which will act as a soft network switch and sets security boundaries

In this slide we need to show the audience how virtual node connects ACI to Kubernetes
Virtual node installs a go binary on the control plane that converts the communications between ACI and Kubernetes so Kubernetes sees ACI as a worker node in the cluster
Also for more reading on how the horizontal pod autoscaler works https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

ACI creates a container runtime on demand with the security of vm isolation

We also create a subnet on our vnet to talk to ACI so virtual node communications are private

In this slide we want to show the audience that when we set up availability zones in AKS the worker nodes will be split across Azure regions to make the cluster more highly available.
This is done using an Azure load balancer for more reading https://docs.microsoft.com/bs-latn-ba/azure/aks/availability-zones
This slide will give the audience context for the failure section

From this slide on we are going to walk through how we create an AKS cluster with the 4 added features we covered in the previous architecture slides CNI, Virtual node and Availability zones.
The first step, is to create an Azure resource group
We define the region with the –l flag
And name with the -n

In this slide we are adding a vnet to our resource group
Also creating a subnet for our cluster to use with the name myVnet

In this slide we are creating a second subnet that will be used for communication between Kubernetes and the ACI API in the region of the cluster
We are tying it to our resource group we created earlier, also to our vnet myVnet

In this slide we are creating a service principal for the cluster to use to create other resources on Azure it might need.
An example of this would be a public IP for ingress or schedule pods on ACI for virtual node

In this slide are showing how to create your first AKS cluster.
This meant to be an introduction to creating the cluster, by not adding any flags you will get the default networking stack and will not be able to add any of the features we need for the managements ask.
This leads into the next slide where are going to add flags to make our cluster fit the functionality that we have been request of via the management
Remember once a cluster is created you cant add network policy, or availability zones

In this slide we are going to contrast the pervious slide which is just getting a cluster up and running in 5 minutes. Here we are looking at a production use case with business needs.
In this command we are create the cluster like the last slide
* Add the cluster to our resource group
* Give the cluster a name
* Add 3 worker nodes, this will put one node in each availability zone
* Add the service principal and client secret so the cluster can provision other cloud resources
* Generate a set of ssh keys to access the work nodes
* We specify the network plugin, Azure cni this allows us to create network policies and is a pre requisite for that.
* We specify the DNS address the cluster will use for internal service discovery
* The docker bridge address is an ip range that has to be there for Docker to start as systemd will fail without it
* We add the vnet id of the subnet we created in the earlier steps
* We then add a load balancer, this is a pre requsite for avaliblity zones. It is used for the control plane to speak to the worker nodes
* We then enable vmms (virtual machine scaling sets https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/overview) this is also need for avalibility zones
* We then specify the availability zones that the worker nodes will go in. This ties into the node count placing one node in each zone
* Lastly we add the network policy plugin

Now we have our cluster created we are going to add the virtual node plugin, this will allow us to burst our workloads to ACI (https://docs.microsoft.com/en-gb/azure/container-instances/)
This will install a service on our cluster that will talk to Kubernetes and ACI and represent ACI as a node to Kubernetes.*

*First we enable the addon
* We then tie the addon to our resource group that our cluster
* We tie it to the cluster we just created by the clusters name
* The addon flag is virtual node as that is the addon we want to use
* We use the subnet that we created earlier to communicate with ACI on an internal connection

In this slide we are emulating a feature request coming in from management, Management want us to add a new consumer that takes customer data and colate all there orders to see if we can better server them when hitting the site.
We are going to add a new message que with RabbitMQ and then have our new service take the data from the que and do its work

Cluster autoscaler
The cluster autoscaler watches for pods that can't be scheduled on nodes because of resource constraints. The cluster then automatically increases the number of nodes.

1) CA watches for pods in a pending state
2) If found any, starts a scale-out of the cluster by adding additional nodes
3) CA watches for underutilized node
4) If found, and evicting pods are not in violation of other rules, a node will be drained and removed

Horizontal pod autoscaler
The horizontal pod autoscaler (HPA) uses the Metrics Server in a Kubernetes cluster to monitor the resource demand of pods. If a service needs more resources, the number of pods is automatically increased to meet the demand.

1) HPA obtains resource metrics and compares them to user-specified threshold
2) HPA evaluates whether user specified threshold is met or not
3) HPA increases/decreases the replicas based on the specified threshold
4) The Deployment controller adjusts the deployment based on increase/decrease in replicas

For tailwind traders we are going to use HPA with virtual node as we want the speed of processing and cluster autoscaler will not suit our needs. Also using virtual node we pay per second for the usage on ACI so we only pay when we use it.

In the diagram we list the pod primatives virtual kubelet exposes to Kubernetes via the ACI API
This allows the user to define there pods and execute the workloads on ACI with Kubernetes orchastrating for more information on virtual kubelet https://docs.microsoft.com/en-us/azure/aks/virtual-kubelet
For more information on pod and podspecs https://kubernetes.io/docs/concepts/workloads/pods/pod/

More reading
Here is an explanation of a kubelet

The kubelet is the primary “node agent” that runs on each node. The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn’t manage containers which were not created by Kubernetes.

Other than from a PodSpec from the apiserver, there are three ways that a container manifest can be provided to the Kubelet.

Virtual Kubelet is an open-source Kubernetes kubelet implementation that masquerades as a kubelet.

This allows Kubernetes nodes to be backed by Virtual Kubelet providers such as serverless cloud container platforms.

In this slide we look at the different types of pods that are supported.
So virtual nodes support most types of workloads

To make sure workloads are scheduled on virtual node we have to specify a node selector in our deployments.
Nodes selectors are used to define the node a workload will run on, by default no workload will be scheduled on virtual node without the above code added to your yaml at deployment time
Here is a full example https://github.com/scotty-c/kubecon-china/blob/master/demo/manifests/consumer/consumer.yaml#L26

This slide introduce the full suite of security features that we offer at Azure, this is a 30,000 foot overview and then will introduce us to the networking and pod indentity topic that we will go deeper into

Image and container level security
AAD authenticated Container registry access
ACR image scanning and content trust for image validation

Node and cluster level security
Automatic security patching nightly
Nodes deployed in private virtual network subnet w/o public addresses
Network policy to secure communication paths between namespaces (and nodes)
Pod Security Policies
K8s RBAC and AAD for authentication

Pod level security
Pod level control using AAD Pod Identity
Pod Security Context

Workload level security
Azure Role-based Access Control (RBAC) & security policy groups
Secure access to resources & services (e.g. Azure Key Vault) via Pod Identity
Storage Encryption
App Gateway with WAF to protect against threats and intrusions

When pods need access to other Azure services, such as Cosmos DB, Key Vault, or Blob Storage, the pod needs access credentials. These access credentials could be defined with the container image or injected as a Kubernetes secret, but need to be manually created and assigned. Often, the credentials are reused across pods, and aren't regularly rotated.

Managed identities for Azure resources (currently implemented as an associated AKS open source project) let you automatically request access to services through Azure AD. You don't manually define credentials for pods, instead they request an access token in real time, and can use it to access only their assigned services. In AKS, two components are deployed by the cluster operator to allow pods to use managed identities:

These are the two components that handle talking between AAD (Azure active directory) and Kubernetes to lease a MSI (Microsoft secure identity)

The steps to walk through the pod identity are as follows

1) Cluster operator first creates a service account that can be used to map identities when pods request access to services
2) The NMI server and MIC are deployed to relay any pod requests for access tokens to Azure AD.
3) A developer deploys a pod with a managed identity that requests an access token through the NMI server.
4) The token is returned to the pod and used to access an Azure SQL Server instance.

All pods in an AKS cluster can send and receive traffic without limitations, by default. To improve security, you can define rules that control the flow of traffic. Back-end applications are often only exposed to required front-end services, for example. Or, database components are only accessible to the application tiers that connect to them.
Network Policy is a Kubernetes specification that defines access policies for communication between Pods. Using Network Policies, you define an ordered set of rules to send and receive traffic and apply them to a collection of pods that match one or more label selectors.
By default the network in Kubernetes is flat, without a policy engine all pods can speak to one another

In this slide we introduce the two options that we offer in network security policy.
This slide leads into the next two slides that we will deep dive into the implementation details further
The main point here to call out between the two is Calico sets up Kernel routes and Azure network policy filters on the bridge. This detail will be important for customers that are highly regulated that go through audits.
Both implementations use Linux IPTables to enforce the specified policies. Policies are translated into sets of allowed and disallowed IP pairs. These pairs are then programmed as IPTable filter rules.

In this slide we are going to visualize the packet flow and how all the components hang together.
We have to logical sets of pods in this diagram one is labeled "test" the other "prod" both the policy engines work on labels to define the pod grouping.
Then we can see both groups of pods talk directly to the bridge, this is where the policy engine plugs into and enforces IPtables rules preventing test from talking to prod
You can then see the bridge connects to the vm-nic that is protected via the network security group on the Azure physical network

In this slide we are going to visualize the packet flow and how all the components hang together.
We have to logical sets of pods in this diagram one is labeled "test" the other "prod" both the policy engines work on labels to define the pod grouping this is the same as the last slide
Then we can see both groups of pods use the network layer in the kernel to apply iptables and layer 3 routing. So no packets leave the kernel and make there way to the bridge. Calico plugs into the kernel to apply the policies there.
You can then see the bridge connects to the vm-nic that is protected via the network security group on the Azure physical network

Availability Zones is a high-availability offering that protects your applications and data from datacenter failures. Zones are unique physical locations within an Azure region. Each zone is made up of one or more datacenters equipped with independent power, cooling, and networking. To ensure resiliency, there’s a minimum of three separate zones in all enabled regions.

Here are the main features that using avalibility zones gives use.

AKS clusters can currently be created using availability zones in the following regions:

East US 2
North Europe
Southeast Asia
West Europe
West US 2

Other limitations that need to be called out are
You can only enable availability zones when the cluster is created.
Availability zone settings can't be updated after the cluster is created. You also can't update an existing, non-availability zone cluster to use availability zones.
You can't disable availability zones for an AKS cluster once it has been created.
The node size (VM SKU) selected must be available across all availability zones.
Clusters with availability zones enabled require use of Azure Standard Load Balancers for distribution across zones.
You must use Kubernetes version 1.13.5 or greater in order to deploy Standard Load Balancers.

Here are the main features that using avalibility zones gives use.

The code example on this slide shows a deployment that has 3 replicas (highlighted in red) This is to show how we would handle failures if a pod, node or region died.

After this slide we will cut to the terminal and run show the availability zones the cluster has with kubectl describe nodes | grep -e "Name:" -e "failure-domain.beta.kubernetes.io/zone“
We will then turn a node off and check that the application is still available

Consolidating Infrastructure with Azure Kubernetes Service - MS Online Tech Forum

Davide Benvegnù

Consolidating Infrastructure with Azure Kubernetes Service - MS Online Tech Forum