Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Overview of kubernetes network functions

305 Aufrufe

Veröffentlicht am

In this slides, I briefly introduce the network function in the kubernetes and explain how kubernetes implement them.
Those function includes the container network interface (CNI) and kubernetes service.
In the last, I introduce the multus CNI which is designed for multiple networks in the container and it's necessary in some use case, such as SDN/NFV/5G

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Overview of kubernetes network functions

  1. 1. Kubernetes networkoverview
  2. 2. COSCUP2018 x openSUSE.Asia GNOME.Asia I am Hung-Wei Chiu Co-organizer of SDNDS-TW Co-organizer of CNTUUG I love Linux Network/Kubernetes/SDN You can find me at: blog.hwchiu.com
  3. 3. COSCUP2018 x openSUSE.Asia GNOME.Asia Outline What network functions about k8s How does above function implement What is the challenge about k8s
  4. 4. COSCUP2018 x openSUSE.Asia GNOME.Asia What network function kubernetes providers?
  5. 5. COSCUP2018 x openSUSE.Asia GNOME.Asia What Container Network ○ Connectivity ○ DNS Kubernetes services
  6. 6. COSCUP2018 x openSUSE.Asia GNOME.Asia Do You Know How Container Works?
  7. 7. COSCUP2018 x openSUSE.Asia GNOME.Asia Containersvs.VMs Containers are isolated, but share OS and where appropriate bins/libraries
  8. 8. COSCUP2018 x openSUSE.Asia GNOME.Asia HowDockerWorks We know docker is isolated, but how does it works? Linux kernel support the Namespaces mechanisms to partition kernel resources to different processes
  9. 9. COSCUP2018 x openSUSE.Asia GNOME.Asia HowDockerWorks Mount namespaces IPC namespaces PID namespaces Network namespaces User namespaces UTS namespaces ○ Unix Time System
  10. 10. COSCUP2018 x openSUSE.Asia GNOME.Asia NetworkNamespace Isolate the network functions. Including the ○ Network interfaces ○ Routing rules ○ Netfilter (iptables)
  11. 11. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host
  12. 12. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 Linux Host Linux Host
  13. 13. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 ns1 Linux Host Linux Host Linux Host
  14. 14. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 docker0 ns1 ns1 veth1 veth0 Linux Host Linux Host Linux Host Linux Host
  15. 15. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 docker0docker0 ns1 ns1ns1 vth1 vth0 veth eth0 Linux Host Linux Host Linux Host Linux HostLinux Host
  16. 16. COSCUP2018 x openSUSE.Asia GNOME.Asia Before we talk about service, we must know why service exist.
  17. 17. COSCUP2018 x openSUSE.Asia GNOME.Asia Pods/Deployments We can deploy our applications as a containers in the kubernetes. There’re many kind of the container we can deploy ○ Pod ○ Deployment ○ Statefulset ○ DaemonSet
  18. 18. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○ Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.58
  19. 19. COSCUP2018 x openSUSE.Asia GNOME.Asia Access How we application access those Nginx servers? By IP address ○ 10.123.234.56:80 ○ 10.123.234.57:80 ○ 10.123.234.58:80 What’s the problem
  20. 20. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○ Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.58
  21. 21. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○ Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.75
  22. 22. COSCUP2018 x openSUSE.Asia GNOME.Asia Access How we application access those Nginx servers? By IP address ○ 10.123.234.56:80 ○ 10.123.234.57:80 ○ 10.123.234.58:80 ○ 10.123.234.75:80 It’s not easy for our application to handle those ip-changed situation.
  23. 23. COSCUP2018 x openSUSE.Asia GNOME.Asia The Service is used to solve this problem.
  24. 24. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.56 10.123.234.57 10.123.234.58 App Service Nginx
  25. 25. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Application to Service ○ We use the DNS to access the service. ○ $(service).$(namespace).cluster.local Service to Pods ○ Service maintains all IP addresses of all Pods. ○ We call it endpoints
  26. 26. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.56 10.123.234.57 10.123.234.58 App Service Nginx nginx.default endpoints
  27. 27. COSCUP2018 x openSUSE.Asia GNOME.Asia How does kubernetes implements those functions?
  28. 28. COSCUP2018 x openSUSE.Asia GNOME.Asia What Container Network Connectivity ○ Container Network Interface(CNI) Kubernetes Services ○ There’re many implementation we can choose
  29. 29. COSCUP2018 x openSUSE.Asia GNOME.Asia ContainerNetworkInterface
  30. 30. COSCUP2018 x openSUSE.Asia GNOME.Asia ContainerNetworkInterface Cloud Native Computing Foundation Project Consists of a specification and libraries. Configure network interfaces in Linux containers Concerns itself only with network connectivity of containers ○ Create/Remove
  31. 31. COSCUP2018 x openSUSE.Asia GNOME.Asia ContainerNetworkInterface Removing allocated resources when the container is deleted
  32. 32. COSCUP2018 x openSUSE.Asia GNOME.Asia Who is using CNI?
  33. 33. COSCUP2018 x openSUSE.Asia GNOME.Asia FromtheGITHUB l rkt - container engine l Kubernetes - a system to simplify container operations l OpenShift - Kubernetes with additional enterprise features l Cloud Foundry - a platform for cloud applications l Apache Mesos - a distributed systems kernel l Amazon ECS - a highly scalable, high performance container management service
  34. 34. COSCUP2018 x openSUSE.Asia GNOME.Asia So, How to use the CNI?
  35. 35. COSCUP2018 x openSUSE.Asia GNOME.Asia StepbyStep Create a kubernetes cluster Setup your CNI plugin Deploy your first Pod
  36. 36. COSCUP2018 x openSUSE.Asia GNOME.Asia Just follow the installation to install the kubernetes
  37. 37. COSCUP2018 x openSUSE.Asia GNOME.Asia How do we install the CNI?
  38. 38. COSCUP2018 x openSUSE.Asia GNOME.Asia
  39. 39. COSCUP2018 x openSUSE.Asia GNOME.Asia Handbyhand In the kubelet, we have the following parameters for CNI. --cni-bin-dir ○ /opt/cni/bin --cni-conf-dir ○ /etc/cni/net.d/ We should config the CNI for every k8s nodes.
  40. 40. COSCUP2018 x openSUSE.Asia GNOME.Asia
  41. 41. COSCUP2018 x openSUSE.Asia GNOME.Asia Let Deploy a Pod
  42. 42. COSCUP2018 x openSUSE.Asia GNOME.Asia Beforewestart Pod ○ A collection of containers
  43. 43. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load the Pod config ○ Multiple containers Find a node to deploy the pod Create a Pause container
  44. 44. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container
  45. 45. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load the Pod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config
  46. 46. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Load the CNI config from /etc/cni/net.d/…
  47. 47. COSCUP2018 x openSUSE.Asia GNOME.Asia
  48. 48. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load the Pod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config Execute the CNI
  49. 49. COSCUP2018 x openSUSE.Asia GNOME.Asia Callthebinary Load the binary from the config Find the binary from the /opt/cni/bin/
  50. 50. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Call the /opt/cni/bin/flannel Network Connectivity
  51. 51. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load the Pod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config Execute the CNI Create target containers and attach to Pause container
  52. 52. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Call the /opt/cni/bin/flannel Network Connectivity Busybox
  53. 53. COSCUP2018 x openSUSE.Asia GNOME.Asia Attachtoothercontainer docker run –net=… ○ bridge ○ host ○ containerID
  54. 54. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Call the /opt/cni/bin/flannel Network Connectivity Busybox Pod
  55. 55. COSCUP2018 x openSUSE.Asia GNOME.Asia
  56. 56. COSCUP2018 x openSUSE.Asia GNOME.Asia Kubernetesservice There’re three implementation now. ○ User-space Kube-Proxy ○ Kernel-space iptables (default) ○ Kernel-space ipvs We use the iptables to explain how service(ClusterIP) works
  57. 57. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.56 10.123.234.57 10.123.234.58 App Service Nginx nginx.default endpoints
  58. 58. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get the Service ○ kubectl get service
  59. 59. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get the endpoints ○ kubectl get endpoints
  60. 60. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get the pod ip address ○ kubectl get pods –o wide
  61. 61. COSCUP2018 x openSUSE.Asia GNOME.Asia Now.Trytofetchthenginx Service name is k8s-nginx-cluster Use the nslookup to lookup the IP ○ nslookup k8s-nginx-cluster ○ nslookup k8s-nginx-cluster.default ■ default is the namespace of the service
  62. 62. COSCUP2018 x openSUSE.Asia GNOME.Asia
  63. 63. COSCUP2018 x openSUSE.Asia GNOME.Asia Trytofetchthenginx In the pod: curl k8s-nginx-cluster
  64. 64. COSCUP2018 x openSUSE.Asia GNOME.Asia Howitworks? First, we can get the VIP from the hostname. ○ It’s just a DNS request. Second, we can access the nginx from that VIP address. ○ iptables!!
  65. 65. COSCUP2018 x openSUSE.Asia GNOME.Asia Lte’swatchtheiptables First, we can use the service name to filter the iptables rules. sudo iptables-save | grep ”k8s- nginx-cluster”
  66. 66. COSCUP2018 x openSUSE.Asia GNOME.Asia Lte’swatchtheiptables Remember? There’re three endpoints for the service now.
  67. 67. COSCUP2018 x openSUSE.Asia GNOME.Asia workflowsPackets Packets Packets Match Services’ ClusterIP Find the endpoints DNAT KUBE-SERVICES KUBE-SVC-XXXX KUBE-SEP-XXXX Enter iptables PREROUTING Jump to other chain Jump to other chain Jump to other chain
  68. 68. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? When we match the clusterIP:Port, goto another custom chain. ○ 10.105.100.214:80
  69. 69. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? Use the random to choose what endpoint we use.
  70. 70. COSCUP2018 x openSUSE.Asia GNOME.Asia EP1 EP2 EP3 P < 0.33 P < 0.5 EP1 EP2 EP3 P= 1/3 P= 2/3 * 1/2 = 1/3 P= 2/3 * 1/2 = 1/3
  71. 71. COSCUP2018 x openSUSE.Asia GNOME.Asia EP1 EP2 EP3 P < 0.2 P < 0.25 EP1 EP2 EP3 EP4 EP5 P < 0.33 P < 0.5 EP4 EP5 P = 0.2 P = 4/5 * 1/4 = 1/5 P = 4/5 * 3/4 * 1/3 = 1/5 P = 4/5 * 3/4 * 2/3 * 1/2 = 1/5 P = 4/5 * 3/4 * 2/3 * 1/2 = 1/5
  72. 72. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? K8S create a custom chain for each endpoints. First rule is a SNAT ○ The Ngnix want to access outside. Second is DNAT ○ Change the IP to one of the endpoints
  73. 73. COSCUP2018 x openSUSE.Asia GNOME.Asia Now, We Know The Basic Function Of Kubernetes Network.
  74. 74. COSCUP2018 x openSUSE.Asia GNOME.Asia What is the next step of kubernetes network
  75. 75. COSCUP2018 x openSUSE.Asia GNOME.Asia challenges For different use cases ○ 5G/NFV/IoT Network features ○ High performance ○ Low latency Network infrastructure ○ Multiple network ■ Separate the data/control network.
  76. 76. COSCUP2018 x openSUSE.Asia GNOME.Asia Networkfeaturesc We use the hardware/smart NIC for those requirements before. We also have some mechanism in the software. ○ DPDK ○ SR-IOV ○ … ○ etc
  77. 77. COSCUP2018 x openSUSE.Asia GNOME.Asia Network Interface Card Linux Kernel Network Stack Network Driver Application Network Interface Card Linux Kernel Network Stack Network Driver Application Kernel Space User Space DPDK
  78. 78. COSCUP2018 x openSUSE.Asia GNOME.Asia How integrate those with kubernetes?
  79. 79. COSCUP2018 x openSUSE.Asia GNOME.Asia CNI We use some CNI for those functions. Intel had developed the CNI for those functions. ○ Call sriov-cni ○ https://github.com/intel/sriov-cni
  80. 80. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16)
  81. 81. COSCUP2018 x openSUSE.Asia GNOME.Asia Problem For those container using the DPDK/SR-IOV, it can’t use any kubernetes service now. Since the network function is handled by DPDK/SR-IOV now. How to solve this?
  82. 82. COSCUP2018 x openSUSE.Asia GNOME.Asia Multus There’s a discussion in the github about that requirement. Intel develop a CNI plugin to support multiple network for a Pod. ○ It’s called Multus CNI Multus call CNIs one by one.
  83. 83. COSCUP2018 x openSUSE.Asia GNOME.Asia Pod Container eth0 Pod Container eth0 eth1 eth2 Previous What Multus Provide
  84. 84. COSCUP2018 x openSUSE.Asia GNOME.Asia , , You need to create first
  85. 85. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
  86. 86. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
  87. 87. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
  88. 88. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
  89. 89. COSCUP2018 x openSUSE.Asia GNOME.Asia Q&A

×