DevOps Guide to Container Networking
•Als PPTX, PDF herunterladen•
3 gefällt mir•2,432 views
Programmable network connectivity and network overlay technologies like Docker libnetwork, Weave Net, and Calico are essential tools for DevOps engineers using orchestration tools to manage and deploy Docker containers in production. Because network troubleshooting and optimization falls within the jurisdiction of DevOps, it’s vital that DevOps engineers understand exactly how network overlays work. Participants will learn the fundamentals of container networking, see practical examples of common network overlays, and receive guidance on effectively using and tuning network overlays.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie DevOps Guide to Container Networking
Ähnlich wie DevOps Guide to Container Networking (20)
Mehr von Dirk Wallerstorfer
Mehr von Dirk Wallerstorfer (7)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
DevOps Guide to Container Networking
- 1. DevOps Guide to Container Networking Dirk Wallerstorfer DevOpsSummit New York, June 8th
- 2. 2 Technology Lead SDN, OpenStack dirk.wallerstorfer@dynatrace.com @wall_dirk blog.ruxit.com
- 13. 13Dirk Wallerstorfer, @wall_dirk web:$ docker run -itd wordpress
- 14. 14Dirk Wallerstorfer, @wall_dirk web:$ docker run -itd wordpress
- 15. 15Dirk Wallerstorfer, @wall_dirk web:$ docker run -itd wordpress user:wordpress$ ping 8.8.8.8 iptables –t nat –A POSTROUTING –o eth0 –j MASQUERADE
- 16. 16Dirk Wallerstorfer, @wall_dirk web:$ docker run –itd –p 8080:80 wordpress
- 17. 17Dirk Wallerstorfer, @wall_dirk web:$ docker run –itd –p 8080:80 wordpress iptables –t nat –A PREROUTING ... –j DOCKER iptables –t nat –A DOCKER --dport 8080 --redirect-to 172.18.0.2:80
- 19. Dirk Wallerstorfer, @wall_dirk 19 Three reasons for SDN • Permanent connectivity • Virtualization of everything • Paradigm shift in software development
- 20. Dirk Wallerstorfer, @wall_dirk 20 Three reasons for SDN Networking had to keep up somehow! Continuous delivery Virtualize everything Permanent connectivity
- 21. Dirk Wallerstorfer, @wall_dirk 21 SDN • Classic SDN • SD WAN • Network Overlay
- 22. Dirk Wallerstorfer, @wall_dirk 22 SDN • Classic SDN • SD WAN • Network Overlay
- 23. Dirk Wallerstorfer, @wall_dirk 23 SDN • Classic SDN • SD WAN • Network Overlay
- 24. Dirk Wallerstorfer, @wall_dirk 24 SDN • Classic SDN • SD WAN • Network Overlay
- 27. Dirk Wallerstorfer, @wall_dirk 27 Multi-host Container Networking No SDN db:$ docker run -itd –p 3306:3306 mysql web:$ docker run -itd –p 8080:80 –e WORDPRESS_DB_HOST=172.16.198.248:3306 wordpress
- 28. Dirk Wallerstorfer, @wall_dirk 28 Multi-host Container Networking Prerequisites • Underlying network • Distributed K/V store • Accessible ports
- 29. Dirk Wallerstorfer, @wall_dirk 29 Multi-host Container Networking Overlay No overlay http://s568.photobucket.com/user/LMG_09/media/CrowdSurfftw.jpg.html Ocean’s Eleven, Warner Bros, 2001
- 30. Dirk Wallerstorfer, @wall_dirk 30 Multi-host Container Networking Overlay Protocols • VXLAN Outer Ethernet Outer IP Outer UDP VXLAN Ethernet IP TCP Payload
- 31. Dirk Wallerstorfer, @wall_dirk 31 Multi-host Container Networking Overlay Protocols • VXLAN Outer Ethernet Outer IP Outer UDP VXLAN Ethernet IP TCP Payload Ethernet IP UDP VXLAN Ethernet IP TCP Payload
- 32. Dirk Wallerstorfer, @wall_dirk 32 Multi-host Container Networking Overlay Protocols • VXLAN Outer Ethernet Outer IP Outer UDP VXLAN Ethernet IP TCP Payload Ethernet IP UDP VXLAN Ethernet IP TCP Payload Flags Reserved VXLAN Network Identifier (VNI) Reserved
- 33. Dirk Wallerstorfer, @wall_dirk 33 Multi-host Container Networking Overlay Protocols • VXLAN Outer Ethernet Outer IP Outer UDP VXLAN Ethernet IP TCP Payload Ethernet IP UDP VXLAN Ethernet IP TCP Payload 14 bytes 20 bytes 8 bytes 8 bytes + 50 bytes
- 34. Dirk Wallerstorfer, @wall_dirk 34 Multi-host Container Networking Overlay Protocols • VXLAN • Ethernet in UDP, defacto standard, won the overlay war • NVGRE • Ethernet in IP, Microsoft’s answer to a question nobody asked • STT • Ethernet in fake TCP, to utilize TSO of NIC • Geneve • Ethernet in UDP, best of breed approach • A+ for extensibility • https://packetpushers.net/podcast/podcasts/pq-show-68-geneve-data-center-overlay-update/
- 35. Dirk Wallerstorfer, @wall_dirk 35 Multi-host Container Networking Overlay • Docker Libnetwork • WeaveNet • Flannel
- 36. Dirk Wallerstorfer, @wall_dirk 36 Docker libnetwork https://blog.docker.com/2015/04/docker-networking-takes-a-step-in-the-right-direction-2/
- 37. Dirk Wallerstorfer, @wall_dirk 37 Docker libnetwork
- 38. Dirk Wallerstorfer, @wall_dirk 38 Docker libnetwork
- 39. Dirk Wallerstorfer, @wall_dirk 39 Docker libnetwork
- 40. Dirk Wallerstorfer, @wall_dirk 40 Docker libnetwork
- 41. Dirk Wallerstorfer, @wall_dirk 41 Docker libnetwork
- 42. Dirk Wallerstorfer, @wall_dirk 42 Docker libnetwork
- 43. Dirk Wallerstorfer, @wall_dirk 43 Docker libnetwork
- 44. Dirk Wallerstorfer, @wall_dirk 44 Docker libnetwork
- 49. Dirk Wallerstorfer, @wall_dirk 49 Multi-host Container Networking No overlay • Project Calico • Flannel host-gw • Romana • Contiv • MACVLAN/IPVLAN
- 50. Dirk Wallerstorfer, @wall_dirk 50 Project Calico https://www.projectcalico.org/docker-libnetwork-is-almost-here-and-calico-is-ready/
- 53. Dirk Wallerstorfer, @wall_dirk 53 © http://de.slideshare.net/grkvlt/metaswitch-project-calico
- 54. Dirk Wallerstorfer, @wall_dirk 54 © http://de.slideshare.net/grkvlt/metaswitch-project-calico Host Host Containers Containers
- 55. Dirk Wallerstorfer, @wall_dirk 55 Project Calico • Host is a router for the workloads • BGP to distribute routes • etcd backed • Pure Layer 3, no encapsulation
- 56. Dirk Wallerstorfer, @wall_dirk 56 Project Calico
- 57. Dirk Wallerstorfer, @wall_dirk 57 Project Calico
- 58. Dirk Wallerstorfer, @wall_dirk 58 Project Calico
- 60. Dirk Wallerstorfer, @wall_dirk 60 Location of services k8s pods, marathon application groups, swarm constraints, fleet units
- 62. Dirk Wallerstorfer, @wall_dirk 62 Connectivity Problems nf_conntrack: table full, dropping packet. dirk@fueldev:~$ sudo sysctl –a | grep conntrack ... net.netfilter.nf_conntrack_buckets = 8192 net.netfilter.nf_conntrack_count = 0 net.netfilter.nf_conntrack_max = 31760 ... • Large number of iptables rules
- 63. Dirk Wallerstorfer, @wall_dirk 63 Connectivity Problems • The notorious MTU • https://www.youtube.com/watch?v=H2lBkj5zbYs dirk@fueldev:~$ ip addr show enp0s3 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:f3:4e:5d brd ff:ff:ff:ff:ff:ff inet 172.16.99.14 brd 172.16.11.255 scope global enp0s3 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:fef4:4e56/64 scope link valid_lft forever preferred_lft forever
- 64. Dirk Wallerstorfer, @wall_dirk 64 TCP/IP over VXLAN Overhead Ethernet IP UDP VXLAN Ethernet IP TCP Payload 14 bytes 20 bytes 8 bytes 8 bytes + 50 bytes Send 1MB of data 1,000,000 bytes = 710 packets á 1410 bytes 710 x 50 bytes = 35,500 bytes overhead 1,035,500 bytes are transmitted 3.55 %
- 65. Dirk Wallerstorfer, @wall_dirk 65 Send 1MB of data 1,000,000 bytes = 736 packets á 1330 bytes 736 x 100 bytes = 73,600 bytes overhead 1,073,600 bytes are transmitted TCP/IP over VXLAN over VXLAN Overhead Ethernet IP UDP VXLAN Ethernet IP UDP VXLAN Ethernet IP TCP Payload 14 bytes 20 bytes 8 bytes 8 bytes 14 bytes 20 bytes 8 bytes 8 bytes + 100 bytes 7.36 %
- 69. Dirk Wallerstorfer, @wall_dirk 69 1460 1410 1360 1310 1260 1210 1160 MTU overhead 25,9% 20,7% 15,9% 11,5% 7,4% 3,6% 0%
- 70. 70Dirk Wallerstorfer, @wall_dirk February 20, 2016 http://machinezone.github.io/research/networking-solutions-for-kubernetes/ Performance Comparison of Networking Solutions for Kubernetes
- 71. 71Dirk Wallerstorfer, @wall_dirk Performance Comparison of Networking Solutions for Kubernetes February 20, 2016 http://machinezone.github.io/research/networking-solutions-for-kubernetes/
- 72. 72Dirk Wallerstorfer, @wall_dirk Performance Comparison of Networking Solutions for Kubernetes February 20, 2016 http://machinezone.github.io/research/networking-solutions-for-kubernetes/
- 73. 73Dirk Wallerstorfer, @wall_dirk https://github.com/machinezone/tcpkali serving 350 byte responsemaking 250,000 requests per second Performance Comparison of Networking Solutions for Kubernetes Different network options - latency?
- 74. 74Dirk Wallerstorfer, @wall_dirk 250,000 requests per second, 350 bytes response February 20, 2016 http://machinezone.github.io/research/networking-solutions-for-kubernetes/
- 75. 75Dirk Wallerstorfer, @wall_dirk > 3 secresponse time will leave the page
- 76. 76Dirk Wallerstorfer, @wall_dirk +0.5 sresponse time in revenue
- 77. keep it manageable keep it simple keep it fast
- 78. 78Dirk Wallerstorfer, @wall_dirk http://i.coastingfish.com/image/3M Volume-oriented network metrics Quality-oriented network metrics
- 83. 83 Technology Lead SDN, OpenStack dirk.wallerstorfer@dynatrace.com @wall_dirk blog.ruxit.com Image sources: pixabay.com (3, 4, 5, 7, 9, 10, 23, 41, 57, 59, 60, 61)
Hinweis der Redaktion
- to get a feeling how much experience you guys have, I want to start with a joke – it’s also a good way to start the day, either you will laugh because of the joke or because of me trying to tell the joke – either way, laughing is healthy no matter what, so here we go
- containers, history, usage
- container networking, different options, network overlays, flat networks, we’ll take a look at options and compare the most common options
- software defined everything! int his case it’s about software defined networking and why you should care
- Guidance, pitfalls, and tuning what is the takeaway? you will use these technologies, and there are some problem that you will run into if you get started, I want to tell you upfront what these problems are and how you can prevent them there is not that much experience with these technologies today, but we have already seen some pitfalls that I would want to make you aware of to that end, I want to explain to you how container networking works in detail, what kind of options there are for container networking, about the compatibility of the networking options to common orchestration tools you would definitely need if you did want to run containers in production, and
- container networking, different options, network overlays, flat networks, we’ll take a look at options and compare the most common options
- software defined everything! int his case it’s about software defined networking and why you should care
- today there are proprietary protocols, hundreds of network components that need to be configured correctly, there are solutions but vendor lock-in network operators today have to “coax” their network equipment and its embedded control plane to meet their networking objectives worst case scenario: exaggeration: new loadbalancer, webservers, database cluster -> network configuration must be done by hand -> this will take forever classical SDN, ideas have been around for many years, promote concepts that can help make network administration easier again, e.g. centralized vs. decentralized control plane, global network-wide view of network state intents can be communicated through the northbound API of the controller that is independent of the physical network components; no intent: “create a new OSPF route on switch 16 and update the access control list on switches 3 to 5 and routers 1, 4, and 7”; intent: “connect server A and B, make sure that there is at least 20 MBit/s bandwidth available and the best possible transport quality” direct control to the underlying network components through the southbound API of the controller (e.g. OpenFlow), that configures forwarding and routing tables in commodity hardware
- today there are proprietary protocols, hundreds of network components that need to be configured correctly, there are solutions but vendor lock-in network operators today have to “coax” their network equipment and its embedded control plane to meet their networking objectives worst case scenario: exaggeration: new loadbalancer, webservers, database cluster -> network configuration must be done by hand -> this will take forever classical SDN, ideas have been around for many years, promote concepts that can help make network administration easier again, e.g. centralized vs. decentralized control plane, global network-wide view of network state intents can be communicated through the northbound API of the controller that is independent of the physical network components; no intent: “create a new OSPF route on switch 16 and update the access control list on switches 3 to 5 and routers 1, 4, and 7”; intent: “connect server A and B, make sure that there is at least 20 MBit/s bandwidth available and the best possible transport quality” direct control to the underlying network components through the southbound API of the controller (e.g. OpenFlow), that configures forwarding and routing tables in commodity hardware
- not for large enterprises that run their own networks over several locations, for mid-size enterprise that for example use MPLS VPN services of their ISP but don’t want to route all the traffic over this connection since it’s quite expensive in terms of dollar/bit, alternative connection like the public Internet is often good enough
- operates on existing underlying network, regardless of how the underlying network is configured little invest necessary, if you already have a working network in place NOTHING NEW: IPSec, MPLS, GRE, SSL VPN, ... overlay and encapsulation have been around for many years
- although this works, you are limited to IP addresses of hosts again – so there is virtually no benefit service resiliency
- etcd, zookeeper, consul: to keep network overlay information in sync and share with all other overlay participants some ports must be accessible for these tools to exchange information, ports vary
- look in detail how these two networking options work, and take one example
- VXLAN is the defacto standard, everybody knows it, has been around for X years now, hardware support from several vendors – it’s the technology to go for, if you don’t know what to choose – for now VMware NSX also relies on VXLAN, a lot of networking that is going on in the OpenStack area relies on VXLAN Open vSwitch relies on VXLAN STT – stateless tunneling protocol, fake TCP header to utilize TCP segmentation offloading of NIC and not all computing has to be done by the CPU NVGRE – alternative to VXLAN, works quite similar, works on IP level Geneve – the new’er’ kid on the block, already supported by OVN and OpenV Switch,
- VXLAN is the defacto standard, everybody knows it, has been around for X years now, hardware support from several vendors – it’s the technology to go for, if you don’t know what to choose – for now VMware NSX also relies on VXLAN, a lot of networking that is going on in the OpenStack area relies on VXLAN Open vSwitch relies on VXLAN STT – stateless tunneling protocol, fake TCP header to utilize TCP segmentation offloading of NIC and not all computing has to be done by the CPU NVGRE – alternative to VXLAN, works quite similar, works on IP level Geneve – the new’er’ kid on the block, already supported by OVN and OpenV Switch,
- VXLAN is the defacto standard, everybody knows it, has been around for X years now, hardware support from several vendors – it’s the technology to go for, if you don’t know what to choose – for now VMware NSX also relies on VXLAN, a lot of networking that is going on in the OpenStack area relies on VXLAN Open vSwitch relies on VXLAN STT – stateless tunneling protocol, fake TCP header to utilize TCP segmentation offloading of NIC and not all computing has to be done by the CPU NVGRE – alternative to VXLAN, works quite similar, works on IP level Geneve – the new’er’ kid on the block, already supported by OVN and OpenV Switch,
- VXLAN is the defacto standard, everybody knows it, has been around for X years now, hardware support from several vendors – it’s the technology to go for, if you don’t know what to choose – for now VMware NSX also relies on VXLAN, a lot of networking that is going on in the OpenStack area relies on VXLAN Open vSwitch relies on VXLAN STT – stateless tunneling protocol, fake TCP header to utilize TCP segmentation offloading of NIC and not all computing has to be done by the CPU NVGRE – alternative to VXLAN, works quite similar, works on IP level Geneve – the new’er’ kid on the block, already supported by OVN and OpenV Switch,
- VXLAN is the defacto standard, everybody knows it, has been around for X years now, hardware support from several vendors – it’s the technology to go for, if you don’t know what to choose – for now VMware NSX also relies on VXLAN, a lot of networking that is going on in the OpenStack area relies on VXLAN Open vSwitch relies on VXLAN STT – stateless tunneling protocol, fake TCP header to utilize TCP segmentation offloading of NIC and not all computing has to be done by the CPU NVGRE – alternative to VXLAN, works quite similar, works on IP level Geneve – the new’er’ kid on the block, already supported by OVN and OpenV Switch,
- explain the joke at this slide! if the encapsulation protocol would also use TCP, then the self healing powers of TCP would be redundant! the TCP stack of the container takes care of making sure that the connection is intact – if a UDP datagram is dropped along the way, TCP takes care of requesting this segment of data again – so stateless transport at this level is sufficient
- explain the joke at this slide! if the encapsulation protocol would also use TCP, then the self healing powers of TCP would be redundant! the TCP stack of the container takes care of making sure that the connection is intact – if a UDP datagram is dropped along the way, TCP takes care of requesting this segment of data again – so stateless transport at this level is sufficient
- macvlan, ipvlan ONS2016 overlay adds a layer of abstraction = adds a layer of complexity VXLAN good for development environment greenfield environment that don’t have legacy stuff IPVLAN future of large scale networking! ridiculously fast, near wire no ARP, no broadcast turn primary NIC on a linux host into a router
- how I imagine the idea for Calico was found
- how I imagine the idea for Calico was found
- Apache Felix for route configurations
- In reality it is like with every other network ... you’ll run into similar troubles, e.g. connectivity problems, while you can’t
- Location of services that talk to each other: k8s pods, marathon application groups, docker swarm constraints watch out that services that talk to each other a lot a colocated to a certain extent, e.g. k8s pods, marathon application groups, swarm constraints, fleet units, ... or manually; make sure that the traffic between two services doesn’t go through every switch in your datacenter before arriving at your target
- workload runs in container with any of the previously mentioned container networking options and you notice packet loss and connectivity problems of the offered services check log files iptables is a stateful firewall and TCP is a stateful protocol, the conntrack table tracks all active connections that are processed concurrently on one host. if there are several workloads with high network activity, the connection table can get full and you cannot statefully track new connections anymore until older connections are closed
- Commonly, the largest IP packet that can fit into Ethernet frame MTU: encapsulation transfer and CPU overhead, measurements from packetpushers, ... 3-6% overhead with encapsulation! encapsulation protocols have impact on MTU; consider scenario with multiple encapsulations ... what about data connections between two locally distributed sites that are connected with a IPSEC/SSL-VPN tunnel? VXLAN vs STT vs Geneve: TCP checksum offloading, TCP segmentation offload, ready for encryption, room for custom data
- you can destroy your network performance by making administration easier ... the cost/tradeoff should be clear to everyone up front!
- one major US retailer correlates 0.5 second site slowdown to 11% reduction in conversions!
- one major US retailer correlates 0.5 second site slowdown to 11% reduction in conversions! the choice of your networking driver for your multi-host container communication actually does matter – in fact, it matters a lot!
- solving a problem with complexity almost never turns out great, keep it as simple as possible while respecting your needs and requirements make sure you have an overview of what’s going on and don’t jump to conclusion too quickly follow twitter and see which orchestrators and networking vendor are the most active and have the most integrations, these are also the ones that will help you when you’re stuck and these will be the ones that are going to be around for the longest time
- volume-oriented and quality-oriented metrics
- volume-oriented and quality-oriented metrics
- short summary about what we’ve talked – let’s see if there are any questions