Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Automating auto-scaled load balancer based on linux and vm orchestrator

512 Aufrufe

Veröffentlicht am

story about developing automatic scaleout loadbalancer using ECMP and Orchestrator for the container service

Veröffentlicht in: Ingenieurwesen
  • Girls for sex in your area are there: tinyurl.com/areahotsex
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Sex in your area is here: www.bit.ly/sexinarea
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Dating for everyone is here: www.bit.ly/2AJerkH
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Sex in your area for one night is there tinyurl.com/hotsexinarea Copy and paste link in your browser to visit a site)
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Girls for sex are waiting for you https://bit.ly/2TQ8UAY
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Automating auto-scaled load balancer based on linux and vm orchestrator

  1. 1. L.T.H. Automating auto-scaled load balancer based on linux and vm orchestrator Andrew Yongjoon Kong
  2. 2. Base network model for the VM infrastructure 32bit subnet, VRF, BGP and single NIC ( Introduced Tokyo 2016) kakao eth0 Compute node nova-compute neutron- linuxbridge- agent neutron-dhcp- agent Gateway 10.10.100.1 linux bridge vm IP:10.10.100.2/32 Routing Table 1 10.10.100.2/32 via 192.1.1.201 BGP 192.1.1.202 BGP Virtual Switch block Process block vlan.60 vlan.0 Virtual Router Service Route Table 1 192.1.1.202
  3. 3. What we solved? Compute node2 nova-compute neutron- linuxbridge-agent neutron-dhcp-agentlinux bridge Switch Namespace global name space Compute node1 nova-compute neutron- linuxbridge-agent neutron-dhcp-agentlinux bridge Switch Namespace global name space AZ1 Compute node2 nova-compute neutron- linuxbridge-agent neutron-dhcp-agentlinux bridge Switch Namespace global name space Compute node1 nova-compute neutron- linuxbridge-agent neutron-dhcp-agentlinux bridge Switch Namespace global name space AZ2 Compute node2 nova-compute neutron- linuxbridge-agent neutron-dhcp-agentlinux bridge Switch Namespace global name space Compute node1 nova-compute neutron- linuxbridge-agent neutron-dhcp-agentlinux bridge Switch Namespace global name space AZ3 tor1 tor2 tor3 vm10.10.100.2/32 Routing Table 1 10.100.10.2/32 via tor1 rt1 rt2 Routing Table 1 10.100.10.2/32 via RT1 rt3 rt4 rt5 rt6 Routing Table 1 10.100.10.2/32 via RT3 Routing Table 1 10.100.10.2/32 via tor2
  4. 4. What we solve? Simple IP planning – only IP ranges matter. (no more VLAN, IP subnet, Router planning) Resource imbalancing – No chance of IP imbalancing. Fault Resilience – If one router gone, it propagated by Dynamic routing protocol to other router Distributed – deciding routing path is very distributed. No single point of failure. – scale out nature.
  5. 5. 1563projects 632 pull request since 2014.9 88about VMs are created/deleted per day 8703 vms 2,xxxprojects 913 pull request since 2014.9 100about VMs are created/deleted per day 20,xxx vms 2016.8 2017.11 1xx,xxx active cores The Result is Stunning
  6. 6. The Dev to Prod Ratio
  7. 7. What is container? ¡ Container comprises multiple namespaces ¡ Standardized resource ¡ Brick or Lego of Process kakaoNetdevconf
  8. 8. Typical container orchestrator’s network ¡ Nothing different from anything, In terms of Networking l But if you don’t prepare or just listen to developers requirement, it’s getting different and difficult! l Now, you need to understand, what is the container kakaoNetdevconf
  9. 9. Scalable container network: Kakao’s case ¡ Have to deal with those when you try to use overlay. l Have to re-think about performance l Have to think about fault-resiliency, and migration issues. l Still consider how send the packet out of the system. kakaoNetdevconf
  10. 10. Scalable container network: Kakao’s case, cont. l Use node port and Load balancer ¡ It’s very easy. ¡ Had issue with scalability node port has limited port range. ¡ Only have 5digits number of containers ¡ Load balancer is expensive. kakaoNetdevconf
  11. 11. Scalable container network: Kakao’s case, cont. l Use routable container bridge subnet and bgp injector ¡ Predefine subnet for each containers bridge router ¡ Have to provision before resource depleted. Route Injector BGP Router Cluster Container Node1 subnet1 subnet2 subnet3 Container Node2 Container Node3
  12. 12. Application of /32bit network: /32bit route + DNAT à 1:1 NAT (A.K.A FloatingIP ) eth1 Compute node1 linux bridge vm IP:10.10.100.2/32 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 connected dest 192.168.100.2 Routing Table 1 10.10.100.2/32 via 192.1.1.201 2 10.10.100.3/32 via 192.168.1.202 3 192.168.100.2/32 via 192.168.1.201 192.1.1.202 Switch Namespace global name space IPTable DNAT Dest 192.168.100.2 is forwarded to 10. 10.100.2 Compute Node Router
  13. 13. Application of /32bit network: ECMP + DNAT à Scalable Loadbalancer eth1 Compute node1 linux bridge LB IP:10.10.100.2/32 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 connected dest 192.168.100.2 192.1.1.202 Switch Namespace global name space IPTable DNAT Dest 192.168.100.2 is forwarded to 10. 10.100.2 Compute Node Router eth1 Compute node2 linux bridge LB IP:10.10.100.3/3 2 192.1.1.202 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.3/32 to 10.10.100.1 connected dest 192.168.100.2 Switch Namespace global name space IPTable DNAT Dest 192.168.100.2 is forwarded to 10. 10.100.3 Compute Node Router TOR1 TOR2 Aggregation VIP: 192.168.100.2 is ECMPed
  14. 14. Application of /32bit network: Multiple Routing Entry ( AKA, Fixed IPs) + Container Bridge Network à Scalable Container Network eth1 Compute node1 linux bridge IP:10.10.100.2/32 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.3~33/32 to 10.10.100.1 Routing Table 1 10.10.100.3~33/32 via 192.168.1.201 192.1.1.202 Switch Namespace global name space Compute Node Router vm linux bridge Container Container Routable IP to Container: • Can use legacy IP base Monitoring • No Overlay ( No complexity )
  15. 15. Auto Scaleout Loadbalancer Design Requirements 1. Create/Delete Software Loadbalancer with API 2. The API should have compatibility with inhouse cloud (KRANE, openstack based) 3. Loadbalancers cluster log and metric can be gathered by inhouse measuring cloud. 4. Loadbalancers cluster should have same entering IP 5. Loadbalancers cluster should check each other’s state automatically
  16. 16. Auto Scaleout Loadbalancer Model I (1) 1. It’s starting from neutron floating IP 2. It’s mixed with BGP advertise option (connected), virtual IP and IPtables eth1 Compute node1 linux bridge vm IP:10.10.100.2/32 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 connected dest 192.168.100.2 Routing Table 1 10.10.100.2/32 via 192.1.1.201 2 10.10.100.3/32 via 192.168.1.202 3 192.168.100.2/32 via 192.168.1.201 192.1.1.202 Switch Namespace global name space IPTable DNAT Dest 192.168.100.2 is for warded to 10.10.100.2 Compute Node Router Veth pair Gateway 10.10.100.1 neutron- linuxbridge-agent neutron-dhcp-agent neutron-l3-agent Host Rout e dest 10.10.100.2/32 to 10.10.100.1 New IP 192.168.100.2 connected dest 192.168.100.2 Neutron Floating IP
  17. 17. Auto Scaleout Loadbalancer Model I (2) Basic scenario is " install haproxy to VM, attach floating IP, grouping it and double it” haproxy attach floating ip grouping double
  18. 18. Auto Scaleout Loadbalancer Model I (3) Basic scenario is " install haproxy to VM, attach floating IP, grouping it and double it” eth1 Compute node1 linux bridge vm IP:10.10.100.2/32 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 connected dest 192.168.100.2 Routing Table 1 10.10.100.2/32 via 192.1.1.201 2 10.10.100.3/32 via 192.168.1.202 3 192.168.100.2/32 via 192.168.1.201, 192.1 68.1.202 192.1.1.202 Switch Namespace global name space IPTable DNAT Dest 192.168.100.2 is for warded to 10.10.100.2 Compute Node Router Veth pair Gateway 10.10.100.1 neutron- linuxbridge-agent neutron-dhcp-agent neutron-l3-agent Host Rout e dest 10.10.100.2/32 to 10.10.100.1 New IP 192.168.100.2 connected dest 192.168.100.2 Neutron Floating IP eth1 Compute node1 linux bridge vm IP:10.10.100.3/32 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.3/32 to 10.10.100.1 connected dest 192.168.100.2 Switch Namespace global name space IPTable DNAT Dest 192.168.100.2 is for warded to 10.10.100.2 Compute Node Router Veth pair Gateway 10.10.100.1 neutron- linuxbridge-agent neutron-dhcp-agent neutron-l3-agent Host Rout e dest 10.10.100.3/32 to 10.10.100.1 New IP 192.168.100.2 connected dest 192.168.100.2 Neutron Floating IP 192.1.1.202
  19. 19. CBL alpha First thing to mention is that “Model is one thing, Implementation is the other thing”
  20. 20. CBL alpha tries First target was “Kubernetes” 1. It’s Golang based API server 2. Using ETCD for a database 3. Follow the model I (floating ip based ECMP) 4. Support scale in / out api 5. SNMP support for the haproxy status ( the metric can be pulled by central event system) 6. It’s working
  21. 21. CBL alpha First target was “Kubernetes” 1. Using Kubernetes event stream as triggering source 2. For approach from external, kubernetes have to have loadblancer for container 3. In alpha stage, k8s(from legacy container candidate) is dropped (OMG) etcd CBL (API server) kubernetes VM LB CBL cloudprovider POD POD Kubernetes nodes Openstack VM
  22. 22. CBL alpha Next target was “Integrating with Openstack Loadbalancer API” 1. Openstack has its own resource management cycle with loadbalancer 2. Now, CBL support pool management, vip management, member management 3. BTW, where is floating IP? – I used VIP ip address for the floating IP resource and attaching it to the VM – But from openstack perspective, floating IP also can be attached to VIP (What?) 4. What if just forget about Openstack? – Openstack is company’s legacy cloud. – I don’t want to recreate the new cloud
  23. 23. CBL alpha Except for the Floating IP issues, There are Other considerations 1. There’s no LB scale out API in openstack – I implemented this by scale-in/out when ’8.8.8.8’ backend members added and deleted – But you cannot add more than 2 backend with the same ip address in openstack – so scale in/out should be done in real automatic way 2. SSL off loading should be added. 3. LB cluster membership management should be added to CBL 4. I can’t do everything from server api to client sdk and module
  24. 24. CBL Beta Giving up the floating IP based ECMP, Multi NIC based ECMP 1. The good thing about the floating IP approach is that you don’t need to care about anything inside of the vm. But you can’t do this with openstack. 2. Changed Multi NIC based Design – There’s changes over Base model – have changes on haproxy VM in terms of network model
  25. 25. Auto Scaleout Loadbalancer Model II • Multi NIC for VM model doesn’t seem change much. but it radically changes everything eth1 Compute node1 linux bridge LB vm IP:10.10.100.2/32 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 dest 192.168.100.2 to 10.10.100.1 connected 192.1.1.202 Switch Namespace global name space IPTable DNAT empty Compute Node Router Veth pair Gateway 10.10.100.1 neutron- linuxbridge-agent neutron-dhcp-agent neutron-l3-agent Host Route dest 10.10.100.2/32 to 10.10.100.1 dest 192.168.100.2 to 10.10.100.1 connected empty Neutron Floating IP LB vm 10.10.100.2 dhclient deactivate vm agent eth0 eth1 VIP192.168.100.2
  26. 26. CBL beta – using vm orchestrator • CBL alpha is based on inhouse code • It maintains by One and Only • SDKs for Every Cloud driver • API and API engine/worker • UI • Using Inhouse vm orchestrator instead • Inhouse Cloud Now supports Openstack Heat
  27. 27. CBL beta – Based on VM orchestrator (openstack HEAT) • Heat provides Template (HOT & AWS CFN) • Heat provides Resource (as resource plug-in) like instance, volume, networking • Heat provides Stack , • 1 stack == several resources • Heat provides Auto-scaling • you don’t need to write code for auto-scaling • Heat provides Software-Deployment • you don’t need to write code for deployment • How to configure the cloud application? • Heat provides Events • you don’t need to write code for track the progress of life-cycle
  28. 28. CBL beta – Based on VM orchestrator (openstack HEAT) • Heat provides Template (HOT & AWS CFN) • Heat provides Resource (as resource plug-in) like instance, volume, networking • Heat provides Stack , • 1 stack == several resources • Heat provides Auto-scaling • you don’t need to write code for auto-scaling • Heat provides Software-Deployment • you don’t need to write code for deployment • How to configure the cloud application? • Heat provides Events • you don’t need to write code for track the progress of life-cycle
  29. 29. CBL beta – Based on VM orchestrator (openstack HEAT) II Templates Heat-API AMPQ Heat-Engine-1 DB Heat-Engine-2 Heat-Engine-3 Stack-1 Stack-2
  30. 30. CBL beta – Based on VM orchestrator (openstack HEAT) II heat_template_version: 2015-04-15 description: > parameters: resources: my_instance: …. my_scalingroup: …. outputs: ecmpip: Heat-API output: • LB IP • scale in url • scale out url
  31. 31. CBL beta – Based on VM orchestrator (openstack HEAT) III • With VM orchestrator, You can have the triggering end points. All that matter is How you trigger! • Kakao has inhouse metric information system called KEMI eth1 Compute node1 linux bridge LB vm IP:10.10.100.2/32 192.1.1.201 Routing Table Default GW 192.168.1.1 eth1 Host Route dest 10.10.100.2/32 to 10.10.100.1 dest 192.168.100.2 to 10.10.100.1 connected 192.1.1.202 Switch Namespace global name space IPTable DNAT empty Compute Node Router Veth pair Gateway 10.10.100.1 neutron- linuxbridge-agent neutron-dhcp-agent neutron-l3-agent Host Route dest 10.10.100.2/32 to 10.10.100.1 dest 192.168.100.2 to 10.10.100.1 connected empty Neutron Floating IP LB vm 10.10.100.2 dhclient deactivate vm agent eth0 eth1 VIP192.168.100.2
  32. 32. CBL beta – Based on VM orchestrator (openstack HEAT) & KEMI • From KEMI, We can setup the alarm condition and notification for each VM and serivce group heat_template_version: 2015-04-15 description: > parameters: resources: my_instance: …. my_scalingroup: …. outputs: ecmpip: Heat-API output: • LB IP • scale in url • scale out url KEMI: • alarm triggering
  33. 33. CBL beta – Based on VM orchestrator (openstack HEAT) & KEMI • From Performance Point of view Nothing Changed From CBL Alpha to Beta • But From Developing Cost, It has dramatic savings and progress • The CBL engine code is from 4404 line to 446 line (1/10) • After model changes, I used three day to change engine • IF you have a good Cloud, It’s very easy to make new network function
  34. 34. Q&A Thanks

×