The document discusses load balancing for a large GitHub Enterprise cluster with zero downtime requirements. It describes using OpenResty (Nginx + Lua) as a scripting platform to conduct traffic via code and enable features like customized request rate limiting. It also discusses using a blue/green deployment approach with two load balancer stacks and switching DNS records to route traffic between stacks for experiments. Key aspects of the infrastructure include testing Docker images and hosts with RSpec and Serverspec, using Vault for secret management, and blocking mode protection with Signal Sciences.
2. For What?
• GitHub Enterprise Cluster
• On Internet
• Zero downtime
• 100M+ HTTP requests per week
• 30k+ attacks per week
• 26k+ git clone per hour
(https://help.github.com/enterprise/2.8/admin/guides/installation/maintenance-mode/)
3.
4. Design Goals
• Scripting Platform
• traffic conducting via code
• do social coding
• Observable
• Blue/Green deployment
• High performance
• Security from day one
7. Blue/Green Deployment
• Can not terminate any TCP connection
• Two stacks:
• load-balancer-green
• load-balancer-blue (for experiment)
• Cloud DNS
• Switching A record + short TTL (~5m)
• Simple/Weighted Routing policy
• Run experiment by using docker image tags
• Real time metrics collection by librato.com
8. • Test docker images
• RSpec + Serverspec
• Travis CI
• Test docker host
• RSpec + Serverspec
• Test Kitchen
Test Driven for Container
9. ❤vault
• Secret mgmt via API - https://www.vaultproject.io/
• retrieve all secrets for provisioning load balancer
via a single token with TTL 5min
10. Blocking mode in Production
• Signal Sciences - https://signalsciences.net/
11. Summary
• Conducting HTTPS traffic via Lua code
• Blue-green deployment of Load balancer via DNS
• Testing docker with RSpec + Serverspec
• SignalSciences