Want to learn how Facebook scales their load balancing infrastructure to support more than 1.3 billion users? We will be revealing the technologies and methods we use to global route and balance Facebook's traffic. The Traffic team at Facebook has built several systems for managing and balancing our site traffic, including both a DNS load balancer and a software load balancer capable of handling several protocols. This talk will focus on these technologies and how they have helped improve user performance, manage capacity, and increase reliability.

2. Production Engineer, Traffic Team
Building a Billion User Load
Balancer
Patrick Shuff

4. We’ll be talking about
© 2015 Facebook | Dublin | Credits: network icon by Daniel Gamage https://thenounproject.com/term/network/49138/, tools icon by Kiryl Sytsko https://thenounproject.com/term/tools/122204/, chess icon by Dream Icons, https://thenounproject.com/term/chess/127867/, Paraglider by Matt Brooks https://thenounproject.com/term/paraglider/114410/, Question by Henrik Lund Mikkelsen
https://thenounproject.com/term/question/8325/ https://creativecommons.org/licenses/by/3.0/ https://thenounproject.com/term/web-search/18390/
Serving Dynamic
Facebook Requests
L4/L7 Load
Balancing
Edge PoP and
Reducing Latency
Global DNS
Load Balancing
Q&A

5. Traffic @ fb

6. Facebook scale
as of March 2015
936 million daily active
people on average
798 million mobile daily
active people on average
1.25 billion mobile
monthly active people
1.44 billion monthly
active people
© 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain

7. Facebook scale
as of March 2015
936 million daily active
people on average
798 million mobile daily
active people on average
1.25 billion mobile
monthly active people
1.44 billion monthly
active people
Approximately 82.4% of our daily active users are outside the US and Canada
© 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain

8. What is facebook?
Photos
Videos
Newsfeed
Likes
© 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain
Dynamic Requests
Javascript/CSSStatus Updates
Static Requests
Messaging
(from traffic's perspective)

9. What is facebook?
Photos
Videos
Newsfeed
Likes
© 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain
Dynamic Requests
Javascript/CSSStatus Updates
Static Requests
Messaging
(from traffic's perspective)
Terabits of egress (outgoing bits per second)

10. What is facebook?
Photos
Videos
Newsfeed
Likes
© 2015 Facebook | Dublin | Credit: Network icon by Daniel Gamage https://thenounproject.com/term/network/49138/ Public Domain
Dynamic Requests
Javascript/CSSStatus Updates
Static Requests
Messaging
(from traffic's perspective)

11. © 2015 Facebook | Menlo Park | Credit:
What are we talking about?
HHVM

12. © 2015 Facebook | Menlo Park | Credit:
What are we not talking about?
HHVM
MySQL
Cache
Msgs
Feed
Ads

14. 7 Days
Egress
Ingress
Weekly egress cycle

15. 11 AM
24 hours
3 PM
Diurnal egress Cycle
Time zone == Pacific
(-0800 GMT)

16. United Kingdom
Canada
Indonesia
7 PM
24 hours
9 AM
Sum of timezones
Time zone == Pacific
(-0800 GMT)

17. TCP/IP Review

18. © 2015 Facebook | Menlo Park | Credit:
OSI Model
Layer Purpose Ex
7: Application High-Level API HTTP, SPDY, MQTT
6: Presentation Data Translation ASCII, JPEG
5: Session Communication Session RPC
4: Transport Transmission TCP, UDP
3: Network Address, Routing, Flow IPv6, IPv4
2: Data Link Reliable Physical Comm. IEEE, 802.2
1: Physical Raw bit transmission DSL, USB

19. © 2015 Facebook | Menlo Park | Credit:
IP Header (OSI Layer 3)
Version DSCP ECN Flow Label
Payload Length Next Header Hop limit
Source Address
Destination Address
Data

20. © 2015 Facebook | Menlo Park | Credit:
TCP Header (OSI Layer 4)
Source Port Destination Port
Sequence Number
Acknowledgement Number
….
Application Payload

21. © 2015 Facebook | Menlo Park | Credit:
HTTP Request (OSI Layer 7)
GET / HTTP/1.1
host: www.facebook.com

22. © 2015 Facebook | Menlo Park | Credit:
Putting it all together
Version DSCP ECN Flow Label
Payload Length Next Header Hop limit
Source Address
Destination Address
Source Port Destination Port
Sequence Number
Acknowledgement Number
….
GET / HTTP/1.1
host: www.facebook.com
L3
L4
L7

23. © 2015 Facebook | Menlo Park | Credit:
Putting it all together
IP Packet
TCP Segment
HTTP Request

24. Serving Dynamic Facebook
Requests

25. © 2015 Facebook | Menlo Park | Credit:
FB Request -- one web server
low rps
how do we get more
rps?!
GET /
HHVM
AAAA www.facebook.com
DNS
rps = requests per second
<html>...
AAAA 2a03:2880:2130:cf05:face:b00c::1

26. © 2015 Facebook | Menlo Park | Credit:
Add a load balancer!
www ?
GET /
<html>...
L7LB
(proxygen)
PHP
PHP
HHVM
GET /
<html>...
AAAA 2a03:2880:2130:cf05:face:b00c::1
DNS
rps = requests per second
how do we get more
rps?!
lots more rps low rps
AAAA www.facebook.com

27. © 2015 Facebook | Menlo Park | Credit:
Add another load balancer!
www ?
GET /
<html>...
L7LB
(proxygen)
PHPL4LB
(ipvs)
PHP
PHP
GET /
<html>...
L7LB
(proxygen)
L7LB
(proxygen)
GET /
<html>...
DNS
how do we get more rps?!
network bound lots more rps low rps
PHP
HHVM
AAAA 2a03:2880:2130:cf05:face:b00c::1
AAAA www.facebook.com

28. © 2015 Facebook | Menlo Park | Credit:
Add another load balancer!
GET /
<html>...
L7LB
(proxygen)
L4LB
(shiv)
GET /
<html>...
L7LB
(proxygen)
L7LB
(proxygen)
GET /
<html>...
L4LB
(ipvs)
DNS
network bound
lots more rps low rps
L7LB
(proxygen)
PHP
PHP
PHP
PHP
HHVM
PHP
PHP
PHP
PHP
HHVM
ECMP
AAAA 2a03:2880:2130:cf05:face:b00c::1
AAAA www.facebook.com

29. © 2015 Facebook | Menlo Park | Credit:
Front end Web
Cluster
~10
~100
Thousands

30. © 2015 Facebook | Menlo Park | Credit:
cont.
x 10 or more

31. © 2015 Facebook | Menlo Park | Credit:
More RPS? Add another cluster!
w
GET /
<html>...
L7LB
(proxygen)
PHP
L4LB
(shiv)
PHP
HHVM
how do we get more rps?!
GET /
<html>...
L7LB
(proxygen)
L7LB
(proxygen)
GET /
<html>...
L4LB
(ipvs)
GET /
<html>...
L7LB
(proxygen)
PHP
L4LB
(shiv)
PHP
HHVM
GET /
<html>...
L7LB
(proxygen)
L7LB
(proxygen)
GET /
<html>...
L4LB
(ipvs)
DNS
AAAA www.facebook.com
AAAA 2a03:2880:2130:cf05:face:b00c::1

32. © 2015 Facebook | Menlo Park | Credit:
Add another datacenter!
w
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
G
<
DNS
AAAA www.facebook.com
AAAA 2a03:2880:2130:cf05:face:b00c::1

33. Not really top down
1.2.3.4
L4LB
L7LB
HHVM

34. © 2015 Facebook | Menlo Park | Credit:
Datacenter Locations

35. Load Balancing: L4/L7

36. © 2015 Facebook | Menlo Park | Credit:
Let's break it down
GET /
<html>...
L7LB
(proxygen)
PHP
L4LB
(shiv)
PHP
HHVM
GET /
<html>...
L7LB
(proxygen)
L7LB
(proxygen)
GET /
<html>...
L4LB
(ipvs)
www?
A 1.2.3.4 DNS
L4 = IP/port
L7 = HTTP

37. © 2015 Facebook | Menlo Park | Credit:
OSI Model: What is L4/L7?
L7LB
(proxygen)
L4LB
(ipvs)
Layer Purpose Ex
7: Application High-Level API HTTP, SPDY, MQTT
6: Presentation Data Translation ASCII, JPEG
5: Session Communication Session RPC
4: Transport Transmission TCP, UDP
3: Network Address, Routing, Flow IPv6, IPv4
2: Data Link Reliable Physical Comm. IEEE, 802.2
1: Physical Raw bit transmission DSL, USB
ECMP

38. L4LB L4LB
L4LB
ECMP
Hash
BGP
L7LB
L7LB
L7LB
L7LB
NOTE: L7 (proxygen) Listens to the VIP on loopback (lo) interface. Not eth0.
face:b00c::1 face:b00c::1
face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)

39. L4LB
State
Table
+
Hash
ECMP
Hash
BGP
L4LB L4LB
L7LB
L7LB
L7LB
L7LB
NOTE: L7 (proxygen) Listens to the VIP on loopback (lo) interface. Not eth0.
Service
Discovery
(Zookeeper)
face:b00c::1 face:b00c::1
face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)

40.
L7LB
L7LB
L7LB
L7LB
L4LB L4LB
L4LB Example
State
Table
+
Hash
ECMP
Hash
A
face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)
face:b00c::1 face:b00c::1

41.
L7LB
L7LB
L7LB
L7LB
L4LB L4LB
L4LB Routing
D
State
Table
+
Hash
ECMP
Hash
A
A
face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)
face:b00c::1 face:b00c::1

42.
L7LB
L7LB
L7LB
L7LB
L4LB L4LB
L4LB Routing
B
State
Table
+
Hash
ECMP
Hash
face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)
face:b00c::1 face:b00c::1

43.
L7LB
L7LB
L7LB
L7LB
L4LB L4LB
L4LB Routing
B
State
Table
+
Hash
ECMP
Hash
face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)
face:b00c::1 face:b00c::1

44. Direct
Server
Return
TCP Routing
Facebook
HHVM
HHVM
L4LB
HHVM
HHVM
L7LBL7LBL7LBL7LB
TCP
SSL
HTTP
face:b00c::1
face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo) face:b00c::1 (lo)

45. © 2015 Facebook | Menlo Park | Credit:
Remember this?
Original IP Packet
TCP Segment
HTTP Request

46. © 2015 Facebook | Menlo Park | Credit:
IP Packet from IPVS
IP in IP encapsulation
Original IP Packet
TCP Segment
HTTP Request

47. Direct
Server
Return
Original IP Packet
TCP Segment
HTTP Request
L4LB
L7LB
Response IP Pckt
TCP Segment
HTTP Response
face:b00c::1 (lo)
IP Packet from IPVS
Original IP Packet
TCP Segment
HTTP Request
L4LB L7LB
TCP/SSL Term
Forward HTTP RequestIP Packet from IPVS
Original IP Packet
TCP Segment
HTTP Request
face:b00c::1 (lo)face:b00c::1
face:b00c::1

48. Edge PoP’s & Reducing Latency

49. International RTT
circa 11/2011

50. Seoul -> Oregon
TCP Connect: 150ms

51. HTTPS Seoul -> Oregon
GET
HTTP 1.1
ChangeCipherSpec
ChangeCipherSpec
ACK
ServerHello
SYN+ACK
SYN
ClientHello
TCP conn
established:
150 ms
SSL session
established:
450 ms
Response
Received
600 ms
75ms

52. Seoul -> Tokyo -> Oregon
TCP Connect: 30ms
SSL Session: ??
HTTP Response: ??
NRT

53. HTTPS Seoul->Tokyo->Oregon
GET
HTTP 1.1
Sessions
established:
90 ms
(vs 450 ms)
Response
Received:
240 ms
60ms
GET
HTTP 1.1 200
15ms
Request
Received

54. Seoul -> Oregon
NRT
TCP Connect: 150ms
SSL Session: 450ms
HTTP Response: 600ms
30ms
90ms
240ms

55. © 2015 Facebook | Menlo Park | Credit:
Edge POP Locations
*POP = points of presence.

56. © 2015 Facebook | Menlo Park | Credit:
How do the LB’s in PoP’s work?
"Flag of Sweden". Licensed under Public Domain via Wikipedia - http://en.wikipedia.org/wiki/File:Flag_of_Sweden.svg#/media/File:Flag_of_Sweden.svg
TCP Routing
(ip/port)
Facebook
L4LB
L7LB
TCP/SSL
HTTP
HHVM
face:b00c::1
face:b00c::1 (lo)
HHVM
L7LB
L4LB
face:b00c::1
face:b00c::1 (lo)

57. DNS LB: Cartographer

58. © 2015 Facebook | Menlo Park | Credit:
Edge POP Locations
*POP = points of presence.

59. DNS LB Decision
Considerations:
•Closest Edge** to user
•Capacity
•Health
•Geo data
???

60. Sonar: Measuring “Closeness”
NRT
HKG
SEA
LAX
ORD

61. Cartographer Architecture
Internet
ISP DNS
resolver
Facebook Infrastructure
DNS mapcartographer
offline processing
DNS Serverresolvers/latency
capacity
routing
health

62. Cartographer Architecture

63. Cartographer in action

64. Regional Load Shedding

65. Global Load Shedding

66. Monitoring

67. Monitoring (Not enough time)

68. Open Source

69. Open Source Components
• Proxygen Libs
https://github.com/facebook/proxygen
• HHVM
https://hhvm.com
• TinyDNS
https://cr.yp.to/djbdns/tinydns.html
• IPVS (IP Virtual Server)
http://www.linuxvirtualserver.org/software/ipvs.html
• ExaBGP
https://github.com/Exa-Networks/exabgp
• Python
https://python.org
• Zookeeper
https://zookeeper.apache.org

70. Photo Credits
http://www.flickr.com/photos/27587002@N07/5170590074
http://www.flickr.com/photos/yaketyyakyak/7001664846
http://www.flickr.com/photos/hinnosaar/3778903507
http://www.flickr.com/photos/eamoncurry/8698726494
http://www.flickr.com/photos/43158397@N02/4514113429
http://www.flickr.com/photos/nobusue/6876280595
http://www.flickr.com/photos/29487672@N07/14760573314
http://www.flickr.com/photos/joyosity/3595242078
http://www.flickr.com/photos/kyntharyn74/3262089319
http://www.flickr.com/photos/rexipe/826987087
http://www.flickr.com/photos/lablasco/6815671096
https://thenounproject.com/term/iphone-profile/54906/
https://thenounproject.com/term/browser/59091/