Making decisions today for tomorrow's technology—from DNS to AWS Direct Connect, ELBs to ENIs, VPCs to VPNs, the Cloud Network Engineering team at Netflix are resident subject matter experts for a myriad of AWS resources. Learn how a cross-functional team automates and manages an infrastructure that services over 125 million customers while evaluating new features that enable us to continue to grow through our next 100 million customers and beyond.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Another Day in the Life of a Cloud
Network Engineer at Netflix
Joel Kodama
Cloud Network SRE
Netflix
N E T 3 1 2
Donavan Fritz
Cloud Network SRE
Netflix

3. RE:INVENT 2018
NET-312
AWS Cloud

4. RE:INVENT 2018
NET-312
AWS Cloud
Amazon EC2 Classic VPC
VPN
Connection

5. RE:INVENT 2018
NET-312
Subnets
IGWs
VGWs
NGWs
Route tables
Endpoints
Prefix lists
AWS Direct Connect
CGWs

6. RE:INVENT 2018
NET-312
?
?
?
?
?
?
??
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?

7. RE:INVENT 2018
NET-312
Amazon EC2 Classic VPC
Zone 1
Region
AWS Cloud
Zone 2 Zone 3 Zone 1 Zone 2 Zone 3
Public Public Public
Private Private Private
NAT Gateway NAT Gateway NAT Gateway

8. RE:INVENT 2018
NET-312
Amazon EC2 Classic VPC
Zone 1
Region
AWS Cloud
Zone 2 Zone 3 Zone 1 Zone 2 Zone 3
Public Public Public
Private Private Private
NAT Gateway NAT Gateway NAT Gateway

9. RE:INVENT 2018
NET-312
●Inter-VPC communication
VPC
Region
VPC
Instances Instances
1
2
3
AWS Cloud
VPC
peering
Internet

10. RE:INVENT 2018
NET-312
Bidirectional private IP
communication
Internet
VPC peering
AWS Direct
Connect
●Inter-VPC communication

11. RE:INVENT 2018
NET-312
Bidirectional private IP
communication
Internet
VPC peering
AWS Direct
Connect
●Inter-VPC communication

12. RE:INVENT 2018
NET-312
Bidirectional private IP
communication
Security group
references
Internet
VPC peering
AWS Direct
Connect
●Inter-VPC communication

13. RE:INVENT 2018
NET-312
Bidirectional private IP
communication
Security group
references
Internet
VPC peering
AWS Direct
Connect
●Inter-VPC communication

14. RE:INVENT 2018
NET-312
Bidirectional private IP
communication
Security group
references
No bandwidth
constraints
Internet
VPC peering
AWS Direct
Connect
●Inter-VPC communication

15. RE:INVENT 2018
NET-312
Bidirectional private IP
communication
Security group
references
No bandwidth
constraints
Internet
VPC peering
AWS Direct
Connect
●Inter-VPC communication

16. RE:INVENT 2018
NET-312
Account 4 VPC
Account 1 VPC Account 2 VPC
Account 3 VPC
Region 1
Instances
Instances
Instances Instances

17. RE:INVENT 2018
NET-312
AWS Cloud
VPC
us-west-2
Office building
Netflix MPLS Backbone
VPC
us-east-1
VPC
eu-west-1
SEA IAD LHR
Office building

18. RE:INVENT 2018
NET-312
Single unified network
across all accounts and regions

19. RE:INVENT 2018
NET-312
Unique

20. RE:INVENT 2018
NET-312
$ python3
>>>

21. RE:INVENT 2018
NET-312
$ python3
>>> import cldnet
>>>

22. RE:INVENT 2018
NET-312
$ python3
>>> import cldnet
>>>
>>> vpc0 = cldnet.create_vpc(account='1234567', region='us-east-1', name=‘vpc0', prefix_length=16)
>>> vpc1 = cldnet.create_vpc(account='9876543', region=‘us-east-1', name=‘vpc1', prefix_length=16)
>>>

23. RE:INVENT 2018
NET-312
$ python3
>>> import cldnet
>>>
>>> vpc0 = cldnet.create_vpc(account='1234567', region='us-east-1', name=‘vpc0', prefix_length=16)
>>> vpc1 = cldnet.create_vpc(account='9876543', region=‘us-east-1', name=‘vpc1', prefix_length=16)
>>>
>>> vpc0.peer_to(vpc1)
>>>

24. RE:INVENT 2018
NET-312
$ python3
>>> import cldnet
>>>
>>> vpc0 = cldnet.create_vpc(account='1234567', region='us-east-1', name=‘vpc0', prefix_length=16)
>>> vpc1 = cldnet.create_vpc(account='9876543', region=‘us-east-1', name=‘vpc1', prefix_length=16)
>>>
>>> vpc0.peer_to(vpc1)
>>>
>>> vpc0.resize(prefix_length=16)
>>>

25. RE:INVENT 2018
NET-312
$ python3
>>> import cldnet
>>>
>>> cldnet.standardize_vpc_peerings(”peerings.yaml”)
>>>
# peerings.yaml
---
VpcGroups:
UsWest2FullMesh:
- vpc-111111
- vpc-222222
- vpc-333333
UsEast1FullMesh:
- vpc-444444
- vpc-555555
- vpc-666666
VpcPeerings:
- FromGroup: UsEast1FullMesh
ToGroup: UsEast1FullMesh
- FromGroup: UsWest2FullMesh
ToGroup: UsWest2FullMesh

26. RE:INVENT 2018
NET-312

27. RE:INVENT 2018
NET-312
Account 1 VPC
us-west-2
Instances
AWS Cloud
Account 1 VPC
us-east-1
Instances
Account 1 VPC
eu-west-1
Instances

28. RE:INVENT 2018
NET-312
Network-level
encryption
Inter Region VPC peering
AWS Direct Connect
●VPC inter-region communication

29. RE:INVENT 2018
NET-312
Network-level
encryption
Inter Region VPC peering
AWS Direct Connect
●VPC inter-region communication

30. RE:INVENT 2018
NET-312
Network-level
encryption
No bandwidth
constraints
Inter Region VPC peering
AWS Direct Connect
●VPC inter-region communication

31. RE:INVENT 2018
NET-312
Network-level
encryption
No bandwidth
constraints
Inter Region VPC peering
AWS Direct Connect
●VPC inter-region communication

32. RE:INVENT 2018
NET-312
Network-level
encryption
No bandwidth
constraints
Security group
references
Inter Region VPC peering
AWS Direct Connect
●VPC inter-region communication

33. RE:INVENT 2018
NET-312
Network-level
encryption
No bandwidth
constraints
Security group
references
Inter Region VPC peering
AWS Direct Connect
●VPC inter-region communication

34. RE:INVENT 2018
NET-312
Network-level
encryption
No bandwidth
constraints
Security group
references
Summarizable
inter-region VPC
routes
Inter Region VPC peering
AWS Direct Connect
●VPC inter-region communication

35. RE:INVENT 2018
NET-312
Network-level
encryption
No bandwidth
constraints
Security group
references
Summarizable
inter-region VPC
routes
Inter Region VPC peering
AWS Direct Connect
●VPC inter-region communication

36. RE:INVENT 2018
NET-312
Pushing limits

37. RE:INVENT 2018
NET-312

38. RE:INVENT 2018
NET-312
●Titus network requirements
IP per container (fidelity of IP address)
Fast deployments

39. RE:INVENT 2018
NET-312
IP
IPv4 addressElastic Network
Interface
Amazon EC2
Instance

40. RE:INVENT 2018
NET-312
IP
IP
IP
IP
IP
IP
IPv4 address Container
Elastic Network
Interface
Amazon
EC2 Instance

41. RE:INVENT 2018
NET-312
●Container startup AWS API calls
1. CreateNetworkInterface
2. AttachNetworkInterface
3. AssignPrivateIpAddress
4. ModifyNetworkInterfaceAttribute

42. RE:INVENT 2018
NET-312
Can we populate network
interfaces at start up?

43. RE:INVENT 2018
NET-312
0
100000
200000
300000
400000
500000
600000
1 2 3 4 5
NumberofIPaddress
Instances w/ 8 Elastic Network Interfaces (ENIs)
2 IPs per ENI
4 IPs per ENI
8 IPs per ENI
16 IPs per ENI
32 IPs per ENI
1x /16 IPv4 CIDR
5x /16 IPv4 CIDR
m4

44. RE:INVENT 2018
NET-312
VPC
Public
Private
NAT Gateway
Private
Availability Zone
Public
Private
NAT Gateway
Private
Availability Zone
Public
Private
NAT Gateway
Private
Availability Zone
EC2 Instances
Containers
EC2 Instances
Containers
EC2 Instances
Containers
EC2 Instances
Containers
EC2 Instances
Containers
EC2 Instances
Containers

45. RE:INVENT 2018
NET-312
IP isolation
Co-tenant
●Titus network options

46. RE:INVENT 2018
NET-312
IP isolation
Co-tenant
●Titus network options

47. RE:INVENT 2018
NET-312
IP isolation Shared security
groups
Co-tenant
●Titus network options

48. RE:INVENT 2018
NET-312
IP isolation Shared security
groups
Co-tenant
●Titus network options

49. RE:INVENT 2018
NET-312
IP isolation Shared security
groups
AWS API isolation
Co-tenant
●Titus network options

50. RE:INVENT 2018
NET-312
IP isolation Shared security
groups
AWS API isolation
Co-tenant
●Titus network options

51. RE:INVENT 2018
NET-312
IP isolation Shared security
groups
AWS API isolation Max number of IPs
Co-tenant ~98K shared
●Titus network options

52. RE:INVENT 2018
NET-312
IP isolation Shared security
groups
AWS API isolation Max number of IPs
Co-tenant ~98K shared
Dedicated subnet ~196K
●Titus network options

53. RE:INVENT 2018
NET-312
IP isolation Shared security
groups
AWS API isolation Max number of IPs
Co-tenant ~98K shared
Dedicated subnet ~196K
Dedicated VPC ~295K
●Titus network options

54. RE:INVENT 2018
NET-312
IP isolation Shared security
groups
AWS API isolation Max number of IPs
Co-tenant ~98K shared
Dedicated subnet ~196K
Dedicated VPC ~295K
Dedicated account ~295K
●Titus network options

55. RE:INVENT 2018
NET-312

56. RE:INVENT 2018
NET-312
●DNS service discovery

57. RE:INVENT 2018
NET-312
Auto Scaling group State
Eureka
Foo
Bar
Where is bar?
●DNS service discovery

58. RE:INVENT 2018
NET-312
Auto Scaling group
Auto Scaling group State
Eureka
In service
Out of service
Foo
Bar v0
Bar v1
Where is bar?
●DNS service discovery

59. RE:INVENT 2018
NET-312
Auto Scaling group
Auto Scaling group State
Eureka
Out of service
In service
Foo
Bar v0
Bar v1
Where is bar?
●DNS service discovery

60. RE:INVENT 2018
NET-312
Eureka
Foo
Where is
bar.netflix.net?
●DNS service discovery
Auto Scaling group
Auto Scaling group State
Out of service
In service
Bar v0
Bar v1

61. RE:INVENT 2018
NET-312
Eureka Amazon Route53
Sync State
Foo
Where is
bar.netflix.net?
●DNS service discovery
Auto Scaling group
Auto Scaling group State
Out of service
In service
Bar v0
Bar v1

62. RE:INVENT 2018
NET-312
Eureka Amazon Route53
Sync State
Foo
Where is
bar.netflix.net?
●DNS service discovery
Auto Scaling group
Auto Scaling group State
Out of service
In service
Bar v0
Bar v1

63. RE:INVENT 2018
NET-312
Decision time

64. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
●DNS service discovery decision

65. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
●DNS service discovery decision

66. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
●DNS service discovery decision

67. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
●DNS service discovery decision

68. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
Route53
auto naming
●DNS service discovery decision

69. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
Route53
auto naming
●DNS service discovery decision

70. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
Route53
auto naming
●DNS service discovery decision

71. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
Route53
auto naming
●DNS service discovery decision

72. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
Route53
auto naming
???
●DNS service discovery decision

73. RE:INVENT 2018
NET-312
Auto remove stale
DNS records
Allow
“out of service”
AWS API rate limits
Existing solution
Route53
auto naming
Eureka DNS
●DNS service discovery decision

74. RE:INVENT 2018
NET-312
Auto Scaling group
Auto Scaling group State
Eureka Amazon Route53
Sync
In service
State
Out of service
Foo
Bar
Bar
Where is
bar.netflix.net?

75. RE:INVENT 2018
NET-312
Auto Scaling group
Auto Scaling group State
Eureka
In service
Out of service
Foo
Bar
Bar
Where is
bar.netflix.net?
Auto Scaling group
Eureka DNS

76. RE:INVENT 2018
NET-312
Static IPv4 IPv6 UDP
Authoritative name server static IP

77. RE:INVENT 2018
NET-312
Static IPv4 IPv6 UDP
Network Load
Balancer
Authoritative name server static IP

78. RE:INVENT 2018
NET-312
Static IPv4 IPv6 UDP
Network Load
Balancer
Authoritative name server static IP

79. RE:INVENT 2018
NET-312
Static IPv4 IPv6 UDP
Network Load
Balancer
Elastic IP
Authoritative name server static IP

80. RE:INVENT 2018
NET-312
Static IPv4 IPv6 UDP
Network Load
Balancer
Elastic IP
Authoritative name server static IP

81. RE:INVENT 2018
NET-312
Static IPv4 IPv6 UDP
Network Load
Balancer
Elastic IP
Elastic network
interface
Authoritative name server static IP

82. RE:INVENT 2018
NET-312
Static IPv4 IPv6 UDP
Network Load
Balancer
Elastic IP
Elastic network
interface
Authoritative name server static IP

83. RE:INVENT 2018
NET-312
Authoritative name server static IP
VPC
us-west-2
Subnet
203.0.113.100
2001:db8::100

84. RE:INVENT 2018
NET-312
Eureka DNS v0
Authoritative name server static IP
VPC
us-west-2
Subnet
203.0.113.100
2001:db8::100

85. RE:INVENT 2018
NET-312
Eureka DNS v0
Authoritative name server static IP
VPC
us-west-2
Subnet
203.0.113.100
2001:db8::100
Eureka DNS v1

86. RE:INVENT 2018
NET-312
Eureka DNS v1 Eureka DNS v0
Authoritative name server static IP
VPC
us-west-2
Subnet
203.0.113.100
2001:db8::100

87. RE:INVENT 2018
NET-312
Eureka DNS v1
Authoritative name server static IP
VPC
us-west-2
Subnet
203.0.113.100
2001:db8::100

88. RE:INVENT 2018
NET-312
Authoritative name server static IP
VPC
Public
us-west-2
AWS Cloud
us-west-2a
Auto Scaling group
Eureka DNS
Public
us-west-2b
Eureka DNS
Public
us-west-2c
Eureka DNSEureka DNS Eureka DNS Eureka DNS

89. RE:INVENT 2018
NET-312
DNS delegation hierarchy
eureka.netflix.net

90. RE:INVENT 2018
NET-312
DNS delegation hierarchy
test prod
eureka.netflix.net

91. RE:INVENT 2018
NET-312
DNS delegation hierarchy
test
us-east-1 us-west-2
prod
us-east-1 us-west-2
eureka.netflix.net

92. RE:INVENT 2018
NET-312
DNS delegation hierarchy
test
us-east-1 us-west-2
MyService MyService
prod
us-east-1 us-west-2
MyService MyService
eureka.netflix.net

93. RE:INVENT 2018
NET-312
DNS delegation hierarchy
test
us-east-1 us-west-2
MyService MyService
prod
us-east-1 us-west-2
MyService MyService
myservice.us-east-1.test.eureka.netflix.net
myservice.us-west-2.test.eureka.netflix.net
myservice.us-east-1.prod.eureka.netflix.net
myservice.us-west-2.prod.eureka.netflix.net
eureka.netflix.net

94. RE:INVENT 2018
NET-312
Let’s be more proactive!

95. RE:INVENT 2018
NET-312
DNS steering
VPC
us-west-2
AWS Cloud
VPC
us-east-1
VPC
eu-west-1
Where is www.netflix.com?

96. RE:INVENT 2018
NET-312
DNS steering
VPC
us-west-2
AWS Cloud
VPC
us-east-1
VPC
eu-west-1
?
?
?

97. RE:INVENT 2018
NET-312
DNS steering: Geography
VPC
us-west-2
AWS Cloud
VPC
us-east-1
VPC
eu-west-1
?
?
?

98. RE:INVENT 2018
NET-312
DNS steering: Geography
VPC
us-west-2
AWS Cloud
VPC
us-east-1
VPC
eu-west-1
Users in San Francisco

99. RE:INVENT 2018
NET-312
DNS steering: Geography
VPC
us-west-2
AWS Cloud
VPC
us-east-1
VPC
eu-west-1
Users in London

100. RE:INVENT 2018
NET-312
DNS steering: Geography
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8
Latency: Not optimized

101. RE:INVENT 2018
NET-312
DNS steering: Geography
Perfect
balance
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8
Latency: Not optimized

102. RE:INVENT 2018
NET-312
DNS steering: Geography
Perfect
balance
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8
Latency: Not optimized
Availability
risk

103. RE:INVENT 2018
NET-312
DNS steering: Geography
Perfect
balance
(3 region)
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8
Latency: Not optimized
Availability
risk
Cost driver
Perfect
balance
(2 region)

104. We can do one better

105. RE:INVENT 2018
NET-312
Amazon Route53
latency based routing

106. RE:INVENT 2018
NET-312
DNS steering: Route 53 latency routing
VPC
us-west-2
AWS Cloud
VPC
us-east-1
Account 1 VPC
eu-west-1
Users in San Francisco

107. RE:INVENT 2018
NET-312
VPC
us-west-2
AWS Cloud
VPC
us-east-1
Account 1 VPC
eu-west-1
Users in London
DNS steering: Route 53 latency routing

108. RE:INVENT 2018
NET-312 Latency: optimized
DNS steering: Route 53 latency routing

109. RE:INVENT 2018
NET-312 Latency: optimized
Availability
risk
Cost driver
Perfect
balance
(3 region)
Perfect
balance
(2 region)
DNS steering: Route 53 latency routing

110. RE:INVENT 2018
NET-312
DNS steering

111. RE:INVENT 2018
NET-312
DNS steering

112. RE:INVENT 2018
NET-312
DNS steering
120ms
80ms
100ms

113. RE:INVENT 2018
NET-312
DNS steering
120ms
80ms
100ms 20ms
200ms
100ms

114. RE:INVENT 2018
NET-312
DNS steering
120ms
80ms
100ms
300ms
120ms
150ms
20ms
200ms
100ms

115. RE:INVENT 2018
NET-312
DNS steering
120ms
80ms
100ms
300ms
120ms
150ms
20ms
200ms
100ms

116. RE:INVENT 2018
NET-312
us-west-2 us-east-1 eu-west-1
Wisconsin, USA 100ms 80ms 120ms
Argentina 150ms 120ms 300ms
Spain 200ms 100ms 20ms
Real User Measurements
DNS steering: Route 53 latency routing

117. RE:INVENT 2018
NET-312
us-west-2 us-east-1 eu-west-1
Wisconsin, USA 100ms 80ms 120ms
Argentina 150ms 120ms 300ms
Spain 200ms 100ms 20ms
Real User Measurements
DNS steering: Geography

118. RE:INVENT 2018
NET-312
us-west-2 us-east-1 eu-west-1
Wisconsin, USA 100ms 80ms 120ms
Argentina 150ms 120ms 300ms
Spain 200ms 100ms 20ms
Geo IP Prefix
Wisconsin, USA 203.0.113.0/24
Argentina 192.0.2.0/24
Spain 2001:DB8::/32
Geo Database
Real User Measurements
DNS steering: Geography

119. RE:INVENT 2018
NET-312
Prefix steering

120. RE:INVENT 2018
NET-312
DNS steering: Prefix steering
us-west-2 us-east-1 eu-west-1
203.0.113.0/24 100ms 80ms 120ms
192.0.2.0/24 150ms 120ms 300ms
2001:DB8::/32 200ms 100ms 20ms
Real User Measurements

121. RE:INVENT 2018
NET-312
DNS steering: Prefix steering
us-west-2 us-east-1 eu-west-1
203.0.113.0/24 100ms 80ms 120ms
192.0.2.0/24 150ms 120ms 300ms
2001:DB8::/32 200ms 100ms 20ms
Real User Measurements
100ms latency delta

122. RE:INVENT 2018
NET-312
DNS steering: Prefix steering
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8
Latency: delta optimized
Perfect
balance
Availability
risk

123. RE:INVENT 2018
NET-312

124. RE:INVENT 2018
NET-312

125. RE:INVENT 2018
NET-312
Decision time

126. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Joel Kodama
Cloud Network SRE
jkodama@netflix.com
Donavan Fritz
Cloud Network SRE
dfritz@netflix.com

127. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.