Netflix was one of the earliest very large AWS customers. By 2014, we were running hundreds of applications in Amazon EC2. That was great, until we needed to move to VPC. Given our scale, uptime requirements, and the decentralized nature of how we manage our production environment, the VPC migration (still ongoing) presented particular challenges for us and for AWS as it sought to support our move. In this talk, we discuss the starting state, our requirements and the operating principles we developed for how we wanted to drive the migration, some of the issues we ran into, and how the tight partnership with AWS helped us migrate from an EC2-Classic platform to an EC2-VPC platform.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Andrew Braham, Manager - Cloud Network Engineering, Netflix
Laurie Ferioli, Senior Program Manager, Netflix
December 1, 2016
Moving Mountains
Netflix’s Migration into VPC
NET304

4. 2008

5. 2010 2011 2012 2013 2014

6. Why.
Learnings.
How.
What.

7. Security.
Networking.
Configurability.
Diagnostics.
VPC Advantages.

8. Netflix Ecosystem.
Lots and lots.

9. 10s of critical tools.

10. Delivery.

11. Monitoring.

12. 10s of critical tools.
100s of databases & ELBs.
1,000s of services.
10,000s of instances.

13. Migration Management.
If “plan A” didn’t work,
the alphabet has 25 more letters.

14. Guiding Principles.
Seamless to engineers.
Velocity of innovation.
Opportunistic improvements.

15. Service Classification.
Small non-critical
Large non-critical
Small critical
Large critical
Critical to members
Size

16. 2016 – The Migration.
Jan Feb Mar Apr May June Jul Aug Sept Oct
Infrastructure
Large non-critical apps
Small non-critical apps
The long poles – services with long migrations
Large critical apps
Small critical apps
C
l
e
a
n
u
p

17. VPC Exploration.
EXPECTATIONS.
SURPRISES.

18. Primary Goals.
Identify environmental differences.
Alignment on desired VPC end state.
Develop migration strategy.

19. Technical Challenges.
Network Routing.
DNS.
Security Groups.

20. Account Rationale.
Security compartmentalization.
Administrative Domain.
Rate Limit Restrictions.
Capacity Constraints.

21. Account Classifiers.
Business Purpose.
Operational Model.
User Access.

22. Regional Routing.
PACKET KUNG-FU.

23. ClassicLink.
ClassicLink is a feature that allows EC2-
Classic instances the ability to communicate
directly with instances in a single VPC in the
same region.

24. IP Addressing Allocation.
Amazon AWS utilizes 10.0.0.0/8.
Globally non-overlapping IP addresses.
Network Size.

25. RFC 6598.
100.64.0.0/10 network is a reserved
block to facilitate Carrier Grade
Network Address Translation (CGN).

26. IP Addressing Reservation.
Cloud IP (VPC EIP API).
ENI Auto-attach.

27. VPC Subnet Layout.
External Subnets.
Internal Subnets.
Partner Subnets.

28. VPC Subnet Layout.
/16
External Subnets.
Internal Subnets.
Partner Subnets.

29. VPC Subnet Layout.
External Subnets.
Internal Subnets.
Partner Subnets.
/18 /18
/18 /18

30. VPC Subnet Layout.
/18 /18
/18
/20/20
/20 /20
External Subnets.
Internal Subnets.
Partner Subnets.

31. VPC Subnet Layout.
/18 /18
/18
/20
/20 /20
/22/22
/22/22
External Subnets.
Internal Subnets.
Partner Subnets.

32. VPC Subnet Layout.
Availability Zone A Availability Zone B Availability Zone C
/20
0/0 => IGW
/20
0/0 => IGW
/20
0/0 => IGW
/18
0/0 => NGW
/18
0/0 => NGW
/22
0/0 => NGW
/22
0/0 => NGW
/22
0/0 => NGW
/18
0/0 => NGW
Internet
Gateway
(IGW)
VPN
Gateway
(VGW)
NAT
Gateway
(NGW)

33. Scaling NAT Gateways.
NAT
Gateway
(NGW)
Availability Zone A
/18
0.0.0.0/0 => NGW

34. Scaling NAT Gateways.
Availability Zone A
/18
0.0.0.0/1 => NGW #1
128.0.0.0/1 => NGW #2
NAT
Gateway
(NGW #1 )
NAT
Gateway
(NGW #2)

35. Scaling NAT Gateways.
Availability Zone A
/18
0.0.0.0/2 => NGW #1
64.0.0.0/2 => NGW #2
128.0.0.0/2 => NGW #3
192.0.0.0/2 => NGW #4
NAT
Gateway
(NGW #2 )
NAT
Gateway
(NGW #3)
NAT
Gateway
(NGW #4)
NAT
Gateway
(NGW #1)

36. Scaling NAT Gateways.
NAT
Gateway
(NGW)
Availability Zone A
/18
0.0.0.0/0 => NGW

37. Scaling NAT Gateways.
Availability Zone A
NAT
Gateway
(NGW #1 )
NAT
Gateway
(NGW #2)
/19
0.0.0.0/0 => NGW #1
/19
0.0.0.0/0 => NGW #2

38. Scaling NAT Gateways.
Availability Zone A
NAT
Gateway
(NGW #2 )
NAT
Gateway
(NGW #3)
NAT
Gateway
(NGW #4)
NAT
Gateway
(NGW #1)
/20
0.0.0.0/0 => NGW #3
/20
0.0.0.0/0 => NGW #4
/20
0.0.0.0/0 => NGW #1
/20
0.0.0.0/0 => NGW #2

39. ClassicLink.
EC2-Classic
VPC

40. ClassicLink.
Golf
Zulu
Alpha
gethostname(zulu.public)
10.0.0.100
gethostname(alpha.public)
10.0.0.200

41. ClassicLink.
Golf
Zulu
Alpha
Issue:
gethostname(zulu.public)
54.aaa.bbb.ccc

42. Service Discovery.
Registration
• hostname.public.
• hostname.private.
• ipaddress.public.
• ipaddress.private.
Zulu

43. ClassicLink.
Golf
Zulu
Alpha
Issue:
gethostname(zulu.public)
54.aaa.bbb.ccc
Resolution:
DNS over ClassicLink
gethostname(zulu.public)
100.64.0.100

44. ClassicLink over Peering.
Golf
Zulu Alpha
Expectation:
gethostname(alpha.public)
100.64.128.200

45. ClassicLink over Peering.
Golf
Zulu Alpha
Issue:
gethostname(alpha.public)
54.xxx.yyy.zzz

46. ClassicLink over Peering.
Golf
Zulu Alpha
Issue:
gethostname(alpha.public)
54.xxx.yyy.zzz
Resolution:
ClassicLink over Peering
DNS over Peering
gethostname(alpha.public)
100.64.128.100

47. ClassicLink Everywhere.
Golf
Zulu Alpha
Romeo

48. ClassicLink Everywhere.
Golf
Zulu Alpha
Romeo

49. ClassicLink at Scale.
launch config1
1 2 3 N
. . . . .

50. Service Classification.
Small non-critical
Large non-critical
Small critical
Large critical
Critical to members
Size

51. Dependency Mappings.
Flow Collection.
IP Metadata.
Flow Analysis.

52. Global Routing
MORE PACKET KUNG-FU.

53. AWS Direct Connect.
Omega Bravo
Delta
Netflix Backbone

54. Global Backbone.

55. Direct Connect.
Netflix Backbone
us-west-1 us-east-1
eu-west-1

56. Backbone Traffic.
100.64.0.0/10
Netflix Backbone
10.0.0.0/8 10.0.0.0/8

57. Backbone Traffic.
100.64.0.0/10
Netflix Backbone
10.0.0.0/8 10.0.0.0/8

58. Backbone Traffic.
100.64.0.0/10 10.0.0.0/8
DNS
Netflix Backbone

59. Global Infrastructure.
Netflix Backbone
Region 1
Classic
Classic
VPC
VPC
Corp
Region 2
Classic
Classic
VPC
VPC
Corp
3rd
Party
3rd
Party

60. Retrospective.
MULLIGANS.
SECOND CHANCES.

61. Lessons Learned.
IP address scheme.
Traffic patterns.
Partner engagement.
Technical debt.

62. Features.
ClassicLink.
ClassicLink over Peering.
DNS over Peering.
EC2 DNS for non-RFC 1918.

63. Delivery.

65. Thank you!

66. Remember to complete your
evaluations!

67. Related Sessions
• NET201 – Creating Your Virtual Data Center: VPC Fundamentals and
Connectivity Options
• NET303 – NextGen Networking: New Capabilities for Amazon’s Virtual
Private Cloud
• NET402 – Deep Dive: AWS Direct Connect and VPNs
• NET 403 – Elastic Load Balancing Deep Dive and Best Practices
• NET 404 – Making Every Packet Count

68. Questions?

69. ClassicLink.
Golf
Zulu
Alpha
Expectation:
gethostname(zulu.public)
100.64.0.100

70. ClassicLink.
Golf
Zulu
Alpha
Issue:
gethostname(zulu.public)
54.aaa.bbb.ccc