Customers from over all over the world streamed Forty Two Billion hours of Netflix content last year. The Netflix streaming service had been powered by the Amazon cloud with virtual machines for over five years, blazing a trail for similar architectures. In the last year, it invested in containers for batch-style jobs and service-style applications. Andrew Spyker will explain the potential containers have to help Netflix create a more productive development experience while simultaneously deepening its control over resource management. Join Andrew to see why Netflix is moving forward with containers, how it can leverage its existing operational machinery, and how it’s running containers with a similar guarantee of high availability as current Netflix infrastructure provides.

1. and containers
Andrew Spyker (@aspyker) - Engineering Manager
Not

2. About Netflix
● 86.7M members
● A few thousand employees
● 190+ countries
● > ⅓ NA internet download traffic
● 500+ Microservices
● Many 10’s of thousands VM’s
● 3 regions across the world

3. Netflix has a elastic, cloud native, immutable microservice
architecture using full devops built on VM’s!
3
Why are we messing around with containers?

4. Technical motivating factors for containers
● Simpler management of compute resources
● Simpler deployment packaging artifacts for compute jobs
● Need for a consistent local developer environment

5. Sampling of realized container benefits
Media Encoding - encoding research development time
● VM’s platform to container platform - 1 month vs. 1 week
Continuous Integration Testing
● Build all Netflix codebases in hours
● Saves development 100’s of hours of debugging
Edge Re-architecture using NodeJS
● Focus returns to app development
● Simplifies, speeds test and deployment
5

6. Batch applications

7. Multi-tenant (cgroups/Mesos)
historically used for batch
Linux cgroups

8. What do batch users want?
● Simple shared resources, run till done, job files
● NOT
○ EC2 Instance sizes, autoscaling, AMI OS’s
● WHY
○ Offloads resource management ops, Simpler

9. Introducing Titus
Batch
Job Management
Resource Management & Optimization
Container Execution
Integration
Workflow, Data Analysis, Adhoc
Upstream Systems

10. Netflix Batch Job Examples
● Algorithm Model Training (with GPU’s)

11. Netflix Batch Job Examples
● Media Encoding
● Digital Watermarking
1 1

12. Netflix Batch Job Examples
Open Connect
CDN Reporting
Adhoc
Reporting

13. ● Docker helped generalize use cases
● Scheduling required (GPU, elastic)
● Initially ignored failures (with retries)
● Time sensitive batch came later
Lessons Learned from Batch

14. Current Container Usage - Batch
● 1000’s of container hosts (g2, m4, r3 instances)
● 1000’s containers / hour average
● Large spikes of CI testing and Digital Watermarking
From day of 10/26
● 47K containers
● Bursts of 1000
containers in a
minute

15. Service applications

16. Why Services in containers?
Theory Reality
Developer

17. Opportunities to evolve our baking
● Java focused supported AMI, baking works well!
● However, wanted to allow
○ other stacks to evolve independent of OS updates
○ simplified builds (vs. Java and OS based tooling)
○ reliable smaller instances for dynamic languages
○ ability to develop locally with same image

18. Services are just long running batch?
Services
Job Management
Resource Management & Optimization
Container Execution
Integration
Service Apps
Batch

19. 19
Nope, not that easy - Titus Details
19
Titus UITitus UI
Docker
Registry
Docker
Registry
Rhea
container
container
container
docker
Titus Agent
metrics agent
Titus executor
logging agent
zfs
Mesos agent
docker
RheaTitus API
Cassandra
Titus Master
Job Management &
Scheduler
S3
Zookeeper
Docker
Registry
EC2 Autocaling
API
Mesos Master
Titus UI
Fenzo
container
Pod & VPC network
drivers
containercontainer
AWS
metadata proxy
Integration
AWS VM’sCI/CD

20. Services more complex
● Services resize constantly and run forever
○ Autoscaling
○ Hard to upgrade underlying hosts
● Require IPC integration
○ Routable IPs, service discovery
○ Ready for traffic vs. just started/stopped
● Existing well defined dev, deploy, runtime & ops tools

21. Real networking is hard

22. Multi-tenant
Need IP per container - in VPC
Using security groups
Using IAM roles
Considering network resource isolation

23. Enabling VPC Networking
No IP, SecGrp A
Task 0
SecGrp Y,Z
Task 1 Task 2 Task 3
Titus EC2 Host VMeth1
ENI1
SecGrp=A
eth2
ENI2
SecGrp=X
eth3
ENI3
SecGrp=Y,Z
IP 1
IP 2
IP 3
pod root
veth<id>
app
SecGrp X
pod root
veth<id>
app
SecGrp X
pod root
veth<id>
appapp
veth<id>
Linux Policy Based
Routing + Traffic Control
Titus
EC2
Metadata
Proxy
169.254.169.254
IPTables NAT (*)
* **
169.254.169.254
Non-routable IP
*

24. Solutions
● VPC Networking driver
○ Supports ENI’s - full IP functionality
○ Scheduled security groups
○ Support traffic control (resource isolation)
● EC2 Metadata proxy
○ Adds container “node” identity
○ Delivers IAM roles

25. Reuse existing infrastructure services
VMVM
EC2
AWSAutoScaler
VMs
App
Cloud Platform
(metrics, IPC, health)
VPC
Netflix Cloud Infrastructure (VM’s + Containers)
Atlas Eureka Edda

26. Enable them for containers
VMVM
EC2
AWSAutoScaler
VMs
App
Cloud Platform
(metrics, IPC, health)
VPC
Netflix Cloud Infrastructure (VM’s + Containers)
VMVM
Atlas
TitusJobControl
Containers
App
Cloud Platform
(metrics, IPC, health)
Eureka Edda
VMVM
Batch
Containers

27. Spinnaker

28. Deploy based
on new images
tags

29. Basic resource
requirements
IAM Roles & Sec
Groups per
container
Deploy
Strategies
Same as VM’s

30. Easily see
health &
discovery

33. Secure Multi-tenancy
Common to VM’s and tiered security needed
● Protect the reduced host IAM role, Allow containers to have specific IAM
roles
● Needed to support same security groups in container networking as VM’s
User namespacing
● Docker 1.10 - Introduced User Namespaces
● Didn’t work /w shared networking NS
● Docker 1.11 - Fixed shared networking NS’s
● But, namespacing is per daemon, Not per container, as hoped
● Waiting on Linux
● Considering mass chmod / ZFS clones

34. Titus Advanced Scheduling
● Support for AZ balancing
● Multiple instance types selected based on workload
● Elastic underlying common resource pool
○ Bin packing managed transparently across all apps
● Hard and soft constraints
● Resource affinity and task affinity
● Capacity guarantees (critical tier)
34

35. Fenzo - Keep resource scheduling extensible
Fenzo - Extensible Scheduling Library
Features:
● Heterogeneous resources & tasks
● Autoscaling of mesos cluster
○ Multiple instance types
● Plugins based scheduling objectives
○ Bin packing, etc.
● Plugins based constraints evaluator
○ Resource affinity, task locality, etc.
● Scheduling actions visibility

36. Current Container Usage - Service
● Still small ~ 2000 long running containers
● NodeJS Device UI Scripts Apps
● Stream Processing Jobs - Flink
● Various Internal Dashboards

37. Questions?
Andrew Spyker (@aspyker) - Engineering Manager