SlideShare ist ein Scribd-Unternehmen logo
1 von 61
Containers at Netflix - An
Evolving Story
Sangeeta Narayanan
Engineering Manager @Netflix
@sangeetan
Going Global
Recent Launches: Italy, Spain, Portugal
Early 2016: Korea, Hong Kong, Taiwan, Singapore
Growing BIGGER
Winning Moments of Truth
Getting BETTER
Jan
Apr
Jul
O
ct
2015
Jan
Apr
Jul
O
ct
2015
Jan
Apr
Jul
O
ct
2015
A/B Tests
New features
Jan
Apr
Jul
O
ct
2015
A/B Tests
New features
Device Launches
Going FASTER
Cloud Native Architecture
Automation & Tooling
Culture
New Considerations
Diversity - Users, Tech stacks, Workloads
Dev Ex - Dev iterations, Debugging, Packaging
Rapid Delivery
Resource Sharing
Dev-centric packaging &
reproducibility
Infrastructure abstraction
Faster dev iterations
Resource sharing
Why Docker?
Docker @Netflix - Use Cases
Algorithms Engineering
Data Pipeline
Edge Engineering
Docker @Netflix - Use Cases
Algorithms Engineering
Data Pipeline
Edge Engineering
Algorithms Engineering
R&D of algorithmic components for product
Precompute
Online
Services
Model Trainer
New data available
Infrastructure abstraction
Polyglot support & Local Dev Ex
Dynamic resource allocation &
distributed job scheduling
Requirements
Model Trainer Online
Services
Precompute
New data available
Distributed scheduler
+
Distributed Scheduling & Task Management
Model Trainer Spinnaker Other Apps
Fenzo
Infrastructure abstraction
Polyglot + Local Dev
Distributed Scheduling
Num of Containers
Docker @Netflix - Use Cases
Algorithms Engineering
Data Pipeline
Edge Engineering
Data Pipeline
Publish, Collect, Process, Move Data
Local Dev Ex
Infrastructure abstraction
Fine-grained resource management
Requirements
Job
Infrastructure abstraction
Local Dev Ex
Fine grained resource Management
Num of Containers

(~10K steady state)
Docker @Netflix - Use Cases
Algorithms Engineering
Data Pipeline
Edge Engineering
Edge Engineering
Entry point to Netflix for all device traffic
Automation & Tools
Automation & Tools
Node.js
Html,
JS
Infrastructure abstraction
Local dev, Polyglot - Java, Groovy,
Node.js
Fine-grained resource
management
Requirements
Ops Tooling
Networking, IPC
Security
Insights
Failure Testing
Performance
And more…
Requirements++
Long running Services
vs.
Batch Jobs
Ops Tooling
Networking, IPC
Security
Insights
Failure Testing
Performance
And more…
Requirements++
Long running Services
vs.
Batch Jobs
Local Dev
Test/
Debug
Open PR
CI
Build/Test
AMI
Bakery
Deploy
Spinnaker
Dist Job
Sched
Gradle docker plugin to generate Dockerfile
Spinnaker CloudProvider provides abstraction
Bakery support for image generation
Docker registry for internal images
libnetwork plugin for addressable IP
Building out the pieces
Dev Ex Wins
Faster dev iterations
Run prod image locally; attach debugger
Multiple branch/config variations
Hot reloading of files
‘Roman Riding’ with AMIs
ONNot There Yet!
‘PaaS-like' Dev Experience
Production scale/Operational readiness
Optimized resource utilization
Infrastructure Abstraction
Dev Ex
Production ready Online Services
Polyglot env, Packaging
YMMV
Thank You
@sangeetan
The Netflix Algorithms Engineering & Data Pipeline
teams
Acknowledgements
Photo Credits
https://goo.gl/sP5T4X

https://goo.gl/JaNlyf

https://goo.gl/rZXws1

http://goo.gl/V4UFmR

https://goo.gl/R08GPQ

http://tastyk.deviantart.com/art/Evolution-149719618

https://goo.gl/W2UKv0

https://goo.gl/fRcpPN

https://goo.gl/tF199H

https://goo.gl/K1OH9T wildflowers

https://goo.gl/KSaL5s roadmap

https://goo.gl/8JCDQy odometer

https://goo.gl/c4RuSR takeaway box

Weitere ähnliche Inhalte

Was ist angesagt?

Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automation
Puppet
 

Was ist angesagt? (20)

Getting Started with IT Service Intelligence
Getting Started with IT Service IntelligenceGetting Started with IT Service Intelligence
Getting Started with IT Service Intelligence
 
КОСТЯНТИН СЕВЕРЕНЧУК «Monitoring and Automation in DevTestSecOps world» Lviv ...
КОСТЯНТИН СЕВЕРЕНЧУК «Monitoring and Automation in DevTestSecOps world» Lviv ...КОСТЯНТИН СЕВЕРЕНЧУК «Monitoring and Automation in DevTestSecOps world» Lviv ...
КОСТЯНТИН СЕВЕРЕНЧУК «Monitoring and Automation in DevTestSecOps world» Lviv ...
 
Cloud-native Patterns (July 4th, 2019)
Cloud-native Patterns (July 4th, 2019)Cloud-native Patterns (July 4th, 2019)
Cloud-native Patterns (July 4th, 2019)
 
Enforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automationEnforce compliance policy with model-driven automation
Enforce compliance policy with model-driven automation
 
ACA-Mobile - Creating Enterprise Apps with MADP
ACA-Mobile - Creating Enterprise Apps with MADPACA-Mobile - Creating Enterprise Apps with MADP
ACA-Mobile - Creating Enterprise Apps with MADP
 
Integration Monday - Logic Apps: Development Experiences
Integration Monday - Logic Apps: Development ExperiencesIntegration Monday - Logic Apps: Development Experiences
Integration Monday - Logic Apps: Development Experiences
 
The Cloud journey in an Enterprise - Delivery of Things World - Berlin April ...
The Cloud journey in an Enterprise - Delivery of Things World - Berlin April ...The Cloud journey in an Enterprise - Delivery of Things World - Berlin April ...
The Cloud journey in an Enterprise - Delivery of Things World - Berlin April ...
 
The Cloud Journey in an Enterprise - IDC Multicloud - Stockholm November 20, ...
The Cloud Journey in an Enterprise - IDC Multicloud - Stockholm November 20, ...The Cloud Journey in an Enterprise - IDC Multicloud - Stockholm November 20, ...
The Cloud Journey in an Enterprise - IDC Multicloud - Stockholm November 20, ...
 
Cloud @ Scania - Södertälje Science Week
Cloud @ Scania - Södertälje Science WeekCloud @ Scania - Södertälje Science Week
Cloud @ Scania - Södertälje Science Week
 
Avoiding the DevOps Tax
Avoiding the DevOps Tax Avoiding the DevOps Tax
Avoiding the DevOps Tax
 
Scania's DevSecOps approach - Gamifying Security - auto:CODE
Scania's DevSecOps approach - Gamifying Security - auto:CODEScania's DevSecOps approach - Gamifying Security - auto:CODE
Scania's DevSecOps approach - Gamifying Security - auto:CODE
 
DevOps in Azure : Puppetize the Cloud
DevOps in Azure : Puppetize the CloudDevOps in Azure : Puppetize the Cloud
DevOps in Azure : Puppetize the Cloud
 
DevOps and Splunk
DevOps and SplunkDevOps and Splunk
DevOps and Splunk
 
Humana digitally transforming health and well-being with Pivotal cloud foundr...
Humana digitally transforming health and well-being with Pivotal cloud foundr...Humana digitally transforming health and well-being with Pivotal cloud foundr...
Humana digitally transforming health and well-being with Pivotal cloud foundr...
 
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
DOES16 San Francisco - Marc Ng - SAP’s DevOps Journey: From Building an App t...
 
AWS18 Startup Day Toronto- Launching your Application the Amazon Way
AWS18 Startup Day Toronto- Launching your Application the Amazon WayAWS18 Startup Day Toronto- Launching your Application the Amazon Way
AWS18 Startup Day Toronto- Launching your Application the Amazon Way
 
Continuous Integration @ Haptik
Continuous Integration @ HaptikContinuous Integration @ Haptik
Continuous Integration @ Haptik
 
AWS re:Invent "The secrets to building and delivering amazing apps at scale"
AWS re:Invent "The secrets to building and delivering amazing apps at scale"AWS re:Invent "The secrets to building and delivering amazing apps at scale"
AWS re:Invent "The secrets to building and delivering amazing apps at scale"
 
Advanced Automation in Your API Lifecycle
Advanced Automation in Your API Lifecycle Advanced Automation in Your API Lifecycle
Advanced Automation in Your API Lifecycle
 
PayPal's History of Microservices Architecture
PayPal's History of Microservices ArchitecturePayPal's History of Microservices Architecture
PayPal's History of Microservices Architecture
 

Ähnlich wie Containers at Netflx - An Evolving Story QConSF2015

Vighnesh_Naik_Resume_DevOps
Vighnesh_Naik_Resume_DevOpsVighnesh_Naik_Resume_DevOps
Vighnesh_Naik_Resume_DevOps
VIGHNESH NAIK
 
Big(ger) Data in Software Engineering
Big(ger) Data in Software EngineeringBig(ger) Data in Software Engineering
Big(ger) Data in Software Engineering
Mehdi Mirakhorli
 
Zheng Ma Resume
Zheng Ma ResumeZheng Ma Resume
Zheng Ma Resume
Zheng Ma
 
The Environment for Innovation: Tristan Goode, Aptira
The Environment for Innovation: Tristan Goode, AptiraThe Environment for Innovation: Tristan Goode, Aptira
The Environment for Innovation: Tristan Goode, Aptira
OpenStack
 

Ähnlich wie Containers at Netflx - An Evolving Story QConSF2015 (20)

Vighnesh_Naik_Resume_DevOps
Vighnesh_Naik_Resume_DevOpsVighnesh_Naik_Resume_DevOps
Vighnesh_Naik_Resume_DevOps
 
Zakir_Hussain_cv
Zakir_Hussain_cvZakir_Hussain_cv
Zakir_Hussain_cv
 
Data analytics at a petabyte scale final
Data analytics at a petabyte scale   finalData analytics at a petabyte scale   final
Data analytics at a petabyte scale final
 
Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017
Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017
Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017
 
AnilKumarT_Resume_latest
AnilKumarT_Resume_latestAnilKumarT_Resume_latest
AnilKumarT_Resume_latest
 
Taylor Wicksell and Tom Gianos at SpringOne Platform 2019
Taylor Wicksell and Tom Gianos at SpringOne Platform 2019Taylor Wicksell and Tom Gianos at SpringOne Platform 2019
Taylor Wicksell and Tom Gianos at SpringOne Platform 2019
 
Platform Requirements for CI/CD Success—and the Enterprises Leading the Way
Platform Requirements for CI/CD Success—and the Enterprises Leading the WayPlatform Requirements for CI/CD Success—and the Enterprises Leading the Way
Platform Requirements for CI/CD Success—and the Enterprises Leading the Way
 
DEVNET-1125 Partner Case Study - “Project Hybrid Engineer”
DEVNET-1125	Partner Case Study - “Project Hybrid Engineer”DEVNET-1125	Partner Case Study - “Project Hybrid Engineer”
DEVNET-1125 Partner Case Study - “Project Hybrid Engineer”
 
Big(ger) Data in Software Engineering
Big(ger) Data in Software EngineeringBig(ger) Data in Software Engineering
Big(ger) Data in Software Engineering
 
Space Codesign at TandemLaunch 20150414
Space Codesign at TandemLaunch 20150414Space Codesign at TandemLaunch 20150414
Space Codesign at TandemLaunch 20150414
 
Space Codesign at TandemLaunch Lunch & Learn 20150414
Space Codesign at TandemLaunch Lunch & Learn 20150414Space Codesign at TandemLaunch Lunch & Learn 20150414
Space Codesign at TandemLaunch Lunch & Learn 20150414
 
Accelerating Digital Transformation: It's About Digital Enablement
Accelerating Digital Transformation:  It's About Digital EnablementAccelerating Digital Transformation:  It's About Digital Enablement
Accelerating Digital Transformation: It's About Digital Enablement
 
Measuring slack api_performance_using_druid
Measuring slack api_performance_using_druidMeasuring slack api_performance_using_druid
Measuring slack api_performance_using_druid
 
Luke Young Rhee Resume
Luke Young Rhee ResumeLuke Young Rhee Resume
Luke Young Rhee Resume
 
Zheng Ma Resume
Zheng Ma ResumeZheng Ma Resume
Zheng Ma Resume
 
The Environment for Innovation: Tristan Goode, Aptira
The Environment for Innovation: Tristan Goode, AptiraThe Environment for Innovation: Tristan Goode, Aptira
The Environment for Innovation: Tristan Goode, Aptira
 
Microsoft and Revolution Analytics -- what's the add-value? 20150629
Microsoft and Revolution Analytics -- what's the add-value? 20150629Microsoft and Revolution Analytics -- what's the add-value? 20150629
Microsoft and Revolution Analytics -- what's the add-value? 20150629
 
mjpaguio-resume
mjpaguio-resumemjpaguio-resume
mjpaguio-resume
 
Innovation in the Enterprise Rent-A-Car Data Warehouse
Innovation in the Enterprise Rent-A-Car Data WarehouseInnovation in the Enterprise Rent-A-Car Data Warehouse
Innovation in the Enterprise Rent-A-Car Data Warehouse
 
Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models
Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning ModelsApache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models
Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models
 

Mehr von Sangeeta Narayanan

Mehr von Sangeeta Narayanan (7)

Innovating with DevEx - Developer Relations Conference March 2019
Innovating with DevEx - Developer Relations Conference March 2019Innovating with DevEx - Developer Relations Conference March 2019
Innovating with DevEx - Developer Relations Conference March 2019
 
Netflix Women Living on the "Edge" - WiT event
Netflix Women Living on the "Edge" - WiT eventNetflix Women Living on the "Edge" - WiT event
Netflix Women Living on the "Edge" - WiT event
 
Making Microservices work at Netflix
Making Microservices  work at NetflixMaking Microservices  work at Netflix
Making Microservices work at Netflix
 
Engineering Manager, Edge Insights @Netflix
Engineering Manager, Edge Insights @NetflixEngineering Manager, Edge Insights @Netflix
Engineering Manager, Edge Insights @Netflix
 
Operational Visibility at Global Scale
Operational Visibility at Global ScaleOperational Visibility at Global Scale
Operational Visibility at Global Scale
 
Building and Maintaining your Feedback Muscle
Building and Maintaining your Feedback MuscleBuilding and Maintaining your Feedback Muscle
Building and Maintaining your Feedback Muscle
 
Continuous delivery in a cloud soa
Continuous delivery in a cloud soaContinuous delivery in a cloud soa
Continuous delivery in a cloud soa
 

Kürzlich hochgeladen

%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
masabamasaba
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
VictoriaMetrics
 
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
masabamasaba
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Health
 

Kürzlich hochgeladen (20)

AI & Machine Learning Presentation Template
AI & Machine Learning Presentation TemplateAI & Machine Learning Presentation Template
AI & Machine Learning Presentation Template
 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
 
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdfPayment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
 
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
 
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...
 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the past
 
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
 
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
 
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park %in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
 
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park %in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
 
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 
%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare%in Harare+277-882-255-28 abortion pills for sale in Harare
%in Harare+277-882-255-28 abortion pills for sale in Harare
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
 

Containers at Netflx - An Evolving Story QConSF2015