Andrew Spyker
Senior Software Engineer for Netflix
Find more by Andrew Spyker: http://www.slideshare.net/aspyker
All Things Open
October 26-27, 2016
Raleigh, North Carolina
Building a Distributed & Automated Open Source Program at Netflix
1. Netflix Open Source
Andrew Spyker (@aspyker) - Engineering Manager
Building a distributed and
automated open source program
2. About Netflix
● 86.7M members
● A few thousand employees
● 190+ countries
● > ⅓ NA internet download traffic
● 500+ Microservices
● Many 10’s of thousands VM’s
● 3 regions across the world
3.
4. Trivia
Netflix been open
sourcing, since?
a) Around the start of streaming service - 2007
b) Around when we went international - 2010
c) Around House of Cards release time - 2013
6. Why does Netflix Open Source?
Improve Engineering
● Great feedback from wider community
● Collaborate through open code
Recruit new and retain engineering talent
● Hard problems are openly worked on
9. Open Source Functional Areas
● Contribute to Hadoop, Hive, Pig, Parquet, Presto, Spark
● Genie - RESTful API’s for Big Data Jobs
● Lipstick - Graphical depiction of executing Pig jobs
● Aegisthus - Data pipeline from Cassandra to Big Data
10. Open Source Functional Areas
● Nebula - Plugins for gradle to simplify builds
● Animator - Bakes AMI’s from OS installation packages
● Spinnaker - New continuous delivery platform
12. Open Source Functional Areas
● Photon - Java Interoperable File Format implementation
● VMAF - Perceptual quality metric algorithm and test toolkit
13. Open Source Functional Areas
● Raigad/Priam - Management/ops sidecars for ES and C*
● EVCache - Distributed, replicated memcache++
● Dynomite - Dynamo layer on top of non-dynamo data stores
14. Open Source Functional Areas
● Spectator/Atlas - Monitoring and Telemetry client and server
● Vector - Fine grained per instance performance monitoring
● Vizceral - Worldwide traffic to microservice graph
visualization
16. Open Source Functional Areas
● Work across front end technologies including Restify
● Falcor - Virtual JSON graph & optimized query to backends
● RxJS - Simplify Javascript async event based programming
17. Netflix’s approach to open source
Form a small cross-functional team working
group that centralizes OSS competence,
assisting decentralized teams working with OSS
spend less time focusing on the administrative
aspects (legal, tooling, branding, monitoring,
and community promotion).
18. Open source enabler - OSS Interest Group
● Internal mailing list
● Meets once per month
● Topics from developers
● Help each other with
common problems
21. Open Source Shepherds
● Management with business context
● Consistency across related projects
● Document how area fits together
● Focus on OSS health of each area
22. Common tools accelerate developers
● Security
● Backup
● Github user/group repo management
● Project tracking
● Build systems
● CI systems
23. Security tools
● We scan code for
○ Access keys, credentials, email
addresses, hostnames
● Provide tools and automation to
○ Scan before initial release
○ Scan repeatedly on github
24. Source code management
● Backup and archival
○ Github down != Netflix down
● Internal mirrors we could build from
25. Project Ownership
All projects have
● Development lead, Management lead
● Shepherd from OSS function area
Only projects with active leads stay active!
26. Github management
● Has to be easy
○ Otherwise, teams will go it alone
● Has to be automated
○ Self service - chat ops
○ Following secure best practices
27. Github user management
Support bring github id
● User links to internal id
● All tools then can
associate identity
Two Factor Auth Enforcement
● Automation to boot users who don’t
● Be careful - education on recovery!
28. Github group management
● Owners
○ Limited group - due to power
○ Automate via chatops all owner actions
● Netflixer group
○ Full write permissions on all repos
● Outside contributors
○ Added by netflixers, validated over time
32. ● Repeatable builds
● deb/rpm files for OS
package baking
● Reduces boilerplate for
common best practices
● Standards for
release/version mgmt
Common Build For Gradle/Java
nebula-plugins.github.io
33. Common CI Systems
● Travis CI
○ Populate .travis.yml and sh files
○ Standard targets for snapshots,
candidates, and releases
○ Binary upload credentials handled
○ Consistency across projects
● Cloudbees
○ Job-dsl to create release jobs
34. Using Docker to make projects easier
● A running image is worth a
thousand wiki documents
● Started with ZeroToDocker
○ Monolithic solution
○ Leveraged Dockerhub
trusted builds
35. Introducing TravisCI Docker builds
Function Dockerhub
trusted builds
TravisCI Docker
support
Github commit traceable builds ✔ ✔
Trusted build servers ✔ ✔
Full build control (labels, etc.) ✖ ✔
Easy to integrate with artifact releases ✖ ✔
● Experimenting: OSSTracker & Genie
● Docker compose used across images
36. TODO Group
● Joined 2015
● Collaborate on how
to better collaborate
● Leverage TODO group’s work
○ Github focus
○ Automation innovations
● Good group for helping OSS companies
37. Trivia
Which of the following
does Hystrix lead in?
a) Most PR’s closed d) Most Forks
b) Most Issues closed e) Most contributors
c) Most Stars
41. Recent NetflixOSS Releases
Vizceral
● React and Web Component
● Graph data to visualize traffic
Dynomite
● Dynamo layer on top of data stores
● Redis and memcache
● Manager (config, multi-region, backup)