Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lLZ7M8. Sayli Karmarkar discusses Winston, a monitoring and remediation platform built for Netflix engineers. Winston lets engineers go beyond the current DevOps model of monitoring the system and getting alerted when a failure condition occurs. He talks about the types of use cases that this type of platform targets, adoption challenges and how the architecture of the platform evolved over time. Filmed at qconsf.com. Sayli Karmarkar is a software engineer with 8+ years of experience in developing platforms and services to configure and manage thousands of systems in an enterprise. At Netflix, she has been focusing on building platforms to help developers in diagnosing and remediating their operational failures automatically in response to events.

1. Winston: Helping Netflix Engineers Sleep at Night
Our journey… assisting engineers reduce operational load and MTTR

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-winston

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. On-Call !

5. Sayli Karmarkar
Senior Software Engineer
Diagnostics and Remediation Engineering (DaRE)
skarmarkar@netflix.com
@HikerTechy
https://www.linkedin.com/in/saylikarmarkar
DaRE Team’s Focus
Build platforms, tools and libraries
to help teams reduce MTTR for operational issues.

6. Engineer
Wakes up
PagerDuty
Alert
2:02 AM
Logs in
and ACK
2:07 AM
Checks
runbook
2:15 AM
Studies
the alert
2:10 AM
Fixes/Mitigates
the
problem
2:30 AM
Runs
diagnostics
2:20 AM2:00 AM
Traditional On-Call Timeline

7. Works, but does it scale?!
Scale and Growth Availability

9. MTTR
Productivity
Traditional On-Call Pain Points

10. Solution?
Automate
● Removing False Positives
● Collecting Diagnostic Information
● Mitigating the problem to reduce impact on
the customers
Hands-free
Feed the runbooks to an event-driven
automation platform and have them executed in
response to operational events

11. Unique Problem? Not really ..

12. Define
● Business Goals
● Use-cases
● Customers
● Constraints
● Interactions with other services

13. Winston’s Goals
● Assist engineers in reducing MTTR and pager fatigue by
providing a platform to automate their runbooks
● Provide an easy way to connect automated runbooks to an
event
● Let engineers focus on the business logic of runbooks rather
than infrastructure aka PaaS.
● Provide appropriate wrappers and libraries to interact with other
services
● Ensure best practices for automations and runbook lifecycle
management

14. What is Winston?
Winston is an event driven runbook automation
platform. It is designed to host and execute runbooks in
response to operational events.

15. Engineer
Wakes up
Logs in
and ACK
Checks
runbook
Studies
the alert
Fixes the
problem
Runs
diagnostics
PagerDuty
Alert
2:02 AM 2:07 AM 2:15 AM2:10 AM 2:30 AM2:20 AM2:00 AM
Traditional On-Call Timeline

16. False
Positive
Winston
2:00 AM
2:05 AM
2:05 AM
2:15 AM
Assisted
Diagnostics
Mitigates the
problem
On-Call With Winston

17. Evaluation - Build / Reuse / Buy

18. Stackstorm
+
● A generic pluggable Event-Driven Automation Platform
● Designed with availability and reliability in mind
● Open source + Code following good design practices
● Good alignment with respect to goals and future direction
-
● High availability and reliability not exercised a lot
● Dependency on MongoDB and RabbitMQ
● No easy way of adding and updating automation

19. +
Inbound Integrations
(through SQS)
+
Outbound Integrations
Bolt ...
As a Service (High Availability and Reliability )
Good Starting Point ..
Iterate and Evaluate Regularly

20. A closer look at a Winston Instance

21. V1.0 Winston HA Deployment

22. Challenges
● Added cognitive load resulting in less adoption
● How to help engineers choose operational
efficiency over new features?
● Recommended and safe automation and lifecycle
management practices are often not followed
● Simple use-cases are not trivial to on-board

23. ● One stop portal for all things Winston
● Supports Create, Read, Update, Delete, Execute and Diagnose
functionality
● Implements best practises
○ Compliance/Auditing
○ Persistence
○ Security (Authentication/Authorization)
● Self serve & scalable
Winston Studio

24. Winston Studio

25. Runbook
View

26. Executions

27. Execution
Details

28. Current Winston Deployment

29. Use-case Patterns

30. Sample Use-cases
False Positives
● Broker reporting offline when AWS maintenance takes
down an instance
● Cassandra ring health
Diagnostics - Correlation could point towards the root cause
● Checking current maintenance jobs running on a cluster
when an issue occurs
● Querying dependencies upstream and downstream for
anomalous behavior
● Capture current system state and logs to analyze
failures and reach the root cause quicker
Mitigation
● Restart kafka process
● Clean up disk space

34. The Road Ahead
● Adoption / Usability
○ Find common operational use-cases and allow them to be re-used
○ Improve discoverability of Winston by integrating into existing alerting
systems
○ Polyglot support (Groovy based runbooks)
● Safety
○ Resource isolation using containers
○ Rate limiting capability
● Stronger analytics
○ Provide aggregate visualization of runbook executions

35. Key Takeaways
● Don’t re-invent the wheel
● Start simple and iterate. Have some room for experimentation.
● Start with use-cases where there is more pain and less control
over the source of the problem
● Pay special attention to usability of your product
● Push for changing the culture -- Usage will follow
● Find sponsors for features that are much more involved
● Implement best practices through carefully designed user interface
instead of documentation.
● Discourage anti-patterns that focus on long term mitigation rather
than fixing the root-cause
● Talk to us and others to share insights and learnings

36. Resources
● Winston Tech Blog link
http://techblog.netflix.com/2016/08/introducing-winston-event-driven.html
● Stackstorm documentation
https://docs.stackstorm.com/
● Reach out
skarmarkar@netflix.com
@HikerTechy
We are hiring
Senior Software Engineer - https://jobs.netflix.com/jobs/860752

37. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-winston