One of the key challenges in developing a service-oriented architecture (SOA) is anticipating traffic patterns and scaling the number of running instances of services to meet demand. In many situations, it’s hard to know how much traffic a service will receive and when that traffic will come. A service may see no requests for several days in a row and then suddenly see thousands of requests per second. If developers underestimate peak traffic, their service can quickly become overwhelmed and unresponsive, and may even crash, resulting in constant human intervention and poor developer productivity. On the other hand, if they provision sufficient capacity upfront, the resources they allocate will be completely wasted when there’s no traffic. In order to allow for better resource utilization, many cluster management platforms provide auto-scaling features. These features tend to auto-scale at the machine/resource level (as opposed to the request level) or by deferring to logic in the application layer. A more optimal approach would be to run services when–and only when–there is traffic. Waiter is a distributed auto-scaler that delivers this optimal type of request-level auto-scaling. It requires no input or handling from applications and is agnostic to underlying cluster managers; it currently uses Mesos, but can easily run on top of Kubernetes or other solutions. Another challenge with SOAs is enabling the evolution of service implementations without breaking downstream customers. On this front, Waiter supports service-versioning for downstream consumers by running multiple, individually-addressable versions of services. It automatically manages service lifecycles and reaps older versions after a period of inactivity. With a variety of unique features, Waiter is a compelling platform for applications across a broad range of industries. Existing web services can run on Waiter without modification as long as they communicate over HTTP and support the transmission of client requests to arbitrary backends. Two Sigma has employed the platform in a variety of critical production contexts for over two years, with use cases rising to hundreds of millions of requests per day.

1. Waiter
https://github.com/twosigma/waiter
Shams Imam
Software Engineer, Two Sigma Investments, LP
Two Sigma Open Source Meetup
November 13th, 2017

2. • Motivation?
• Waiter
2
Agenda

3. 3
A typical architecture
Client(s)
Load Balancer
Backend Backend Backend
Request

4. 4
Problem 1: Dead Backend
Client(s)
Load Balancer
AWOL Backend Backend
Request

5. 5
Problem 2 : Load Imbalance
Client(s)
Load Balancer
Backend Backend Backend
Request

6. 6
Problem 3a: Too Many Requests
Client(s)
Load Balancer
Backend Backend Backend
Request

7. 7
Problem 3b: Manual Scaling
Client(s)
Load Balancer
Backend
Request
Backend Backend Backend Backend Backend

8. 8
Problem 3c: Wasting Resources
Client(s)
Load Balancer
Backend
Request
Backend Backend Backend Backend Backend

9. 9
Problem 4: Version Mismatch
RequestClient
Version f8d2
Client
Version c8b5
Backend
Version f8d2

10. • Service platform that
– Runs and auto-scales your services
• Parameterized services
• Usage metrics
• Authentication
– Automatic management of resources
– Provides resiliency when instances fail
– Easy version management
10
What is Waiter?

11. 11
Architecture – MITM HTTP Proxy
Client(s)
WAITER
Backend Backend Backend
Request

12. 12
Architecture – Distributed version
Client(s)
Service-1 Service-1 Service-1
Request
WAITER WAITER WAITER
Service-2 Service-2

13. • We manage both open source and internal versions
of the products
– Internal version relies on plug-in mechanism
• We rely heavily on automated tests to help prevent
regression
– Travis (open source version)
– Jenkins (internal version)
13
Workflow

14. • Leverage the wisdom of crowds to evolve the
platform and integrate the platform with new
technologies we cannot foresee yet
• By open sourcing, we open the door for many other
opportunities
– Waiter UI being built by a collaborator
14
Why open source

15. 15
(loop []
(when-let [comment (or (audience/question)
(audience/feedback))]
(respond-to comment)
(recur))