Lyft open sources several infrastructure projects including Confidant for securely storing secrets, Discovery for service registration and lookup, Ratelimit for rate limiting requests, and Envoy as an edge and internal proxy. Envoy handles all service to service communication at Lyft and provides observability, load balancing, and integration with other services. These projects help Lyft build and operate a microservices architecture at large scale.
7. Behind the scenes
Application
IAM Role
EC2 Instance
Credential
api_key: password123
api_key = os.getenv('CREDENTIAL_API_KEY')
KMS
DynamoDB
Confidant
8. Server-blind secrets
Highly sensitive secrets are encrypted and decrypted by the end-users.
Confidant stores but can't read them.
Confidant
KMS
IAM Role
EC2 Instance
9. lyft / discovery
Provides a REST interface for querying for the list of hosts that belong to a microservices
54
6 contributors
Python
August 2016
11. - Hosts are stored in DynamoDB
- Storage support is abstract
- Hosts removed if not reporting since now - HOST_TTL
- Ecosystem designed to tolerate eventual consistency
unlike Zookeeper, etcd, Consul
- Pair with active healthchecks
Storage
DynamoDB
13. Services list the hosts they want to talk to!
internal_hosts:
- jobscheduler
- roads
external_hosts:
- dynamodb_iad
- kinesis_iad
Envoy per-service configuration
location-service/envoy.yaml
/etc/envoy.conf
(on the box)
15. lyft / ratelimit
Go/gRPC service designed to enable generic rate limit scenarios
224
6 contributors
Go
January 2017
16. Why rate limit?
- Control flow
- Protect against attacks
- Bad actors
- Accidents happen
oops!
17. Rate Limit Service
- Written in Go
- Enable generic rate limit
scenarios
- Decisions based on a domain
and set of descriptors
- Settings configured at runtime
- Backed by Redis
Ratelimit
?
INCR
18. Domains and descriptors
Domain
Defines a container for a set of rate limits
Globally unique
e.g. "envoy_front"
Descriptors
Ordered list of key/value pairs
Case sensitive
e.g. ("destination_cluster", "location-service"), ("user_id", "1234")
23. Why Envoy?
Service Oriented Architecture
- Many languages and frameworks
- Protocols (HTTP/1, HTTP/2, databases, caching, etc…)
- Partial implementation of SoA best practices (retries, timeouts, …)
- Observability
- Load balancers (AWS, F5)
24. What is Envoy?
The network should be transparent to applications.
When network and application problems do occur it
should be easy to determine the source of the problem.
25. What is Envoy?
- Modern C++11
- Runs alongside applications
- Service discovery integration
- Rate Limit integration
- HTTP2 first (get gRPC!)
- Act as front/edge proxy
- Stats, Stats, Stats
- Logging
29. Envoy deployment @Lyft
- > 100 services
- > 10,000 hosts
- > 2,000,000 RPS
- All service to service traffic (REST and gRPC)
- MongoDB, DynamoDB, Redis proxy
- External service proxy (AWS and other partners)
- Kibana/Elastic Search for logging.
- LightStep for tracing
- Wavefront for stats
31. Done!
- Lyft is hiring. If you want to work on large-scale problems in a fast-moving,
high-growth company visit lyft.com/jobs
- Visit github.com/lyft
- Slides available at slideshare.net/danielhochman
- Q&A