RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agility - AWS re: Invent 2012

Rainmakers
How Netflix Operates Clouds for Maximum Freedom and Agility

Jeremy Edberg
Reliability Architect, Netflix

Do you have...

• A release Engineer?
• A QA department?
• Chef or Puppet to
manage your systems?

Tweet @jedberg with feedback!

Do you have...

• Upwards of 100 releases a
day?


With more than 30 million streaming
members in the United States, Canada,
Latin America, the United Kingdom, Ireland
and the Nordics, Netflix is the world's
leading internet subscription service for
enjoying movies and TV programs
streamed over the internet to PCs, Macs
and TV.
Source: http://ir.netflix.com


The Netflix Way
• Everything is “built for three”
• Fully automated build tools to test
and make packages
• Fully automated machine image
bakery
• Fully automated image deployment
• Independent teams responsible for
both Dev and Ops

Philosophy


Automate all the things!


Automate all the things!

• Application startup
• Configuration
• Code deployment
• System deployment


Automation

• Standard base image
• Tools to manage all the
systems
• Automated code deployment


Shared state should be
stored in a shared
service

Data on an instance
should be replicated to
other instances

“Build for Three”
We hold a boot camp for new engineers to teach them
how to build for a highly distributed environment.


Netflix on AWS
2012 2012 2012
IPv6 IPv6 IPv6

Open Connect


Highly aligned, loosely
coupled
• Services are built by different
teams who work together to figure
out what each service will provide.
• The service owner publishes an
API that anyone can use.


Advantages to a Service
Oriented Architecture
• Easier auto-scaling
• Easier capacity planning
• Identify problematic code-paths more
easily
• Narrow in the effects of a change
• More efficient local caching


Freedom and Responsibility

• Developers deploy when they
want
• They also manage their own
capacity and autoscaling
• And fix anything that breaks at
4am!


All systems choices assume
some part will fail at some
point.


The Monkey Theory
• Simulate
things that go
wrong
• Find things
that are
different

Execution

Photo from I, Robot, copyright 20th Century Fox

Netflix built a global PaaS

• Service Oriented
Architecture
• HTTP/Rest interfaces
between services


Netflix PaaS features
• Supports all regions and zones
• Multiple accounts
• Cross region/account replication
• Internationalized, localized and GeoIP
routed
• Advanced key management
• Autoscaling with 1000s of instances
• Monitoring and alerting on millions of
metrics

What AWS Provides
• Instances
• Machine Images
• Elastic IPs
• Load Balancers
• Security groups / Autoscaling groups
• Availability zones and regions


Linux Base AMI (CentOS or
Ubuntu)

Optional
Java (JDK 6 or 7)
Apache
Appdynamics
App Agent
Monitoring
monitoring
Tomcat
Log Rotation
to S3 Application war file, base
Healthcheck, status
GC and servlet, platform,
servelets, JMX interface,
Appdynamics thread interface jars for
Servo autoscale
Machine dump dependent services
Agent logging


The Netflix Platform
Discovery
Circut Breakers
(Eureka)Entrypoints
(Hystrix)
(Edda)Configuration
Cassandra (Priam &
(Archaius)
Astyanax &
Zookeeper (Exhibitor)
CassJMeter) Cryptex
logging (Blitz4j & Honu)
AKMSEvCache
NIWS
Proxiesi18n
Geo
L10n
Base Open Source

Open Source at Netflix

Governator
Blitz4j
Edda
Tweet @jedberg with feedback! Hystrix

Finding things
• Discovery (Eureka)
• Application to instance mapping
• Heartbeat to keep track of health
• Entrypoints (Edda)
• Local database of AWS resources
• NIWS (Netflix Internal Web Service)
• On instance software load balancer
• Handles retry logic
• Geo (Geolocation library)
• Provides IP to Lat/Lon mapping for any service that
needs it.


Entrypoints (Edda)
• REST API
• GET /REST/v2/instance/$id
• Keeps track of all resources
• Autoscaling groups, EIPs,
Instances, Applications, Clusters,
History


Entrypoints Exploration
Find all active GET /REST/v2/view/instances
instances

Find all instances in a GET /REST/v2/group/clusters
cluster

/v2/aws/autoScalingGroups/edda-
Show only ASG name, v123;_pp:(autoScalingGroupName,instances:(
instance ID and health instanceId,lifecycleState))

Which ASG contains a /v2/aws/autoScalingGroups;instances.instanceId=i
-96f3ca3a
particular instance?


Keeping it all Straight
• Configuration (Archaius)
• Global variables (Fast properties)
• Base
• Base system. Prod vs. Test, etc
• Zookeeper (Curator)
• Locks, other similar coordination
• Logging (Blitz4j and Honu)
• Keep track of what happened and store it
for post analysis.

Keeping it Secure
• Cryptex
• Service for key management
• High, medium and low value keys
• AKMS (Amazon Key Management System)
• Hands out keys to instances (and dev boxes)
so they don’t have to store the key on the
instance

Tweet @jedberg with feedback! For more info, see SEC201: Security Panel

Storing it
• Cassandra (Priam, astyanax)
• Configure and access Cassandra
• Provide OO abstractions handle
connection pooling, discovery of
hosts
• EVCache (Eccentric Volatile Cache)
• Wrapper for memcached to handle
zone awareness and replication
• Proxies
• Get data out of the datacenter and
into the cloud.

Data
What do we do with it all?


We store it!

• Cache
(memcached)
• Cassandra
• RDS (MySql)

Cassandra


Why Cassandra?
• Availability over
consistency
• Writes over reads
• We know Java
• Open source + support

Using Cassandra at Netflix
• Priam
• Zero touch auto-config
• State management
• Token assignment
• Node replacement
• Backup/restore to/from S3
• Astyanax
• OO abstraction to
Cassandra
• Multi-region support


Cassandra Architecture


Cassandra Architecture

Tweet @jedberg with feedback! For more info, see DAT202: Optimizing your Cassandra Database on AWS

Tools
• Asgard
• AWS usage
• Atlas
• Chronos
• Build system
• Explorers (Cassandra and SimpleDB)


Elastic Load
Balancer
Auto Scaling
Group

Security
Instances
Group

Launch
Configuration

Amazon Machine
Tweet @jedberg with feedback! Image

api-frontend

api-usprod-v007 api-usprod-v008


Netflix has moved the
granularity from the
instance to the cluster


Why Bake?
Traditional:
•launch OS
•install Generic AMI
Instance
packages
•install app

Netflix:
•launch OS+app
App AMI Instance


Getting Baked
Artifactory
app bundles
Ivy
snapshot / release
libraries
libraries / apps

Jenkins
resolve test publish

sync compile build report
source

Perforce / Git Ant targets Groovy all over


Base
Image S3 / EBS

Baking foundation
AMI
Linux: CentOS, Fedora, Ubuntu
base
AMI
mount snapshot

Ready
for
Yum / Apt app
install Bakery bake
AWS
RPMs: Apache, Java...

ec2 slave instances

App Image
Baking S3 / EBS

base AMI
Linux, Apache, Java, Tomcat

app
AMI
mount snapshot

Jenkins / Yum / Ready
Artifactory to launch!
install Bakery
AWS
app bundle

ec2 slave instances

Ubuntu)

Optional
Java (JDK 6 or 7)
Apache
Appdynamics
App Agent
Monitoring
monitoring
JBoss
Log Rotation
to S3 Application war file, base
Healthcheck, status
GC and servlet, platform,
servelets, JMX interface,
Appdynamics thread interface jars for
Servo autoscale
Machine dump dependent services
Agent logging


Ubuntu)

Optional
Python
Apache

monitoring
Monitoring Django
Log Rotation
to S3 Application file, base
server, platform,
Appdynamics interface libs for
logging
Machine dependent services
Agent


•
The simian army
Chaos -- Kills random instances

• Chaos Gorilla -- Kills zones

• Chaos Kong -- Kills regions

• Latency -- Degrades network and injects faults

• Conformity -- Looks for outliers

• Circus -- Kills and launches instances to maintain zone
balance

• Doctor -- Fixes unhealthy resources

• Janitor -- Cleans up unused resources

• Howler -- Yells about bad things like Amazon limit
violations

• Security -- Finds security issues and expiring certificates
Tweet @jedberg with feedback! For more info, see ARC301: Intro to Chaos Monkey & the Simian Army

What’s going on?!


Atlas


{
"clusters": [
"epic_aggregator",
"epic_aggregator-dev"
], {
"alerts": [ "metricName": "EpicPlugin_MetricCount",
// you can use javascript style comments in the config "applyTo": "instance",
{ "description": "${instanceId} is reporting too many metrics",
"metricName": "EpicPlugin_NumDropped", "condition": {
"applyTo": "cluster", "type": "NumOccurrences",
"condition": { "num": 4,
"type": "StaticThreshold", "condition": {
"max": 0.0 "type": "StaticThreshold",
}, "max": 0.0
"severity": "major", }
"description": "plugin is dropping metrics" },
}, "additionalDetails": {
{ "statusUrl": "http://${publicDnsName}:7001/Status",
"metricName": "EpicPlugin_NumDropped_Instance", "nacClusterUrl": "nac${env}/${region}/cluster/show/${cluster}"
"applyTo": "instance", }
"condition": { "overrides": {
"type": "NumOccurrences", "subject": "${instanceId} is reporting too many metrics",
"num": 4, "incident_key": "${metricName}:${instanceId}",
"condition": { "service_key_override": "12345",
"type": "StaticThreshold", "email_override": "devnull@netflix.com"
"max": 0.0 },
} "severity": "minor"
}, }
"overrides": { ]
"service_key_override": "12345", }
"require_instance_status_not_in: ["DOWN", "OUT_OF_SERVICE"],
"email_override": "devnull@netflix.com"
},
"severity": "minor"
},

Example Alert Config


Alert Tuning


Alert Systems
CORE
Event Paging
Atlas Gatewa Service
alerting y

CORE
Appdynamics Agent Amazon
SES
api

CORE
Agent
api

Other
Team’s
Agent


Chronos


Data Collection Pipeline

Data Processing Pipeline
Text

Tweet @jedberg with feedback! For more info, see BDT303: Data Science with Elastic MapReduce

Chuckwa/Honu messages /
min
63 billion
messages
a day


Best Practices


Incident Reviews
Ask the key questions:
• What went wrong?
• How could we have detected it
sooner?
• How could we have prevented it?
• How can we prevent this class of
problem in the future?
• How can we improve our behavior
for next time?

Best Practices for Data
• Have multiple copies of all data
• Keep those copies in multiple AZs
• Avoid keeping state on a single instance
• Take frequent snapshots of EBS disks
• No secret keys on the instance


Netflix autoscaling
2
Deployment

Text
1

Traffic Peak


AWS Usage
Dollar amounts have been carefully removed


Going multi-zone


Benefits of Amazon’s Zones

• Loosely connected
• Low latency between zones
• 99.95% uptime guarantee per region


Going Multi-region


Leveraging Multi-region

• 100% uptime is theoretically possible.
• You have to replicate your data
• This will cost money


Circuit Breakers (Hystrix)
Be liberal in what you accept, strict in what you send


Just a quick reminder...

• (Some of) Netflix is open source:
• https://github.com/netflix


We are sincerely eager to
hear your feedback on this
presentation and on re:Invent.

Please fill out an evaluation
form when you have a
chance.

Questions?


Getting in touch
Email: jedberg@{gmail,netflix}.com
Twitter: @jedberg
Web: www.jedberg.net
Facebook: facebook.com/jedberg
Linkedin:
www.linkedin.com/in/jedberg

RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agility - AWS re: Invent 2012

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agility - AWS re: Invent 2012

Ähnlich wie RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agility - AWS re: Invent 2012 (20)

Mehr von Amazon Web Services

Mehr von Amazon Web Services (20)

RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agility - AWS re: Invent 2012