With Amazon Web Services it's possible to serve the needs of modern high performance advertising without breaking the bank. This session covers how AdRoll processes more than 60 billion requests per day in less than 100 milliseconds each using Amazon DynamoDB, Auto Scaling, and Elastic Load Balancing. This process generates more than 2 GB of data every single second, which will be processed and turned into useful models over the following hour. We discuss designing systems that can answer billions of latency-sensitive global requests every day and look into some tricks to pare down the costs.

1. November 14, 2014 | Las Vegas, NV
Valentino Volonghi, CTO, AdRoll
Siva Raghupathy, Principal Solutions Architect, AWS

2. 60 billion requests/day

3. We
Must
Stay
Up
1% downtime
=
>$1M

4. No
Infinitely
Deep
Pockets

5. 100ms MAX Latency

6. Paris-New York: ~6000km
Speed of Light in fiber: 200,000 km/s
RTT latency without hops and copper: 60ms
Paris-New York: ~6000km
Speed of Light in fiber: 200,000 km/s
RTT latency without hops and copper: 60ms
6000
km
60 ms
c-RTT

10. a
a
a
r
•Memcache
aOpensource
aMature
aBlazinglyfast
rNostrong guarantees
•Redis
aOpensource
rStoragescale
rNotreally distributed
rOperationallyintense.
•Hbase(we still use this)
aOpensource
aMaturingquickly
aGreatscale
rReallyhard to operate

11. http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html
Cassandra
DynamoDB
Delta
10/90mix, $/month
$287,064
$131,040
219%
50/50 mix, $/month
$287,064
$280,800
~0%
10/90, 3-yrreserved
$27,075.6
($904kupfront)
$15,736
($504k upfront)
180%
•10 people Cassandra ops team: $150k/month (fully loaded)
•0 DynamoDBops team: $0

17. Instance
Nginx
ErlangBidder
Black Box
Supervisor
ASG
Amazon
Kinesis
Amazon S3
DynamoDB
Elastic Load Balancing
Instance
Nginx
ErlangBidder
Black Box
Supervisor
ASG
Elastic Load Balancing

18. Amazon S3
Amazon
Kinesis
DynamoDB
DynamoDB
DynamoDB

20. HashKey
Value
Key
{‘segments’: set([‘123’, ‘345’]),
‘timestamp’:‘1234556’,
‘leads’: ‘to-write-contention’}
HashKey
RangeKey
Value (just a single timestamp)
Key
Segment
1234554343254
Key
Segment1
1231231433235

22. ArrayListListWriteRequestArrayListWriteRequestKeyAtomicIntegerAtomicIntegerSetWriteRequestCollectionsHashSetWriteRequestStringAmazonDynamoDB
RunnableRunnablerun
ListWriteRequest
BatchWriteItemResultBatchWriteItemRequestImmutableMapListWriteRequest
Map

29. Not for me!

37. Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding

38. Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding

39. Data Collection = Batch Layer
Bidding = Speed Layer
Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding

40. US East region
Availability Zone
Availability Zone
Elastic Load Balancing
instances
instances
Auto Scaling group
Amazon S3
Amazon Kinesis

41. US East region
Availability Zone Availability Zone
Elastic Load
Balancing
instances instances
Auto Scaling group
Amazon S3
Amazon
Kinesis
Apache
Storm
DynamoDB
US West region
EU West region
DynamoDB
DynamoDB

42. Data Collection Bidding
US East region
Availability Zone Availability Zone
Elastic Load Balancing
instances instances
Auto Scaling group
Amazon
S3
Amazon
Kinesis
Apache
Storm
DynamoDB
Availability Zone Availability Zone
Auto Scaling group
Elastic Load Balancing

43. Data Collection
•Amazon EC2, Elastic Load Balancing, Auto Scaling
Store
•Amazon S3 + Amazon Kinesis
Global Distribution
•Apache Storm on Amazon EC2
Bid Store
•DynamoDB
Bidding
•Amazon EC2, Elastic Load Balancing, Auto Scaling
Data CollectionBiddingAd Network 2Ad Network 1Auto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupApache Stormv2V3V3v1v2V3V3v1V2V3V3V1Auto Scaling GroupV3V4Elastic Load BalancingElastic Load BalancingElastic Load BalancingElastic Load BalancingDynamoDBWriteReadReadReadReadReadReadWriteWritesWriteWriteReadV3` Elastic Load BalancingElastic Load BalancingElastic Load BalancingElastic Load BalancingElastic Load BalancingElastic Load BalancingDynamoDBData CollectionBiddingDynamoDBWriteReadReadWriteWriteWriteAmazon S3Amazon Kinesis

44. http://bit.ly/awsevals