Distributed Quota Management in an Ad Ad Serving Environment OR How we manage ad spends across data centres without over spending

1. Distributed Quota Management
in an Ad Serving Environment
How Reduce Data Manages its budgets without exceeding ad spends
http://reducedata.com

2. Problem
Managing Quotas / Budgets in An Ad Serving
Environment is not a trivial task.
Ad Campaign Budgets Across a
Distributed infrastructure when not managed
properly leads to excess spends.

3. Previous Solution
Sync Every 5 seconds
Cassandra Data
Ad Server
Ad Server
Ad Server
Kafka Pipe
Data processing
system
Hbase based
campaign
reporting data

4. The Issues
Simple Syncing of data across distributed data structure is easy but sticking
within limits is tough.
5 second refresh not guaranteed to be upto date. Reporting delays caused
spends to exceed.
Server specific budget was not proportionately allocated, based on historical
availability of traffic on that server.
No clear mechanism of allocation of budget spends or reallocation (when
servers go down).

5. Solution
●
●
●
Our Solution was based on this white paper
titled:
Quota enforcement for high-performance
distributed storage systems
http://www.ssrc.ucsc.edu/Papers/pollackmsst07.pdf

6. Key Concepts
●
●
●
●
Epoc : A day is divided into epocs.
At each epoc DSPServers request for quota from Quota
Management Cluster.
Ad Server doesn't decide how much quota it needs instead
a new entity the Quota Management Cluster decides how
much quota to assign to it.
Traffic for a Campaign maybe available at multiple
endpoints(US East, US West, Asia)

7. How Quotas are Decided
●
●
Quota is distributed to end points (different data centre locations) and to
the individual servers depending on the historical traffic / consumption
of budgets.
Two factors that decide how much is allocated in each epoc:
●
●
Pacing variable as decided by the advertiser
Historical consumption data (for example has the campaign been
spending lesser than planned?, if so, increase EPOC allocation)

8. Quota Management Cluster
●
●
Quota management cluster has multiple servers and needs
a data store which supports transactions.
The Quota Management Cluster is Synchronized using
Zookeper.

9. Logical Architecture
Sync at every EPOC 2 ways
Ad Server
Ad Server
Ad Server
Ad servers can sync with the Quota management server at EPOC which could
be say every hour.
Ad servers refund unspent balance to Quota Management Cluster.
Quota management can be in one location initially but can be made redundant
to each data centre.
Quota Management Cluster uses a shared data store on cassandra to sync its
other nodes for allocation.
Quota
Quota
Management
Management
Server
Server
Cassandra based
Cassandra
Quota data based
Quota data
Zookeeper
Zookeeper

10. Sync Process
Ad Server
Quota Management Server
Zookeeper
Cassandra
Sync at every EPOC
Push budgets, get unspent amounts
Sync allocation data to cassandra cluster
Check Server status
Get list of available ad servers
Push available budgets on a restart.
Calculate available spends

11. Handling Ad Server Shut Downs
●
●
Zookeeper tracks server failures.
Quota management system reallocates remaining budget
(last budget allocated - amount spent) to a new or the same
server when it restarts.

12. Handling Communications
The Quota Management Servers talk to Ad Servers using
REST Calls. This required lowest development time and was
built on top of the existing stack.
Alternatives we considered but decided against.
2. Kafka
3. RPC over protobuf

13. Get in Touch
Engineering Team
Reduce Data
reducedata.com
twitter: @reducedata
facebook.com/reducedata
engineering@reducedata.com