1. EEDC
34330
Execution
Environments for Intelligent Placement of
Distributed Datacenters for Internet
Computing Services
Master in Computer Architecture,
Networks and Systems - CANS
Homework number: 6
Members:
Roger Rafanell roger.rafanell@bsc.es

2. Outline
 Introduction
 Motivation
 Placing Datacenters
 Evaluation
 Conclusion
2
2

3. Introduction: Datacenter construction costs
 Each datacenter costs >$100M to construct
– The smaller datacenters are rated at ~25MW
 Examples:
– Microsoft DCs in Virginia & Chicago: $500M each
3
3

4. Outline
 Introduction
 Motivation
 Placing Datacenters
 Evaluation
 Conclusion
4
4

5. Motivation
 Internet services require thousands of servers
 Use multiple “mirror” datacenters
– High availability and fault tolerance
– Low response time
 Spend millions building and operating datacenters
 Consume enormous amounts of brown energy!!
5

6. Outline
 Introduction
 Motivation
 Placing Datacenters
 Evaluation
 Conclusion
6
6

7. Intelligent placement of datacenters
Goal: Manage the monetary and environmental costs
 Define framework
 Model costs and datacenter characteristics
 Create solution approaches
 Collect cost and location-related data
 Create placement tool
7
7

8. Selecting datacenter locations
 Model datacenter placement
– Network latencies
– Availability
8
8

9. Selecting datacenter locations
 Model datacenter placement
– Network latencies
– Availability
 CAPEX costs
– Distance to electricity and networking infrastructure
– Land and construction (maximum PUE)
– Power delivery, cooling, backup equipment
– Servers and networking equipment
9
9

10. Selecting datacenter locations
 Model datacenter placement
– Network latencies
– Availability
 CAPEX costs
– Distance to electricity and networking infrastructure
– Land and construction (maximum PUE)
– Power delivery, cooling, backup equipment
– Servers and networking equipment
 OPEX costs
– Maintenance and administration
– Electricity and water prices (average PUE)
10
10

11. Selecting datacenter locations
 Model datacenter placement
– Network latencies
– Availability
 CAPEX costs
– Distance to electricity and networking infrastructure
– Land and construction (maximum PUE)
– Power delivery, cooling, backup equipment
– Servers and networking equipment
 OPEX costs
– Maintenance and administration
– Electricity and water prices (average PUE)
 Incentives (taxes)
11
11

12. Selecting datacenter locations
 Model datacenter placement
– Network latencies
– Availability
 CAPEX costs
– Distance to electricity and networking infrastructure
– Land and construction (maximum PUE)
– Power delivery, cooling, backup equipment
– Servers and networking equipment
 OPEX costs
– Maintenance and administration
– Electricity and water prices (average PUE)
 Incentives (taxes)
12
12

13. The problem formulation
 Goal
– Minimize CAPEX and OPEX
 Constraints
– Response times < MAX LATENCY for all users
– Min consistency delay between 2 DCs < MAX DELAY
– Min system availability > MIN AVAILABILITY
 Output
– Number of servers at each location
– Minimum cost
13
13

14. Solving the (non-linear) problem
 Linear Programming
– Does not support non-linear costs
 Brute force
– Too slow
 Simple heuristics
– May not produce accurate results efficiently
14
14

15. Our approach for solving the problem
 Evaluate each potential solution
– Quickly via Linear Programming (LP)
 Consider neighboring configurations
– Simulated annealing (SA)
 Cost optimization process
– Combine SA and LP
SA
LP LP
Current solution Near neighbor
15
15

16. Our approach for solving the problem
SA
LP LP
$13.8M/month $10.3M/month
SA
SA
LP LP
$9.2M/month $10.7M/month
16
16

17. Summary of our approach
1) Generate a grid of tentative locations
2) Collect data about each location
3) Define datacenter characteristics
4) Instantiate optimization problem
5) Solve optimization problem
17
17

18. Outline
 Introduction
 Motivation
 Placing Datacenters
 Evaluation
 Conclusion
18
18

19. Comparing locations for 60k-server DC
Servers Land Building Connection Energy Water Staff Networking
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
Austin Bismarck Los New York Orlando Seattle St. Louis
m
o
p
d
n
u
h
C
e
a
s
r
t
)
(
l
Angeles
19
19

20. Interesting questions
 How much does…
… lower latency cost?
… higher availability cost?
… faster consistency cost?
… a green DC network cost?
… a chiller-less DC network cost?
20
20

21. Cost of 60k-server green DC network
Green DC network costs $100k/month more, except when latency <70ms
21
21

22. Cost of a 60k-server chiller-less DC network
14
Chiller-less
12 Traditional
10
8
6
4
2
m
o
d
n
C
a
s
r
t
)
(
l
i
0
30 50 70 90 110
Maximum latency (milliseconds)
Chiller-less DC network is cheaper but it cannot achieve low latencies
22
22

23. Conclusions
 First scientific work on smart datacenter placement
– Proposed framework and optimization problem
– Proposed solution approach
– Characterized many locations across the US
– Built a tool to automate the process
– Answered many interesting questions
 Results show that smart placement can save millions
 Work enables smaller companies to reap the benefits
23
23

24. Future work
 To extend with data from Europe
 Include tax incentives
 Test the tool with data from real services
24
24

25. Questions
25

26. Specially thanks to the real authors
of the work …
Íñigo Goiri, Kien Le, Jordi Guitart,
Jordi Torres, and Ricardo Bianchini
26
26

27. Energy costs and carbon emissions
Energy/year Energy CO2/year
Company #Servers
(MWh) cost/year (Metric tons)
eBay 16K 0.6 x 105 $3.7M 0.4 x 105
Akamai 40K 1.7 x 105 $10M 1.0 x 105
Rackspace 50K 2 x 105 $12M 1.2 x 105
Microsoft >200K >6 x 105 >$36M >3.6 x 105
Google >500K >6.3 x 105 >$38M >3.8 x 105
Sources: [Qureshi’09], EPA
27
27