In this paper, we propose a new metric to support the goal of reducing Macro PUE. This new data is logically constructed along the same formula lines as Macro PUE but is based on sets of individual IT and cooling energy usage data and, we therefore refer to it as Micro PUE.

1. Micro PUE.
The Key to Data Center Energy Savings.
A White Paper
By Robert Hunter and Chet Sandberg
Revision 1

2. Overview
Most in the data center world have heard of the metric Power Usage Effectiveness, or
PUE. PUE was one of the first metrics developed to give data center managers an idea
of the energy efficiency of their operations as compared to others. This metric was
developed by The Green Grid and has since been adopted by the Environmental
Protection Agency as the standard for determining Energy Star achievements for data
centers in the United States.
PUE is defined as follows:
As a brief description, PUE is the average amount of energy used to process and cool
each 1 kWh of IT load throughout a data center. The numerator in the formula includes
the IT load, the cooling load and, to a lesser extent, electrical equipment efficiency
losses. IT and cooling loads comprise the vast majority of power usage in this equation.
It is important to understand that the PUE calculation produces a single, macro figure
for the entire data center. While PUE is a key to understanding the overall energy
efficiency of a computer room, a single summary number does little to help manage
individual components in order to increase energy usage effectiveness. Therefore the
need to supplement Macro PUE with a more granular metric becomes evident.
In this paper, we propose a new metric to support the goal of reducing Macro PUE.
This new data is logically constructed along the same formula lines as Macro PUE but is
based on sets of individual IT and cooling energy usage data and, we therefore refer to
it as Micro PUE.
How PUE is Driven by Cooling Efficiency
In order to understand how to lower PUE, be it either Macro or Micro, it’s important to
understand the math behind the formula. Recall that PUE is defined as follows:
If we break that formula into its components, we get the following:
Because IT Energy appears in both the numerator and the denominator, a change in
that value will have much less effect on the outcome of PUE than other components of
1

3. the formula. If you couple this with the fact that cooling energy is typically 3 to 5 times
the combined total of electrical equipment efficiency losses, it becomes clear that
cooling energy is the primary component that influences the outcome of the PUE
calculation. (Lighting, for the purposes of this discussion is ignored as it accounts for
less than 1% of the equation.)
To illustrate this, consider the following example of a data center with a PUE of 2.00.
Three cases are presented each which reduces one component by 10%. The results
are rather startling. As can be seen, a reduction of 10% in cooling energy has a
significantly higher effect on lowering Macro PUE than the lowering of either IT load or
increasing electrical efficiency. In fact, lowering the IT load actually raises PUE.
Base Case. Macro PUE = 2.00
Case 1. Lower IT load by 10% from base case (efficiency loss from UPS and PDU
drops same percentage of kW). PUE actually rises by 4.7%.
Case 2. Lower Electrical Efficiency Loss by 10% from base case. PUE drops by less
than 1%.
2

4. Case 3. Lower Cooling load by 10%. PUE drops by over 4.2%.
While the example in Case 1 may seem improbable (IT load decreases but cooling load
stays the same) it is not as uncommon as one might think. In sites where virtualization
is employed, IT load reductions are common but, remaining hot spots and newly
created mismatches between IT heat and cooling resources often stifle expected
reductions in cooling energy usage. Thus, simply reducing IT loads without the ability to
manage cooling energy is actually counterproductive in the attempt to lower PUE.
On the other hand, Case 3 shows how reducing cooling loads vis-à-vis IT loading can
lower PUE by a significant amount. In this case, a 10% reduction in cooling energy
usage leads to a reduction in PUE of slightly more than 4.2%. While these examples
use a hypothetical starting point of a Macro PUE of 2.00, the same effects could be
seen from other PUE values as well. In fact, the higher the starting PUE position, the
more effect will be seen by changing cooling energy. These figures make it clear that
PUE is really a measure of the effectiveness of cooling energy usage as compared to IT
loading.
Given those organizations such as the Uptime Institute show average PUE values of 2.5
or morei, it’s clear that data center managers must begin to focus anew on lowering
cooling energy in relation to IT load in order to reduce PUE. Lowering PUE will require
access to cooling energy data and the ability to manage cooling loads in relation to IT
energy use. The principal of Management by Information (MBI) seems very appropriate
to offer assistance in the goal of managing cooling loads and PUE. MBI states that:
“You can only manage what you measure and that you can only manage well what you
measure accurately.” It stands to reason, then, that both IT energy and cooling energy
usage must be measured accurately and continuously.
It is clear that the ability to measure the IT circuits and cooling loads form a key
component to managing PUE. The EPA has already codified PUE as the principal
calculation for their data center Energy Star program but, their initial metering
requirements include only metering for the whole data center and for the IT load at the
UPS output. However, in releasing its new standards, the EPA has made the following
statement: “meters located at the PDU output, or on the servers directly, may be extremely valuable for your
organization. These allow for a more advanced calculation of PUE which can help you measure and improve
Effectiveness of power distribution at your facility. Therefore EPA encourages you to install this type of meter at
ii
The use of branch circuit meters at the PDU (power distribution unit) and
your facility. “
metered power strips at server level are increasing in popularity and this impetus by the
3

5. EPA is likely to accelerate this trend. The crux of their statement is that such sub
metering allows for “a more advanced calculation of PUE” that can improve both the
measurement and management of data center energy efficiency. Micro PUE is formed
on the basis of sub metering data for both IT loads and cooling loads.
Sub-metering, and more specifically, branch circuit sub metering can provide significant
value for managing energy and billing energy usage. While many are focusing on
branch circuit metering for IT circuits within PDU’s, sub metering can also be applied to
individual cooling units and to the cooling system as a whole. It is logical that measuring
and managing energy data for cooling loads would be highly advantageous as well and
could well be the missing piece of data that so many have been looking for. It also is
intuitive that, if reasonably small gains from cooling savings can be achieved, the cost of
branch circuit metering of these circuits could have a relatively short payback time and
provide significant long term benefits.
It can be seen that measuring and managing cooling energy at the micro, or individual
unit level, is necessary to reduce PUE. With that in mind, this paper will examine the
specifics of a metric with that goal in mind: Micro PUE. The following section gives a
background on the philosophy of Micro PUE and discusses how it is calculated.
Measuring Micro PUE
We have seen that changes in cooling energy vis-à-vis IT load are the key in moving
PUE as a number. Unfortunately, as a single number for all data center energy usage,
PUE can indicate the direction of energy effectiveness but, it cannot isolate the causes
of inefficiencies. If one is to manage and reduce PUE, it will be necessary to view the
cooling energy in relation to IT energy at the individual device level, that is, the
individual cooling unit. That will require the knowledge of the energy used by the
individual cooling units and the knowledge of the appropriate share of IT energy that is
removed by each cooling unit.
On the surface, that appears to be a daunting challenge. However, sub metering
technologies now exist that would allow users to complete the first task, the
measurement of energy use of each air conditioning unit. The ability to allocate the pro-
rata share of IT load to each cooling unit requires a bit of math and physics. But, we will
demonstrate that the amount of IT load directly attributable to each cooling unit can be
accurately measured. These two pieces of data will enable us to create a measurement
of Micro PUE.
Sub-metering individual cooling units
Measuring the energy use of cooling units requires the ability to measure the two
primary types of cooling systems that are common to data centers; Computer Room Air
4

6. Conditioning (CRAC) systems and Computer Room Air Handler (CRAH) systems. In a
CRAC system, each data center cooling unit has its own compressor for refrigerant and
each unit releases its heat via a remote heat rejection unit such as an outdoor
condenser or dry cooler. In a CRAH system, each air handler unit in the data center
relies on a central chiller to provide cool liquid and that chiller, in turn, uses a heat
rejection unit such as a cooling tower.
Sub-metering, and more specifically, branch circuit sub metering can provide significant
value for managing energy and billing energy usage. While many are focusing on
branch circuit metering for IT circuits within PDU’s, sub metering can also be applied to
individual cooling units and to the cooling system as a whole. It is logical that measuring
and managing energy data for cooling loads would be highly advantageous.
Energy consumed by CRAC units and their heat rejection units as well as CRAH units
and their chillers can be measured at the source or at the central panel level. If cooling
circuits are clearly marked and accessible within electrical panels, this is the easiest and
most economical method of measuring cooling circuits with branch circuit metering. If
panels are not centrally located then some of the cooling circuits may need to be
monitored at the individual unit itself.
Measuring the pro-rata share of IT load associated with each cooling unit
To get to a point where we can measure the Micro PUE of each air conditioner unit, we
must measure the pro-rata share of IT energy and electrical infrastructure energy that
each unit removes. Fortunately, we can take advantage of some physics related to
energy to provide this data. The key fact that comes into play here is that the energy
consumed by each IT device is equivalent to the heat that it releases. Physics teaches
us that energy is neither created nor destroyed but simply changes states. A great
example of this can be seen in a data center where the electrical energy consumed as
measured in kilowatt hours is the same as the heat energy released in kilowatt hours.
5

7. These two, kilowatt hours of electrical energy consumed and kilowatt hours of heat
radiated are identical, they are simply two sides of the same coin. If we can measure
the amount of heat being removed by each cooling unit, we would then know the exact
amount of energy being consumed by the IT equipment and electrical infrastructure
whose heat is being removed by that cooling unit.
As the figure on the below shows, electrical energy is used and then radiated first by the
UPS, then the PDU and finally the IT equipment. The total amount of electrical energy
consumed is equal to the amount of heat energy released.
The removal of heat energy is traditionally measured in British Thermal Units, or BTU’s.
The BTU’s of heat removed can be calculated by measuring the difference in supply air
or liquid temperature vs. return air or liquid temperature times the flow in cubic feet per
minute for each cooling unit. The formula for calculating BTU’s is as follows:
( )
Where k is a constant associated with the coolant (e.g. air, water, etc.)
BTU’s removed by each air conditioner unit can then be converted into watt hours of
heat removed through the mathematical relationship between the two:
With the data now converted into kilowatt hours of heat removed by each CRAC or
CRAH unit, one can now know the exact amount of wattage of energy being removed
by the unit. With this information, Micro PUE can now be calculated.
Micro PUE is simply the PUE of each individual cooling unit and is defined as follows:
6

8. Because all heat loads, whether from IT or electrical efficiency losses, must pass-
through and be removed by individual cooling units, the wattage of heat removed
automatically includes each of the IT loads and their electrical losses. One needs only
allocate the respective portion of IT vs. efficiency loss by using the procedure described
in the above paragraph to yield a correct answer for Micro PUE.
The total kilowatt hours of heat removed is calculated through measuring BTU’s. The
total energy used by the cooling units is measured by a branch circuit meter. Lastly, the
IT portion of the wattage of heat being removed is simply:
kWh of Heat Removed by Cooling Unit N * (1- Efficiency loss of UPS and PDU)
These efficiency losses are relatively fixed on a day-to-day basis, although they do
change somewhat over time. In general, one can look at the kilowatts in vs. kilowatts
out on a UPS to get electrical efficiency loss. The loss of a PDU would be
approximately 4% for a standard efficiency transformer and 1.5% for a high efficiency
transformer, as discussed by Mazzetti iii, in a thorough analysis of UPS and PDU
efficiency losses. We have added 1% in the examples in this paper to this to allow for
line losses inherent in breakers, junction boxes and wire length losses bringing an
estimate of standard PDU losses to 5%.
As a brief description, Micro PUE is the actual amount of energy used to process and
cool 1 kWh of IT load through a given cooling unit. Thus, Micro PUE looks at PUE
though each IT/cooling segment of a data center. As a granular measurement, it
focuses on discovering the relative efficiencies or inefficiencies of each data center
segment, providing the visibility to users necessary to effectively manage their PUE.
It should be noted that, the measurement for the heat removed by all of the CRAC and
CRAH units will likely be slightly different than total IT load and efficiency losses
because of shell loads from outside heat or cooling. However, studies by the EPA have
shown that, regardless of climate, shell loads add or subtract only a small percentage
energy compared to the total IT load and electrical efficiency losses. Such shell loads
are assumed but never quantified in Macro PUE but can, for the first time, be quantified
by Micro PUE.
Practical uses of Micro PUE for lowering Macro PUE
With Micro PUE data, the user will now have the ability to see their cooling units ranked
in order from most efficient to least efficient in terms of energy usage effectiveness. In
order to reduce Macro PUE, changes may be made to the dynamics of the hardware of
the air conditioning unit, to the air flow patterns of the heat being removed, or both.
7

9. The ability to see Micro PUE also provides a view of the actual movement of heat from
the IT cabinet source to the cooling unit that removes the heat. This can be seen by
addition, subtraction or movement of IT heat load vis-à-vis the corresponding addition or
subtraction to the heat removed by individual or combinations of cooling units. While
calculations such as Computational Fluid Dynamics provide estimates of heat
movements, Micro PUE offers the user the ability to see the actual track of heat flows in
real-time and historical trends.
Micro PUE can expose the actual causes of cooling inefficiencies. Some of the leading
causes of cooling inefficiencies have been identified by organizations that have studied
these problems and include the following:
 Chiller/Compressor Cycling. This condition is analogous to driving a vehicle in
stop-and-go city traffic as opposed to a constant speed on the freeway. In stop-
and-go traffic, the vehicle is constantly cycling up-to or through its optimum
energy savings band (measured in Miles per Gallon or MPG) and never achieves
its optimum MPG range. A chiller or compressor operates in a similar manner.
Each unit has an optimum zone of maximum energy efficiency (measured in
BTU’s or tons – 12,000 BTU’s – removed per kilowatt hour). By cycling above
and below this zone, the user wastes an enormous amount of energy. The ability
to see chiller or compressor energy usage in relation to BTU’s removed can allow
users to tune each CRAC or CRAH unit and chiller for maximum efficiency.
 Fighting CRAC or CRAH units. This condition is analogous to a server’s use of
two power supplies at the same time as opposed to using one supply with the
second in standby. Two or more air units are said to be fighting one another
when they compete to remove the same heat load of a nearby heat source such
as a data cabinet. In this situation, one or more units are always being
underutilized while another is either over-utilized. This creates a problem for the
underutilized units similar to that of chiller/compressor cycling. That is, one or
more units are always operating well below the optimum efficiency band and
significant amounts of energy are wasted. By seeing the Micro PUE of each unit,
especially units that are in close proximity, users can detect a condition of
fighting. This can be alleviated by the use of Variable Frequency Drive fans to
coordinate heat removal and/or by the use of a master system controller unit.
 Low Delta T. Because Delta T is an inherent component of BTU measurement to
compute Micro PUE, it is suggested that it and all components of Micro PUE be
graphed and stored as individual pieces of data. This allows conditions such as
low Delta T to be quickly identified as a cause of high Micro PUE. Low Delta T
can be alleviated in a number of ways including the use of hot isle/cold isle
8

10. technologies, direct venting of IT heat load to return air, lowering the supply air
temperature and others.
Summary
This paper has shown that PUE is essentially a measurement of the cooling efficiency
of a data center in relation to its IT load. The use of Macro PUE to manage data center
energy effectiveness, without the use of its micro-components, has been demonstrated
to be a difficult if not impossible task.
The ability to measure, trend and manage by Micro PUE has been introduced. Micro
PUE was discussed with its ability to reduce or even cure many of the common reasons
for data center cooling efficiency losses. As such, we propose that Micro PUE is a
valuable and perhaps vital tool for data center managers in achieving energy
effectiveness goals set with Macro PUE.
i Power Usage Effectiveness, Mark Fontecchio, ManageDataCenter.com, May 6, 2008
ii Energy Star Rating for Data Centers, Frequently Asked Questions, July 2010
iii William Mazzetti, Where Did My Effectiveness Go? William Mazzetti, Data Center Guru, May 10, 2009
9