1. Power Usage Effectiveness (PUE) and the
Importance of Measurement
September, 29th 2011
Keith Jackson – Sales Director
2. Raritan Corporate Overview
Founded: 1985
Global Offices: Somerset, NJ (HQ), UK, Germany, the Netherlands, Canada,
Japan, Taiwan, India, Australia, France, China, Singapore
Global Reach: Products are sold and supported in 76+ countries
Customer Base: Large and midmarket companies supporting over
50,000 data centers globally
Sales Model: Indirect (resellers and distributors)
Corporate Structure: Privately owned
Employees: ~450
Values: Honesty, integrity, customer focus, personal commitment,
teamwork and value creation
Patents: 38 Pending; 18 Granted
2 Raritan Confidential and Proprietary
5. Power Solution - Adopted by Leading Companies Worldwide
Financial
Technology
Government
Healthcare &
Education
Others
Raritan Confidential and Proprietary
6. PUE Reconsidered
Utility + Limitations of PUE / DCiE as a Datacenter Efficiency Metric
► Historical Basis for PUE – what is the problem?
► Strengths of PUE Metric – what does it measure?
► Weaknesses of PUE Metric – what does it omit?
► The importance of Measurement to Drive Efficiency
► Leveraging the Rack for Data Collection
6 Raritan Confidential and Proprietary
7. Open Secret: Datacenter electrical costs are skyrocketing…
SERVERS
$BILLIONS
OF
MILLIONS
7 Raritan Confidential and Proprietary
8. …fueled by an explosion in power / heat densities
Raritan Confidential and Proprietary
9. Industry groups + vendors propose a wide range of solutions
IT equipment Cooling & moving air
► virtualization ► avoid overcooling
► power save mode ► minimize humidification
► consolidation ► reduce air mixing via hot/cold air
► upgrade technology separation
► decommission servers ► blanking plates to minimize
recirculation
► move apps/jobs to virtualized servers
and shut down servers ► raised floor grommets to reduce
bypass airflow
► batch processing during off-peak
► optimize floor layout (CFD)
► closely couple supply and returns to
the load
9 Raritan Confidential and Proprietary
10. But first, we need to understand the problem…
Typical Real-World Energy Allocation
in Today’s Datacenter
Source: EYP Mission Critical Facilities Inc., New York
10 Raritan Confidential and Proprietary
11. …so The Green Grid proposed PUE as a benchmark in Feb’07
Power Usage Total Facility Power
PUE =
Effectiveness
=
IT Equipment Power
DCiE is the exact same thing, but expressed as an inverse:
Datacenter 1 IT Equipment Power
DCiE = Infrastructure = =
Efficiency PUE Total Facility Power
11 “Green Grid Data Center Power Efficiency Metrics: PUE and DCiE”; by and Proprietary
Raritan Confidential Christian Belady, et. al.; www.thegreengrid.org
12. PUE is extremely valuable as a broad efficiency ratio…
Power Usage Total Facility Power
PUE =
Effectiveness
=
IT Equipment Power
EXAMPLE: Typical 1MW Facility
1 MW
PUE = = 2.0
0.5 MW
“My total facility consumes 2x
the power of the IT equipment load.”
1
DCiE = = 0.5
PUE
“Roughly 50% of the power in my
facility is used to power IT equipment.”
12 Raritan Confidential and Proprietary
13. … and has become a de facto standard metric
Sun Microsystems
Microsoft Google
• Dueling Press Releases (e.g. Google and Microsoft)
• EPA Datacenter “Energy Star” Program
• Department of Energy (DOE) “DCPro” Tool and Survey
13 Raritan Confidential and Proprietary
14. It’s critical to understand what PUE is, and what it is not!
A relative measure of your plant versus your IT equipment
1 MW
PUE = = 2.0
0.5 MW
Does not (in and of itself) address:
• Resilience: Tier I / II / III / IV architecture?
• Trending: How does my efficiency change as IT load changes?
• Diversity: Heterogeneous versus homogenous IT load?
• Utility: Are the servers themselves being used well?
In short, it does not benchmark the efficiency of the IT component.
14 Raritan Confidential and Proprietary
15. Example: Henry’s Super Awesome DatacenterTM (SAD)
• 1000 servers
• Equipment from every Raritan Customer
• Located in Reykjavik, Iceland (max temp = 55)
• Placed outdoors under a tented roof
• 24/7/365 operation
• All servers dedicated full-time to HenryAndHisCat.com
(Video site featuring footage of me playing with my cat.)
PERFECT PUE OF 1.0 !!
Every kilowatt is used to power servers. No energy “wasted”.
• Servers completely underutilized
• No critical resilience whatsoever
• No business value or practicality
• Real-world evaluation = complete waste of energy
15 Raritan Confidential and Proprietary
16. None of the proposed IT initiatives (on left, below) improve PUE
IT equipment Cooling & moving air
► virtualization ► avoid overcooling
► power save mode ► minimize humidification
► consolidation ► reduce air mixing via hot/cold air
► upgrade technology separation
► decommission servers ► blanking plates to minimize
extra savings on cooling recirculation
► move apps/jobs to virtualized ► raised floor grommets to reduce
servers and shut down servers bypass airflow
► batch processing during off-peak ► optimize floor layout (CFD)
► closely couple supply and returns
to the load
Power Usage Total Facility Power
PUE =
Effectiveness
=
IT Equipment Power
16 Raritan Confidential and Proprietary
17. Green Grid recognizes that PUE is not a panacea
Proposed: Three “levels” of PUE sophistication
17 Raritan Confidential and Proprietary
18. Summary
► PUE is valuable (and widespread) as a high-level metric
► You must understand the pros / cons of PUE to properly
evaluate industry, vendor, and press claims
► PUE is not sufficient to improve efficiency tactically
► Datacenters can only be optimized when the energy
chain is fully understood / instrumented:
the appropriate measurements;
at a sufficient level of detail;
trended over time;
easily interpreted;
at incremental CapEx cost;
18 Raritan Confidential and Proprietary
19. Gather data beyond PUE to enable tangible efficiency actions
Important Solution Attributes:
• Open standards / vendor agnostic
• Trended information over time
• Energy data (kwh), not just current (amps)
• Energy data per IT device (not just for rack)
• Product configs to fit all rack densities
• Make sure facilities is happy (UL489)
• Use what you have!
19 Raritan Confidential and Proprietary
20. Energy Metering Use Cases
“You can’t manage what
you don’t measure”
• Reclaim OpEx and CapEx via better
capacity planning.
• Improve cooling costs by removing
hot spots + optimizing set point
• Determine servers do I virtualize?
• Am I within ASHRAE
“recommended” guidelines?
• True PUE measurements
• Prevent branch and panel breaker trips
20 Raritan Confidential and Proprietary
21. To solve the real problem, we need to measure it…
Uptime Institute, “4 Metrics That Define Data Center Greenness”
21 Raritan Confidential and Proprietary
22. Attributes of True Energy Metering
Not Just Current
•kWh is true measure of power cost;
•Cannot assume voltage or power factor (errors are multiplicative);
•Must provide instrumentation for:
• Voltage (V)
• Current (A) Power Consumed =
• Power Factor
• Energy Consumption (kWh) Current
• Active Power (kW)
x Voltage
• Apparent Power (kVA)
x Power Factor
Billing Grade Accuracy
Each fluctuating at
•+/– 1% accuracy
60 times / second
•ANSI standards of “billing grade”
•Constant measurement (i.e. odometer);
not just sample at one point in time;
Raritan Confidential and Proprietary
23. … including environmental information
Source: ASHRAE, “Thermal Guidelines for Data Processing Environments”
23 Raritan Confidential and Proprietary
24. Rack-Level Energy Metering vs. Branch Circuit Metering
Pros
•Far less expensive infrastructure (Ethernet vs. MODBUS);
•Environmental metrics where they actually count (at the cabinet inlet);
•More accurate measurement of PUE;
•Provides remote panel (line) and rack PDU circuit
breaker measurements – increases availability;
•Data more easily accessible by business unit owners / IT
(not stuck in proprietary BMS system / network);
Cons
•Polling interval measured in seconds, not milliseconds;
•Complementary to BMS, not a replacement;
Raritan Confidential and Proprietary
25. Leveraging the Rack Power Strip to Enable Intelligent Cabinets
Philosophy:
•Rack power strips must always be deployed;
•Leverage rack power strip location and connectivity to make
every cabinet truly intelligent — at acceptable price.
Applications:
•Network connectivity
and scalability (WiFi 802.11 b/g/n);
•Environmental monitoring;
•Asset management automation;
•Other use cases not yet conceived!
Objectives only achievable if intelligence onboard is:
•flexible + powerful; Raritan competitive strengths
•reasonably priced; • 20 years as IT infrastructure supplier
• Provide identical components
•extremely reliable; to Dell, AMD, SuperMicro
Raritan Confidential and Proprietary
26. Example: Network Connectivity and Scalability
Choose connectivity Extreme scalability of data and deployment
option — now or later •Pre-configure all attributes via USB stick;
•Standard Ethernet (10/00);
•Data and sample buffering on each power strip;
•WiFi (802.11a/b/g/n);
•Open protocol design (3rd party reporting / integration);
•RFcode (303 / 433MHz);
Outlet-Level Data Sampling for 1000 Rack PDUs
Sample Rate 5 minutes
Avg. # Outlets 30
Data Rate 5.8 megabytes / hr
Network Equivalent 13 kbits/sec
( < 0.02% of a 10/100 network)
SNMP Traffic only 2 GETs per minute!!
Raritan Confidential and Proprietary
27. Example: Power Distribution and Metering
► Outlet meter
(and control)
► 1% KWh
► Intelligent
controller &
platform
► Hundreds of
models
Raritan Confidential and Proprietary
29. Dominion PX Environmental Sensor Options
DPX-T1 DPX-T1H1 DPX-T2H2 DPX-T3H1
RJ12 Connection
To Dominion PX
Rack Power Strip
TEMP TEMP
TEMP + + TEMP
HUMIDITY HUMIDITY
1M (3.3ft)
Cable Between
Components
TEMP TEMP
+ +
HUMIDITY HUMIDITY
Each sensor node barcoded with TEMP
serial number for easy
configuration Raritan Confidential and Proprietary
30. Example: Intelligent Asset Management NEW!
• Locate assets with precision: rack + 1U position
• Intelligent asset tags w/barcode;
• Intelligent rack sensor bar;
• Modular segments snaps together
to make 42U, 45U, 48U, 54U sensors;
• Launches May 2011;
• Works with any Raritan PX2 power strip;
• RGB LED
for each U-
space;
• Asset tag
connector for
each U-space;
Raritan Confidential and Proprietary
31. CONFIDENTIAL
Example: Intelligent Asset Management (Demo Video)
Advantages over passive RFID
•Accurate to individual 1U space;
•Provides visual feedback mechanism (LED), to reduce human error during trouble tickets;
•Leverages existing connectivity infrastructure (network + rack space);
•Much lower cost per asset tag;
Raritan Confidential and Proprietary
32. Raritan provides a real-time, integrated approach
Temp/Humidity
metering per
ASHRAE standards
kWh metering
of every receptacle
Instrumentation Visualization
32
Dominion PX Power IQ and dcTrack
Raritan Confidential and Proprietary
33. Raritan Approach: Any configuration, any option, all with true intelligence
Voltage ( + International Options )
120V 208V 208V 3ph 400V 3ph
Power Capacity
1.4kVA 34.5kVA
Output Type
NEMA C13 C19 Combinations;
5-20P International;
Twist-Lock;
Feature Category
Meter Remote Outlet Outlet
Only Switching Metering Metering
+Switching
Raritan Confidential and Proprietary
34. Build Options
• Over 300 Raritan intelligent rack PDU configurations exist.
• In addition, clients can request other options
(minimum order quantities / increased lead times may apply).
SKU Meters Feet
standard 3M 9.8 ft
–A1 1.5M 4.9 ft
–A2 2M 6.6 ft
–A4 4M 13.1 ft
–A5 4.5M 14.8 ft
Retention Brackets Chassis Colors Input Cord Length
in many different lengths
Non-Proprietary
Locking Outlets
functions with any standard
34
IEC C14 / C20 AC cable
Raritan Confidential and Proprietary
37. eBay
► $334M data center, 250,000 sq ft
► 1500 Dominion PX units for Rooms
1 & 2 (total 3 rooms)
► 400V power distribution to rack
► Outlet-level metering
► 2772 temperature sensors
► Industry’s first WiFi deployment
► Hot-Aisle Containment
► “What is my cost per search?”
Raritan Confidential and Proprietary
38. Microsoft Enterprise Engineering Center
► Building 25, Redmond
600+ servers
More than 2 Petabytes of storage
Central switching fabric of over 20
Terabytes
7 customer labs
3 private conference rooms
A technical briefing center
► 90 Dominion PX iPDU’s
► PX Environment Sensors
► Power IQ energy management
software
► Dominion KX KVM/IP
► CommandCenter
Access Gateway
Raritan Confidential and Proprietary
39. Cisco Development Labs
► Issues and Goals
• Cisco’s mission: “Reduce carbon footprint 25%”
• Cisco’s 2000 labs & datacenters worldwide
– consume 80% of Cisco’s energy
• San Jose, CA = $50M annually on power
► Project
• Identify the best “smart” PDU and management
application vendor and establish a corporate
standard for datacenter and lab environments
• Utilize “Smart PDU” to benchmark power
consumption and drive efficiency and savings
► Results
• Raritan’s PX, Power IQ chosen as the
“Preferred” products for both Smart PDU
and power management application
• So far energy savings measured as high
as 36% in some labs
Raritan Confidential and Proprietary
40. Oasis of the Seas
► World’s largest cruise ship!
1200 feet bow to stern
2700 cabins
6300 passengers, 2100 crew
$1.5 billion dollars
2.5 years to build
► 133 Dominion PX intelligent
rack PDUs on board
1628 metered and remotely controlled
power outlets
► PX environment sensors
266 temp & 266 humidity sensors
► Power IQ for central power control
and environment management
► Dominion KX and SX for centralized
KVM and Serial console management
Raritan Confidential and Proprietary
41. In Summary : Raritan Core Advantages
41 Raritan Confidential and Proprietary
Data center capacity Know where you have space and sufficient power and cooling resources so you can consolidate, upgrade, change and grow IT services. Asset management Know what you own, where it is located, and how it is configured so you can streamline internal operations, align resources to maximize value and identify and minimize discrepancies. Change management Support procedures used for efficient and prompt handling of all changes to IT infrastructure, in order to minimize the impact upon service. Power distribution Intelligent rack power distribution to deliver a wealth of accurate and detailed power information. Environment Easily monitor cool air inlets, air flow, air pressure and hot aisle (deltaT) to maximize efficiency and guarantee availability. Energy management Software, compatible with third-party PDUs, provides trending, usage and billing reports and removes IT vs. facilities ‘silos’ creating greater energy awareness, efficiency and higher availability.
Depending on role facilities vs. IT PUE can mean something different things to different people
Very soon, you’ll be spending more on electrical cost that the Capex for the equipment Congratulations, that point is now. If you buy a standard, HP, IBM, Dell server you will pay more of the life of that machine for electricity, than the cost of the server Today you’ll pay more to power, than CAPex to buy the actual piece of equipment.
As you can see the problem is exponential in Scale Single core to 16 core servers Blades and high End Switches Even a 2u server running 16 cores In 1998, if you were to say that in 2009 you would need 10-20KW per rack they would say you’re crazy
How do people ameliorate that problem? This is like the proverbial finger in the dike Which one is the hole that will cause the dam to break A limited IT staff can only do so much Which brings the bang for the buck, what do you do today?
What’s happening in your datacenter Multiple sources, academic, consultants, Government ( EYP, EPA, Lawrence Berkley) indicate that approximately 50% of energy use in a data center is for IT function and about 40% for moving and cooking air. So if you want to target the problem, look at the servers.
To deal with the source of the problem. The green grid proposed PUE at a benchmark 2007 This quickly became the de facto standard for measuring and comparing data center energy efficiencies. 1MW, how much goes to power IT, switches routers, storage, servers, about 50%
PUE of typical facility is 2.0, DCIE is 50%
No good deed goes unpunished Over the last 2 years, PUE as become very popular as an efficiency metric Google 1.2 Sun 1.28 Microsoft 1.22 De facto talking point EPA Datacenter energy star program, they are adopting PUE as the benchmark for dealing with energy efficiency
Let’s look into the details If you don’t know the definition, the number is meaningless Understand the true application, PUE is a relative measurement, simple math of ratios If you spend a lot of money on both, you could have a good PUE and still be spending a lot of money So this is the issue with simply using PUE as you sole efficiency benchmark It does not address resilient, Google does not have to be tier 4 No trending, over time PUE changes, energy to moving and cooling are is static. However server utilization varies – Think about what happens over the weekend, when everyone leaves and all your servers stop working, party time. There is no correspond change for the cooling so your PUE would take a dive, or in this case jump up. Diversity- All blades using vmotion, you could easily reduce IT load my moving applications and powering down unused blades. In a Heterogeneous environment, you can’t turn off your AS400, that happens to be running one application. Utility – It doesn't address how you use your servers, but your servers is the IT load. It does not benchmark the IT loads
As an example. Not efficient from a business standpoint., You could have a great datacenter from a PUE perspective, but not from a practical business perspective .
None of items on the left will improve you PUE, however they will all improve your power efficiency Reduce IT your PUE will get worse.
Green grid understands and has recognized the problem. And they have taken steps to improve 3 levels of PUE We’re driving level 3, constant time, measure constant at a very low level In the long term there needs to be a more business metric They did not define it, great what is useful work, my wife on the coach thinking about doing work.
Any metric gathering solution for your data center has to be open, and vendor agnostic Hardware is mulit vendor. You’re environment is a heterogeneous mix of vendors. You want your solutions to talk to them all. We have to talk to all products that are sold in North America It can’t be static, it has to be captured over time, if you deployed a bunch of blade servers, what was the impact from a power prospective, from a heat prospective, How was your utilization impacted. How did it impact your CO2 footprint? Kwh Facilities is concerned with reliability how do I not trip a circuit. All I need are the amps. To really understand energy consumption, your going to want to use KwH, your are billed in KwH’s How do I get that data. Per outlet Plan for next build, you don’t know what’s coming around the corner Make sure that your solution is flexible Hardware gear, are facilities are happy, up to code, missed in RFP, does it meet code Leverage what you already have Must popular is APC, has capacity, but not levelageing start gathering data, even that will help you understand you PUE better and start making better decisions.
To really solve the problem, you have to measure and understand your data center at a very Graular level Twinke factor Any major system how many pounds of flour need to arrive, supply Energy comes in, useful output out We run product plants, no metrics in our plant From Uptime A is bill Now we start losing data, many of you have PUE’s with reading but what are you doing with the date, do you trend. Many of you have branch CKT monitoring, but that’s about the extent Create an analogy Green grid states level 3 understand what is that severing doing how much energy is it using. Efficiency of each server is different Fraction of CPU cycles. You have a rack of servers, truck, some are and some are hummers some are priuse. Some hummer are doing little, some prieuse are doing a lot. However, they all look the same. Provisioning is critical, 6KW if you ask it vs facilities you'll get different answers You’ve already paid to cool, 6KW, you might think that your using more and ask for another rack, this is waste Which servers do you want to vitualized, VM has a tool to tell you what the percent utilization is, but your back to the hummer vs prius. Is the hummer working at 5% and the prius running at 50%. You need both, how much energy and utilization.
Another critical component Buying a new CRAC. To save energy increase the set point, there is risk. If you have the data, you can tweak per zone, per rack, per area and sleep well at night.
As it relates to smart racks and power distribution, Raritan has an intelligent PDU offering on the market today consisting of a rack pdu with a really smart controller or computer. This platform when compared to some of what’s on the market is really light years ahead. For example, it has a 5.3x faster processor, 4x more flash memory, 8x more SDRAM. Equivalent to Apple iphone 3G while competitor is equivalent to HP 40G calculator. It even has USB ports, yes, USB ports on a rack PDU! Why would you ever need that? Well if it’s intelligent, why not. We use it for Firmware Upgrades, WiFi networking, and in the future USB camera support and other cool things, maybe even plug in a USB gooseneck lamp if you’re working at the back of a rack and it’s too dark! Actually, I’m charging my phone on the one in the back right now. Embedded software on it is Super Rich in functionality, standards based, open, and ultra extensible. (Based on a linux kernel, running a database, and a webserver with an incredible amount of services, from HTTPS for web browser connectivity, to Telnet or SSH for CLI, to SNMP V1, V2, even V3 with encryption. It can talk to Active Directory an LDAP directory services to ease management of users who can gain rights to it). 1% KWH accuracy is achieved through dedicated energy metering chips per outlet, not shared or multiplexed, or calculated. Its real accumulated kWH data and we are sampling each outlet 40K times a second. Wow, that’s like once every .025 milliseconds. And as you get the data, you can trend, analyze and report against it, slicing and dicing in various ways to understand circuit capacities, power cost, billing business units, and get alarms before bad things happen.
So if this brain and controller existed, why couldn’t we help customers with the age old problem of not knowing what you have and where you have it. In smaller organizations, it could be in someone’s head, in larger organizations, it could be in excel, a home grown system, or a CMDB of some sort not really designed to be a data center operations management tool. Raritan offers an interesting intelligent asset tag and sensor bar solution to help with this problem. Basically you get a S/N coded smart tag that sticks to the server and associate the server to this tag. When the server is installed and the tag is connected to the sensor bar, then voila, the system tells you exactly where that asset is located in the data center, down to the 1U level. Can you imagine how useful this can be in an environment with say 100 racks, 15-20 servers per rack, 1500 - 2000 servers, across multiple business unit owners. You can know which ones are in a rack or not, in production or not. You can even issue work orders that in turn causes the LED to blink for the server you want to work on. (Heck, if it’s connected to an intelligent PDU with sensors, why don’t we change the color of the LEDs base on the inlet temperatures? That’s pretty innovative, what do you think?)
Market problems, landscape, trends, growth Power cost issue (and associated cooling issues) Data center growth chart Power cost growth chart Power cost vs. Industry chart Solution portfolio Yes, it costs more, BUT there is VALUE It ’s not just about the bits and features Outlet metering, so what? Enter PowerIQ Distinctive product solutions The story resonates Today ( “my a lot has changed since we last met”) PX single phase, 3 phase PM series RPC Power IQ Roadmap PX no switching High power for blades Inline meters High density BTO/ETO (show building blocks) Integration with 3 rd parties Key learnings Target customer Facilities vs. data center ops vs. IT Where we are currently at wrt product lifecycle and what that means Example deal, show PO from RA with strips, PIQ, sensors Great Opportunity of legacy customer base and new customer Show growth chart 2 instances at AFCOM, MCA Greater opportunity from new customers JPMC, AVCT shop still opportunity for power Cite all the data center build-out examples going on Gives you new beach-head for KVM Easier door opener b/c people want to talk about it. It is the issue of the day. What next? Learn Leverage, We ’ve got resources, don’t let stone unturned, I’ll get on a plane. Sell more power
Presentation 12/11/12 14:27 Raritan Computers, Inc. Project duration = 9 months competitors included- Server Technology – incumbent (not included in final selection) APC Eaton Avocent Cyber Switching MRV - incumbent (not included in final selection) 3 year project timeline which will include all new construction as well as retrofitting of existing buildings, labs and datacenter facilities. Annual estimates (by Cisco) vary from 5-10K units for both datacenter and lab environment build outs/retrofits.