Uptime Institute Fall 2008 EPO alternatives

All Information in This Report is Confidential and Covered by an Information Exchange Agreement
1© 2008 The Uptime Institute, Inc.
EPO Alternatives Project
2008 Accomplishments
Site Uptime® Network
2008 Fall Conference

Project History
In early 2007, a special project was initiated
by the Network to address the significant risk
to uptime attributed to disconnecting means
controls, generally known as emergency
power off (EPO) switches.
At each Network group’s spring 2007
conference, an EPO Alternatives working
session was held.

Spring 2007 Working Sessions
(Continued)
Participants volunteered ideas for each of
two approaches designed to minimize EPO
downtime:
Best practices for working within existing
code requirements
Recommended changes to the codes
The cumulative list of ideas was recorded
and reviewed last fall by the team of
individual Network members who
volunteered to pursue the project’s ongoing
objectives.

Project Team Members
These Network members participated on the
project team:
Mike Bell of Office Depot
Dan Bonner of HP
Matt Brown of HP
John Clemens of HP
Matt Gustafson of SSA
Jack Knavel of HSBC
Mike Lavazza of Cisco
Jerell Myers of Wachovia

Project Team Members (Continued)
Paul Peoples of Target
Gregg Rudinski of Morgan Stanley
Larry Rushing of HP
Joe Stephenson of Wachovia
Wayne Whitcomb of Boeing
David Boston of the Institute
Richard Schlosser of the Institute

Project Objectives
During the fall 2007 conference, project team
members identified four sub-teams that
would work concurrently on the following
deliverables:
Task Team 1: Develop list of safety
enhancements that will result from
accepting our proposed code changes.
Task Team 2: Refine and complete list of
recommended code changes.
Task Team 3: Create best practices paper
for minimizing risks with installed EPO
systems.

Project Objectives (Continued)
Task Team 4: Create a primer for those
who wish to construct a data center to
comply with the general requirements of
NFPA 70 (no EPO requirement).
These teams worked on their objectives
independently as the collective project team
met monthly thereafter.
In the meantime, AFCOM representatives
confirmed they wished to join our project
team. AFCOM had recently been developing
their own plan to propose code changes.

AFCOM Project Team Members
The following AFCOM individuals participated:
Steve McCluer of APC
Tom Roberts of Trinity Health
Rick Sawyer of EYP
William DiBella, AFCOM’s President
Steve was the only project team member with
previous code proposal writing experience.
AFCOM team members met in person with the
Network twice in January. Included was a
working session for the combined project
team in Dallas.

AFCOM
AFCOM was originally the acronym for the
Association for Computer Operations
Management. Now it’s simply AFCOM.
AFCOM members are professionals related
to data center management and operations,
including data center managers, operations
managers, MIS directors, CIOs, CTOs and
other IS/IT professionals.
AFCOM members represent government,
information services, insurance, healthcare,
manufacturing and distribution, financial
services, universities, high-tech, consulting,
telecommunications, utilities, and more.

AFCOM (Continued)
AFCOM provides data center professionals
with networking opportunities and
educational forums and resources through its
annual Data Center World® conferences,
published magazines, regional chapters, and
industry alliances.
The AFCOM membership base represents
more than 4,000 of the world’s largest data
centers.

Optimizing the Code Change
Process
Task Team 2 sought a personal connection
on Code Making Panel 12 to advise us.
After the 2007 fall Network conference, Greg
Dolence of Progressive provided the project
team with a contact from the NEC’s Code
Making Panel 12: Lori Tennant from
Schneider Electric.
This was the connection we needed.
Lori was supportive and recommended we
talk with Joe Sheehan, the NFPA’s staff
liaison for the NEC, to learn how to optimally
work within CMP-12’s process.

Process (Continued)
We met in January 2008 with Joe Sheehan,
who advised us to request a Task Group
from CMP-12 to provide feedback on our
ideas. This would allow us to hear objections
and find compromises before actually
submitting our proposal.
Joe also advised that we inform local
electrical inspectors (often the Authority
Having Jurisdiction) of our intended code
change proposal by attending their regional
meetings in the fall of 2008. (This approach
had helped another group’s proposal achieve
acceptance.)

Process (Continued)
Joe further assisted us with our desire for a
Task Group by officially introducing us to Tim
Croushore, the CMP-12 chairperson.
In late January, Tim Croushore reviewed a
description of our code change proposal and
assigned Bob Johnson, a member of CMP-
12, to head a Task Group to work with us.
The Task Group initially included the
following CMP-12 members:
An electrical utility representative
A UL representative
An electrical inspector

Process (Continued)
An electrical safety engineer
An electrical equipment manufacturer rep.
A manufacturing company’s electrical eng.
In addition:
Two members of NFPA 75 (protection of
information technology equipment)
Four members of NFPA 76 (fire protection
of telecommunications facilities)
One NFPA staff member
Steve McCluer - representing AFCOM
Richard Schlosser and David Boston -
representing the Network

Project Team Progress
Over the winter and spring months, the four
Task Teams made continued progress.
Task Team 1 was the first to complete their
work. Mike Bell, Matt Gustafson, Paul
Peoples, and Rick Sawyer provided a list of
safety enhancements (for firefighters, data
center occupants, and data processing
customers) that would likely be achieved if
the code change proposal is accepted. This
list was passed on to Task Team 2 to include
with their code change proposal.

Project Team Progress (Continued)
This list included:
Emergency services respond
unnecessarily (taken away from real
emergencies) if disconnecting means are
accidentally deployed
People on site are trained not to use it –
may be a life safety issue in a real
emergency
Many data centers directly threaten life
safety if accidentally powered down (those
that support VOIP, 911, medical, security
companies, etc.)

Wayne Whitcomb was a one man “Task
Team 3.” He developed the Best Practices
for Minimizing Downtime Risks with Installed
EPO Systems paper that you have now as a
handout. Wayne utilized the information
collected from spring 2007 Network working
sessions in combination with his own
experience and ideas from the other Project
Team members.
This document should be an immediate
benefit for the majority of members who
operate facilities with installed EPO systems.

This paper addresses various methods and
designs that have been used to satisfy a
wide range of “Authorities Having Jurisdiction
(AHJs)” in their interpretation of the
requirements of NEC Article 645.10.
In particular, it focuses on established “best
practices,” with particular attention given to
the ones that provide the greatest protection
from accidental, inadvertent, or malicious
EPO activation (and are the most likely to be
acceptable to most AHJ’s).

An optimal EPO design must include the
following elements:
Tamper resistant switching device(s) located
only at each principal exit from the room.
Switching devices that are installed at the
same location at each door and away from
other devices like light switches, fire alarms
and door opening devices.
A highly audible alarmed cover mounted
over the switch. (This is documented to be
the greatest deterrent to accidental and
malicious operation and can help to quickly
identify which switch was activated.)

All conduits and j-boxes used for the EPO
system wiring should be clearly marked to
distinguish them from other systems.
Clear and concise labels and procedures
posted at each switch location, in all
applicable languages.
A normally open activation circuit requiring
the application of power to shut down the
room.
A means of bypassing the system in order to
make modifications or to perform
maintenance.

Manual operation only, with no
interconnection to the smoke detection
system or other means of automatic
activation.
A single source of power for tripping the
circuit breakers feeding the IT equipment in
the room.
A simple and reliable design with good
documentation as well as an obvious
configuration.
Segregation into the smallest “zones” or
quantity of affected IT equipment that is
allowed by the code.

The ability to quickly identify which device
has been activated so that the appropriate
response can be given and the device can
be reset quickly.
Cameras focused on each exit door,
including the EPO switch.
Training, including EPO specifics, for all who
work in the data center. This should be
provided the first day on site and at least
annually thereafter.
Install switches that have a unique
appearance instead of a button –type device.

Disconnecting Means Device Cover

Disconnecting Means Device

Disconnecting Means
Device With Cover

These methods have been approved by a
small number of AHJs and may be worth
considering:
A separate EPO system for each of two
dual power paths, when dual power is
provided. Switches and covers may have
different colors for each power path.
Install a single EPO in the main electrical
room or the Command Center instead of
at exit doors.

In summary, these guidelines should help
existing data center owners with EPO
systems optimize EPO related uptime. Some
of the steps listed may be implemented only
when upgrades to the facility are planned.

Joe Stephenson, Matt Brown, Dan Bonner,
John Clemens, Larry Rushing, and Jerell
Myers worked together to develop Task
Team 4’s primer for Constructing a Data
Center Facility To Comply With the General
Requirements of NFPA 70.
This is the second handout that you have.

Data centers can be constructed to meet
enterprise IT requirements without
conforming to NFPA 70 Chapter 6, Article
645, which requires DISCONNECTING
MEANS (commonly referred to as EPO or
Emergency Power Off) controls.
Only chapters 1 through 4 of the NEC are
mandatory. Chapters 5, 6 and 7 apply to
special occupancies, special equipment or
other special conditions in which additional
requirements are necessary or in which
exceptions can be made to the requirements
of the previous four chapters.

Most data centers today are designed to
conform to Chapter 645, which permits
additional flexibility, cost savings, and
convenience when installing communications
and power cabling, with the provision that a
disconnecting means control be provided at
principal exits.
If the electrical system designer follows the
requirements of chapters 1 through 4 only,
the disconnecting means is not a required
part of the electrical distribution system, if the
local Authority Having Jurisdiction (AHJ)
concurs.

More than one Network member has elected
to design and construct their data center
facility to comply with the general
requirements of NFPA 70 (chapters 1-4).
The design team and owner should receive a
written commitment from the local AHJ
before proceeding with this type of data
center design.

Optional design choices to conform to
Chapters 1 through 4 (subject to local
jurisdictions):
Use non-raised floor design with
overhead power and IT cabling
distribution (low voltage overhead
cabling) in the computer rooms.
Or, install only wiring types permissible
below a raised floor used for air handling
(Chapter 3), such as:
Type MI cable (Article 332)

Type MC cable without a covering
(Article 330)
Type AC cable without a nonmetallic
sheath (Article 320)
Other non-metallic cabling shall be
permitted to be installed in EMT
conduit tubing. (Article 300)
Other raceways without a nonmetallic
sheath

If a raised floor is used, power and
communication cabling under the floor
must be enclosed, secured and
protected from physical damage.
Article 300.22(c) (Articles
320,330,332)
Power receptacles, conduit, and
flexible cables are secured in
place.

Secure plenum rated power and
communication cabling in solid bottom
metal cable tray with solid metal
covers. Cables shall be secured in
cable trays in accordance with Article
392.8.
Enclosure and equipment ground must
be in accordance with Article 250;
cable trays in accordance with Article
392.7

Develop a corporate philosophy that
identifies the cost of data center downtime
events for the company and that prescribes
the need to invest the extra labor and
materials necessary to meet one of the two
optional cabling applications defined in the
previous section.
The additional cost will be minimal in
comparison to the potential downtime costs
avoided.

Build data centers to conform to Chapters 1
through 4 wherever AHJ’s support data
center designs without “disconnecting
means.”
Life/safety design strategies that include
incipient and pre-action fire protection
systems are preferred.
Continuous shift Maintenance/Security,
robust BAS monitoring, and comprehensive
personnel training are preferred practices.

Challenges to Success:
Insurance requirements.
Regional/city specific building codes.
Cost/budget constraints.
Education and agreement of AHJ.
Code interpretations, reasonable and
prudent alignment.

In summary, Article 645 is an option that
allows more lenient wiring rules in the IT
equipment space as compared to the other
Chapters of the NEC.
A decision to design and build a data center
solely within the requirements of Chapters 1
through 4 of the NEC (National Electrical
Code) should be made only after the costs
and physical cabling limitations described
above are carefully weighed vs. the potential
costs of downtime related to an EPO event.

Task Team 2 included Gregg Rudinski, John
Knavel, Mike Lavazza, and Steve McCluer.
This team significantly refined and added to
the list of code change proposal ideas that
the Network originally developed in spring
2007 working sessions.
The team’s objective was to provide a
document that captures the concept for the
changes requested, as well as the details on
how the changes could be implemented.
Steve McCluer then incorporated this
information in the first draft of the actual
change proposal.

All Task Teams have completed their original
deliverables, but their members remain as
participants on the overall Project Team.
They will continue to provide input and
feedback to AFCOM/Network CMP-12
Task Group members as the code change
proposal development progresses.
As a next step, the EPO Alternatives Project
Team selected Steve McCluer, Richard
Schlosser, and David Boston to represent
them as members of the CMP-12 Task
Group.

CMP-12’s Task Group met for the first time
on May 15th. At this point, only CMP-12
members and AFCOM/Network members
were included.
CMP-12 asked us to share the history of the
Disconnecting Means requirement, which
Richard Schlosser had previously
researched.
We then presented a matrix and draft
proposal forms which Steve McCluer
authored, based on input from Task Team 2.

CMP-12 members offered a number of ideas
and suggested edits, most designed to
simplify the proposal and make it easier to
understand and enforce for AHJs.
They also asked for several definitions to be
added and for the substitution of preferred
terminology in a few areas.
The Task Group suggested that the
chairperson invite NFPA members to
participate now, instead of making them
aware after CMP-12 reviewed the final
proposal, as they normally would.

NFPA 75 and NFPA 76 representatives
agreed to participate as part of the Task
Group.
A second Task Group meeting was held July
30th by conference call. The group now
numbered fifteen.
The chairperson asked the group what
purpose they felt the disconnecting means
requirement served.
The group was split between those who
thought it no longer served a purpose and
those who did.

The Task Group will therefore not try to
eliminate the requirement at this time.
Much of the meeting’s discussion focused
on aligning definitions and wording
between Article 645, NFPA 75, and NFPA
76. This was very tedious.
Feedback on the AFCOM/Network
change proposal was only received during
the last hour of a five hour call. Most was
detail oriented, rather than conceptual.

Seven subsequent Task Group meetings
were held by conference call from August
through October. Each lasted four hours.
The process involved gaining consensus on
each line of Article 645 of the Code. CMP-
12’s chairperson had dictated that the group
review the entire article.
Unfortunately, this is indicative of the
exhaustive committee work required to effect
changes for codes (and other critical industry
initiatives like energy efficiency policy).

Final Proposal
The Task Group’s final change proposal for
645.10 reads as follows:
An approved means shall be provided to
disconnect power to all electronic
equipment in the information technology
equipment room or in designated zones
within the room. There shall also be a
similar approved means to disconnect the
power to all dedicated HVAC systems
serving the room or designated zones and
shall cause all required fire/smoke
dampers to close. Disconnecting means
shall be implemented by either (A) or (B)
below.

Final Proposal (Continued)
(A) Remote Controls for Disconnecting
Means
1. Remote Controls for Disconnecting
Means shall be at approved locations
readily accessible in case of fire to
authorized personnel and emergency
responders.
2. The Remote Controls for Disconnecting
Means for the control of electronic
equipment power and HVAC systems
shall be grouped and identified. A single
means to control both shall be permitted.

3. Where multiple zones are created, each
zone shall have an approved means to
confine fire or products of combustion to
within the zone.
4. Additional means to prevent unintentional
operations of remote disconnect controls
shall be permitted.
FPN: For further information see NFPA 75-2008,
Standard for the Protection of Information
Technology Equipment.

(B) No Remote Controls for Disconnecting
Means
Remote Controls for Disconnecting Means
shall not be required for critical operations
data systems when all of the following are
met:
1. An approved procedure has been
established and maintained for removing
power and air movement within the room
or zone.
2. Qualified personnel are continuously
available to meet emergency responders
and to advise them of disconnecting
methods.

3. A smoke sensing fire detection system is
in place.
FPN: For further information on early warning fire
detection systems, see NFPA 72-2007,
National Fire Alarm Code.
4. An approved fire suppression system
suitable for the application is in place.
5. Cables under a raised floor other than
branch circuit wiring and power cords are
in compliance with 300.22(C),
725.154(A), 770.154(A), or 800.154(A).
(As required in Chapters 1-4.)

IAEI Education
Separately, AFCOM and Network project
team members have handed out a summary
of our proposal effort to electrical inspectors
at five regional IAEI conferences this fall.
At each of these conferences, we worked
from a desk or booth in the exhibit hall.
We encouraged discussion and input from
the inspectors (many of whom are AHJs)
regarding our proposal effort.
We believe this effort will help when the
proposal review process occurs.

Additional Assistance - ITIC
The Information Technology Industry Council
(ITIC) is made up of computer hardware and
software manufacturers.
They are frequently involved in change
efforts with national organizations and code
bodies.
ITIC has indicated they will support our
effort. We recently sent them a copy of our
final proposal and asked that they help
where they can.
They have at least two individuals with
experience participating on the code panels.

Next Steps
Because we reached consensus on the
proposal wording before the proposal
deadline, the CMP-12 Task Group has
adopted our proposal and recently submitted
it themselves.
This is considered a significant benefit, as
it should carry a strong endorsement with
the rest of the proposal review group.
As members of the Task Group, we will have
one AFCOM/Network representative present
at the Code Making Panel meetings (review
of proposals) in January, 2009.

Next Steps (Continued)
From that point forward, we will monitor the
proceedings and update Network members
as we learn of decisions or feedback on our
proposal.
If we see opportunities to help by asking
Network, AFCOM, and ITIC members to
share comments in support of the
proposal, we will notify members of each
group.
The projected schedule for the Code change
process follows:

Projected Schedule – Code Changes
This cycle for changes is the 2011 NEC.
The closing date for public proposals for
changes to the 2011 NEC is November 7,
2008.
Code Making Panel meetings will conclude
January 24, 2009.
NFPA’s Report on Proposals - July 14, 2009.
NFPA’s closing date for receipt of comments
– October 23, 2009.

Projected Schedule (Continued)
Code Making Panel meetings close (review
of comments) – December 14, 2009.
NFPA publishes Report on Comments –
March 28, 2010.
Intent to Make a Motion closing date –
May 7, 2010.
Association Meeting – June 5-9, 2010.
Implementation of Changes Accepted –
issued January 2011.
Adopted into Law by Local Jurisdictions –
months to years later.

Annual Reminder
Each year, our AIR trending report shows that
several member sites still:
Do not disable local PDU EPOs.
Do not ensure the fire system has no
connection to the EPO.
Do not provide audibly alarmed covers,
labels and consistent training (to all who will
enter the computer room) for EPO switches.
Have you verified these conditions do not exist
at your facility?

In Closing
If successful, it will be 2011 or later before our
proposed Code changes are effective. The
benefits will be substantial for the industry.
The immediate benefits for our members are
the two documents you received today.
You may implement Best Practices for
Minimizing Downtime Risks with Installed
EPO Systems as soon as you identify a plan.
You may choose to build future data centers
that do not require Disconnecting Means, as
described in Constructing a Data Center
Facility To Comply With the General
Requirements of NFPA 70.

In Closing (Continued)
Please join me in thanking all our Project
Team members, who have contributed many
hours in addition to their “real” jobs:
Mike Bell of Office Depot
Dan Bonner of HP
Matt Brown of HP
John Clemens of HP
Matt Gustafson of SSA
Jack Knavel of HSBC
Mike Lavazza of Cisco
Steve McCluer of APC

Jerell Myers of Wachovia
Paul Peoples of Target
Tom Roberts of Trinity Health
Gregg Rudinski of Morgan Stanley
Rick Sawyer of EYP
Richard Schlosser of the Institute
Joe Stephenson of Wachovia
Wayne Whitcomb of Boeing
It has been a particular pleasure to team with
members of AFCOM for the first time.

Questions?

Uptime Institute Fall 2008 EPO alternatives

Recommended

Recommended

More Related Content

Similar to Uptime Institute Fall 2008 EPO alternatives

Similar to Uptime Institute Fall 2008 EPO alternatives (20)

Recently uploaded

Recently uploaded (20)

Uptime Institute Fall 2008 EPO alternatives