All the answers on your questions regarding Best Practices for IT/OT Convergence.

1. Best Practices for IT-OT
Convergence Q&A
Frequently asked questions
11 September 2013
Make the most of your energySM

2. Please see the Q&A below. For more information on this topic, or for
answers to any additional questions, please contact any of the presenters:
Jeff Meyers, Smart Grid Strategy & Development, Schneider Electric,
jeff.meyers@telvent.com
John Dirkman, Sr. Product Manager Smart Grid Global, Schneider Electric,
john.dirkman@schneider-electric.com
Fred Fletcher, AGM Power Supply, Burbank Water and Power,
ffletcher@burbankca.gov
Where do you draw the line between IT and OT?
Exactly where the line is drawn can be debated, but for the purpose of our webinar:
Operations Technology (OT) includes the devices used to operate the distribution
system (breakers, reclosers, sensors, relays, etc.), the data and functional interfaces
between pieces of equipment, and the control room applications used to monitor and
operate these devices, like SCADA. These are typically owned and supported by the
business, and are mission critical, requiring 24/7 availability.
Information Technology (IT) includes the systems that run the enterprise (CIS/Billing,
AMI/MDM, GIS, Asset Management, Workflow Management, etc.), and the data and
functional interfaces between equipment and humans in business processes. IT
systems are also typically owned by the business but often supported by others outside
the business, as in a traditional IT group. They are pervasive in utilities today, but may
or may not be considered mission critical.
Words like “intelligent” and “smart” devices in OT means that IT is
already present in OT. Is it about integration or increasing the use of IT
and taking advantage of IT tools to better serve OT?
It is true that there is already some IT in OT, but that intelligence is growing and
becoming much more widely distributed, and that has implications. It’s really about
making the convergence and integration of IT and OT as smooth as possible and
maximizing the benefits from a converged IT/OT solution.
How will data analytics and cyber security play a role in it? How much
time will it take the concept to become mainstream?
Both data analytics and cyber security are an essential part of a successfully converged
system. It is important to consider both during IT/OT projects. Cyber security is really
a topic unto itself, worthy of careful consideration as IT/OT proliferates and the ‘threat
surface’ grows. It’s tough to say when the convergence will become fully mainstream,
perhaps in 3-5 years. Certainly some utilities are further ahead than others.
How is IT/OT convergence facilitating compliance with energy
management systems?
IT/OT convergence is as applicable to energy management systems as it is to
distribution management systems. It’s about integrating the EMS devices and
associated software with IT systems.
Frequently Asked Quetsions | 2
>
>
>
>
General

3. Frequently Asked Quetsions | 3
Analytics
>
>
>
What does Schneider Electric offer in this area?
Schneider Electric offers a wide variety of both IT and OT systems as well as integrated
IT/OT solutions, as shown on our web page: http://www.schneider-electric.com/us/en/
customers/utility/smart-grid.page
Also, you can find a collection of white papers specifically for utilities here: http://www.
schneider-electric.com/sites/corporate/en/support/white-papers/white-papers-electric-
utilities.page
How is data analytics going to play a role in OT/IT integration to bring
value to business? How much time do you think it would take Big
Data to penetrate into the business?
The data generated by both the IT and OT systems/devices needs to be analyzed,
whether by an ADMS or other data analytics engine. Big Data has already penetrated
into the business, and to a degree has been a presence as long as utilities have had IT
and OT systems – it’s just that the volume, variety, and velocity of data is increasing.
Do you see “location/geospatial based” attributes as critical
components of the data analytic capabilities enabled through a
smarter grid?
Yes, we absolutely see location/geospatial based attributes as critical. The asset’s
geospatial location and as well as its relationship and connectivity to other assets – typically
maintained in a GIS and imported into ADMS for example - is absolutely necessary for
proper analysis. We sometimes refer to the GIS-managed view of the network and asset
model, shared across all other systems, as ‘the single version of the truth’.
Is the organization structure changing as well? Who are the leaders in
the business now? Who is my contact person now?
Your contact person will remain the same — our goal is to minimize the impact of our
brand change on you, so that you can focus on your business. The structure of the
organization may change, in time, as new growth opportunities arise for our company
and our employees. For now, our leadership team will remain the same, and will report
to key executives on the Schneider Electric team. We will keep you informed of any
significant changes.
Can you explain the steps of developing a sound architecture of
merged IT&OT?
Before you determine your architecture, you need to determine what you need that
architecture to do. To do this, first determine your Smart Grid business goals and
drivers. Next, develop your Smart Grid roadmap, business case, and detailed Smart
Grid workflows and use cases. After this, developing the architecture/topology can
be completed. You will need to consider various requirements, including uptime,
redundancy, disaster recovery, virtualization, communications, and security. In the
converged world, it is critical to think about both enterprise and real-time integration
tools. It would be best to work with a company like Schneider electric who can help
guide you down this path.
Architectural
>
>

4. Frequently Asked Quetsions | 4
>
>
What are your opinion on use of an Enterprise Modeling language
(ArchiMate)?
Tools for enterprise modeling such as ArchiMate can be very valuable in modeling
enterprise architecture. We have also seen tools like Enterprise Architect and similar
tools used to specifically model the integrations between systems.
Do most organizations have an employee dedicated to architecture?
Large organizations especially have groups of people with the role of architect. In
smaller organizations, individuals fulfill that role, but they may not have the formal title
of “Architect”. Considering the complexity of both OT and IT requirements, it may not
be possible for a single individual to have all the knowledge necessary to guide the
development and maintenance of the architecture.
Is the Enterprise Service Bus (ESB) adequate to meet ADMS
applications or does it need a separate Real-time Service Bus and
why?
Typically, an Enterprise Service Bus is architected for lower bandwidth and higher
latency transmission requirements of data, whereas a Real-time Service Bus (RTSB)
is architected for lower latency and higher bandwidth. While one bus, architected
correctly, could suffice for integrations with a DMS, typically we have seen two buses –
an ESB and an RTSB, where the ESB is between the ADMS and typical IT applications
such as GIS and CIS, and the RTSB typically passes data between the ADMS, SCADA,
and AMI/MDM systems. However, where ADMS and SCADA are tightly integrated as
is the case for Schneider Electric’s ADMS, no RTSB is required to pass data between
these systems.
Please provide specification information on the hardware (and
software) involved.
Sorry, this is a difficult question to answer. The specific hardware and software involved
depends entirely on the Smart Grid drivers at your utility.
What is the best practice for AMI in terms of Metering devices,
communication or Smart Grid Architecture?
AMI is typically integrated with MDM and in turn with CIS/billing and OMS/ADMS
systems. The specific devices, communications, and underlying architecture are
determined by your utility’s requirements. In the early days of the Smart Grid, most
people thought that the communications infrastructure for AMI could also support
all other requirements (e.g., DA, SCADA, OMS). Most architects today believe that
realtime distribution requirements cannot be met, at least not completely met, using
AMI communications. Many grid companies with AMI implementations are also looking
towards some form of operational data store (ODS) to help manage the high-volumes of
metering data and provide a faster integration path for important operations data.
What IT/OT services are appropriate for reliance on cloud technology?
Nearly all typically IT services are appropriate for reliance on cloud technology. The
software side of OT (SCADA for example) is not usually considered appropriate for
cloud technology due to the potential for lower bandwidth and higher latency in
communications. However if the communications and cloud systems can meet
required bandwidth and latency requirements, the software side of OT could also be a
candidate for cloud technology.
>
>
>
>

5. Frequently Asked Quetsions | 5
Burbank
>
>
Does Burbank Water & Power encourage CHP at customer sites?
There are currently no CHP facilities in Burbank
Does Burbank have any transmission or they are a distribution utility?
Burbank has 120MW on the Pacific Intertie linking the Pacific Northwest with southern
California, 108MW of the Southern Transmission System linking central Utah with
southern California, 115MW on the Mead Adelanto 500kV that links southern Nevada
with southern California, and 55MW of the Mead Phoenix 500kV that links southern
Nevada with central and western Arizona, as well as associated transmission
agreements involving Hoover Dam and Palo Verde Nuclear Generating Station.
What would be a business case for IT/OT handling of non-technical
power losses?
The business case for this would need to first identify the potential losses, which might
need to include devices, meters, and/or software required to measure the losses. Typically,
an improved model and the capability to monitor realtime energy including historical energy
delivery will reveal areas of theft and/or commercial errors. Don’t forget, though, to include
the cost of mitigation, such as time and effort to prosecute, in your analysis.
Are there utilities who have recently implemented this IT/OT
Convergence, who would be willing to share their experiences with
us?
Absolutely! One way to do this is to speak with other utilities at conferences, like
Schneider Electric’s LINK conference. Another way is to email John Dirkman (john.
dirkman@schneider-electric.com) and he will set up a meeting on this.
How quickly are OT/IT converging? Are there any examples?
How quickly are OT/IT converging? Are there any examples?
Best approach to merging IT/OT communication and what
organization should communications reside in?
Enterprise Service Buses and Real-time Service Buses are the best way to manage
the communication between IT and OT systems. In our experience, ADMS is often
the central integration system between IT and OT. There is no clear answer on which
organization communications should reside in; the key point is that this organization
must work effectively across the company.
For Fred - when it came time to specify what communication
equipment, network management software, etc. how was that
handled? Did IT spec? OT? A supplier?
Communication equipment, for the utility and the City, has been specified by the
Operational Technology section at the Utility since 1991.
Business case
Case studies
Communications
>
>
>
>
>
>

6. Frequently Asked Quetsions | 6
>
>
Demand side
What is the best communication technology in terms of efficiency, cost
and reliability for monitoring a utility’s operations?
There is no one answer to this question; it depends on the Smart Grid IT and OT
systems in place at or planned for the utility
Will this have any impact on utilities’ efforts to implement IEC 61850
communication in substations?
IT/OT Convergence won’t have a specific impact on implementing IEC 61850. However
IEC 61850 is intended to make integration of devices and automation within substations
easier, and that’s a good thing. Several white papers on IEC 61850 can be found here:
http://www.schneider-electric.com/sites/corporate/en/support/white-papers/white-
papers-electric-utilities.page
What is the impact to Burbank SCADA and OMS systems availability
when big storms or earthquakes bring down the standard IT
communications paths--especially the internet/intranet LAN
infrastructure?
When the communication system was copper this was a problem. The
communications paths performed without interruption in the Northridge earthquake in
1994. The fiber/wireless network is on battery backup and has performed very well
under adverse conditions. It has however occasionally failed, typically due to poor work
practices associated with making system changes.
How has IT/OT convergence enabled auto demand response?
Automated (closed loop) demand response is primarily handled on the OT side based
on analysis and control from a governing system like ADMS. However, data from IT
systems, especially GIS, CIS, and WIS (Weather Information Systems), is typically
required to maximize automated demand response benefits. In the future of the smarter
grid, closed-loop control will enable utilities to specifically target areas for demand
response based on analysis, then monitor the impact of a demand response event and
dial in the network optimization tools to account for load changes.
What has been the success and acceptance levels of HAN applications
for optimizing demand response and reducing energy usage?
There is a wide range of success and acceptance levels of HAN applications for
demand response/energy conservation – some implementations have been more
successful than others. Acceptance has depended on the technology, the methods
of implementation, and the target consumer base. Although there have been pockets
of acceptance of various HAN technologies, as of yet no single HAN application or
technology has bubbled up to the surface enough to be called ‘widely accepted.’ A
number of market factors will have to align for that to happen. But with the volume
of smart metering now in place, it is only a matter of time before home energy
management becomes more mainstream.“
What requirements exist and what is the cost for supply side virtual
power plants which integrate various DERs?
This is a difficult question to answer. The requirements vary by utility and governing
regulatory body, with the costs varying accordingly based on the method of
implementation. ADMS is an excellent solution for integrating DER - watch for a future
webinar on this topic.
>
>
>
>

7. Frequently Asked Quetsions | 7
>
>
Lessons Learned
Worse practices learned -’’Don’t try that in your Utility’’ scenarios?
IT and OT staff located in different locations with few opportunities to interact has
certainly created problems at some utilities. An IT organization that operates as an
independent profit center with OT as an unwilling client – and we have seen this at
some larger utilities - has also created complicated and adversarial dynamics. Starting a
number of small, siloed smart grid projects without an overall roadmap and architecture
is also something we absolutely do not recommend.
I’m curious about the how the utility addressed the internal barriers to
IT/OT collaboration. How did they achieve “buy-in” from all parties in
the utility?
Colocation of IT and OT staff often helps to foster communication and remove internal
barriers. Also cross-training and teambuilding exercises can help individuals from IT and
OT organizations gain a deeper appreciation of the knowledge, skills, and demands of
the other organization. Working through the requirements for an IT/OT roadmap and
architecture is an excellent exercise for facilitating collaboration between the business
and IT personnel.
What is the essential skill that we must learn in team management
before start?
Learning how to facilitate and foster good communication between IT and OT staff is
absolutely essential to an optimal IT/OT convergence.
What advice do you have for IT to get the budget needed to support
convergence?
Sufficient IT (and OT) budgets need to be part of the Smart Grid business case(s) at
your utility. No business case will survive without the support of an executive sponsor.
What criteria are you finding for IT vs. OT roles?
Typically IT staff have more training in computer science, where OT staff typically have
more training in electrical systems – however this isn’t a hard and fast requirement per se.
What does an effective organization chart for OT/IT convergence look like?
IT and OT can be and often are shown as separate organizations on an organization
chart, but a more matrixed organization with IT and OT staff working collaboratively on
projects typically produces the best results.
What have you found is the best way to address the fear from the OT side
related to the emerging IT technology (virtualization, IP networking etc.)
The best way is to communicate the befits of converging IT and OT to your OT staff.
You can also consider providing some IT training – even high- to mid-level – to your OT
staff. Colocation of IT and OT staff also helps, as well as teambuilding exercises.
Organizational and
personnel
>
>
>
>
>
>

8. Frequently Asked Quetsions | 8
>
>
In your experience have you seen where IT and OT are managed by
the same function in the business?
Having IT and OT managed by the same function in the business is very rare. Because
of the skill sets involved, and because IT staff serve more than just OT, IT is almost
always a separate function (or department) in the organization. However, it is becoming
more common to see people with IT knowledge working within the OT organization and
vice-versa.
Collaboration requires a mindset shift on a corporate level. With whom
and how do you start this change process?
You need to start by collaboratively determining the overall Smart Grid goals for your
utility, working with corporate management. It is best to have an executive sponsor to
carry the message to corporate management early in the process, to get their buy-in
and support. Again, establishing a team to work through the requirements for an IT/OT
roadmap and architecture is an excellent exercise for facilitating collaboration between
the business and IT personnel.
You can bring IT and OT personal together for projects, but how do
you see this for day- by-day Operations?
Day-to-day operations will be conducted primarily by OT staff. However IT involvement
is still required for maintenance, support, security, and any required troubleshooting,
so you should expect that any required organizational changes for support of the
converged IT/OT world will be more or less permanent.
What about the IT personnel understanding and being trained about
the operations environment and drivers? We regularly see a bigger
gap there than with operations personnel being familiar with IT
technology
It is important to conduct cross-training, to make sure IT staff are trained to have a basic
understanding of OT and vice versa. However, it may be unrealistic to expect people
with computer science backgrounds to gain sufficient operational knowledge, or to
have power systems personnel become IT experts. Especially considering the timing of
implementations, and the possibility of staff turnover, collaboration is more likely to lead
to success.
Which team selected your Meter Data Management, OT or IT? Are
they responsible for managing the MDM on a daily basis?
Our management approach is that the business unit selects the primary systems, so our
MDM system was selected by the Customer Information System/Billing System division
via a recommendation by subject matter experts and with the advice of BWP metering
technology, BWP distribution engineering, BWP power system operations, and BWP
operational technology.
>
>
>

9. Frequently Asked Quetsions | 9
>
>
Tips to correctly access the effort/resources/time for IT/OT transitions?
Accessing the effort, resources, time, and cost for IT/OT transitions and projects
requires a lot of work and collaboration between the utility and solution providers. The
Project Charter development process at Burbank Water and Power is an excellent
way to clearly define the project, including deliverables, schedule, risks, predecessor/
successor projects, etc. From this a good estimate of effort, resources, time, and cost
can be developed.
What are the best practices in merging the siloed IT and OT functions
that would ensure bridging the connection between cyber security
and physical asset security? Pointing to reference material would,
also, be beneficial.
If your IT and OT systems are secured by different groups, certainly having these groups
work together is essential, whether through colocation, crosstraining, working mutually
on IT/OT projects, or other means. In general, the group responsible for a given asset
(cyber or physical) needs to have their mandate in alignment with the expectations for
the asset. For example, a corporate IT group may have a cultural mandate (as well
as a business mandate) to deliver confidentiality and will make sacrifices to achieve
that. Asking them to also take ownership of assets where availability is the priority can
create conflict and introduces confusion. Rather, two teams with linked awareness
(i.e. sharing data, collaborative meetings, etc) can each execute their own appropriate
responses without trying to weigh conflicting objectives. This is the same reason why
many organizations maintain a separate IT organization to support R&D – it involves a
smaller number of systems with much more complex requirements. They still have the
same overarching objectives - i.e. all systems in the company must be patched - but
the approach may be different between the two groups to reflect the unique needs
and workflows. Also, in the IT/OT roadmapping process described in the webinar,
there are steps for laying out the logical architecture, then overlaying the physical
network infrastructure to help define security requirements. This process can be vital
for clarifying, at least a high level, the potential areas for cyber security treatment. For
reference material, see the answer to a similar question below.
Does virtualization violate the NERC CIP security requirements for
Transmission and Distribution Operations Center Systems?
Virtualization does not violate the NERC CIP security requirements. Separate firewalled
VLANS can be used to isolate Production, QA, DMZ, Corporate, etc. security zones.
Care must be taken when designing virtualized deployments to ensure that virtual
machine storage and redundancy models do not violate Electronic Security Perimeters
(ESP) - for example, introducing virtual machines identified as Critical Cyber Assets to
storage or networking hardware that is not considered part of the ESP.
Planning
Security
>
>

10. Frequently Asked Quetsions | 10
>
>
Testing
Jeff at some time stated that security issues and architecture enabling
true coexistence of both enterprise and mission critical 24/7 parts
of the solution is too complex for inclusion in this webinar. Are you
planning for such webinar, as I not only find this subject interesting
but also believe this is one of the keys in making OT open and IT staff
understand complexity and importance of the OT systems.
Absolutely. We will conduct a webinar on IT/OT security in the future. Meanwhile,
Schneider Electric has historically encouraged utilities to have their IT teams participate
in OT system administration courses, as well as specifically the IT Infrastructure
Integration course, created for IT professionals who need to support OT systems.
When you are referring to Security Suite, can you please mention a
few security standards you are following? Or are we trying to develop
the standards? I know a few standards like NERC-CIP?
Schneider Electric is actively engaged in the development and maintenance of
numerous standards used around the world. Key North American standards used in
the product development and secure design process include: NERC CIP, AGA 12, API
1164, NIST800-53, NIST800-82, and ISA99, for example.
Should new substations be independently tested before energizing to
the grid?
Yes, absolutely. Substations, their devices, and especially OT systems in these
substations should be thoroughly tested before energization
Where does weather forecasting fit into the converged IT/OT solution?
Weather forecasting is a vital predictor for optimized utility operations. Weather imposes
the largest external impact on your Smart Grid - load/demand, renewable energy
supply, and outages are all heavily influenced by weather. Intelligent weather integration
is a key factor in efficient Smart Grid management.
What do you consider economic dispatch?
Economic dispatching allocates generation changes among generator units to achieve
optimum area economy. It provides guidelines for optimal utilization of generating
capacities in order to meet power requirements and minimize fuel cost per generator. In
addition, it provides calculation of desired power and optimum control of Area Control
Error (ACE), considering unit constraints.
Other
>
>
>

11. Schneider Electric USA, Inc.
1390 Piccard Drive
Rockville, MD 20850
Tel: 301-354-5566
Fax: 301-354-5567
www.schneider-electric.com/us
©2012SchneiderElectric.Allrightsreserved.
September 2013