Detailed large-scale real-time HYPERSIM EMT simulation for
transient stability assessment (TSA)
Towards cloud-based real-time HIL for wide-area special control and protection system testing
ACDC 2023 : Glasgow
Presentation 16:10-16:30 on the 3rd March
Jean Belanger
President and CTO
OPAL-RT TECHNOLOGIES
Montreal, Canada

• Founded in 1997 in Montreal, QC, Canada
• 300+ employees, and growing
• Thousands of users around the world
• 20% of revenue re-invested in R&D
• Real-time simulators for all industries to
design and test control systems
• Fast simulators on HPC/Cloud
• The way to AI applications and digital twins
OPAL-RT in brief … Built to last
-
50
100
150
200
250
300
350
2015 2016 2017 2018 2019 2020 2021 2022
Number of employees
International
Total
Montreal
International development (Sales, support
and R&D) is essential to the growth of our
Montreal Head office
High-quality jobs and sustainability
325
225
100

3
ABSTRACTS: DETAILED LARGE-SCALE REAL-TIME HYPERSIM EMT SIMULATION
FOR TRANSIENT STABILITY ASSESSMENT (TSA).
TOWARDS CLOUD-BASED REAL-TIME HIL FOR WIDE-AREA SPECIAL CONTROL AND PROTECTION SYSTEM TESTING
High penetration of inverter-based Distributed Energy Resources
(DERs), widespread installation of FACTS and HVDC interconnection
systems, and the decommissioning of thermal and nuclear plants are
significantly reducing inertia in large-scale power systems. Fast power-
electronics based control and protection schemes act to stabilize
these systems, but they are sensitive to harmonics, transients, and
system imbalances. It has been shown that simplified positive-
sequence RMS models alone are insufficient for Transient Stability
Assessment (TSA) of large-scale, low-inertia power grids. Therefore,
utilities and regulators such as NERC, as well as professional
associations such as CIGRE and IEEE, have begun investigating detailed
EMT simulation to assess the transient stability of large-scale, low-
inertia power grids that include power-electronic plant controllers.
However, detailed EMT simulation of large-scale power grids for 20 to
30 second time-frames and hundreds of contingencies presents a
number of computational and analytic challenges including excessive
simulation time, large-scale grid data management and the
unavailability of detailed and validated models of power-electronic
plant controllers. Furthermore, these plant controllers, if they are
provided by OEMs, are in the form of blackbox, pre-compiled DLLs,
which are implemented for specific simulation tools, without any
interoperability standard.
2023-03-13
This presentation will describe OPAL-RT solutions to achieve very
large-scale, detailed grid EMT simulation in real-time for Hardware-
in-the-Loop (HIL) / Software-in-the-Loop (SIL) control and
protection testing, as well as quasi-real-time simulation for fast TSA
evaluation of large-scale, low-inertia power systems. With these
solutions, blackbox control and protection systems can be
implemented natively in the EMT simulation tool, HYPERSIM.
PSCAD DLLs can also be co-simulated with HYPERSIM using a
software interface based on the CIGRE model-interoperability
guidelines.
Such advances will accelerate connection studies and can be used
to implement cloud-native tools to help operators assess system
stability with hundreds of contingencies in 5- to 10-minute time-
frames. This performance can be achieved for grids having several
thousand busses with a 50-microsecond time-step using a few
hundred processors.
As HYPERSIM runs under Windows or LINUX, powerful cloud-based
applications can be implemented for TSA and to test wide area
control and protection systems using SIL or HIL with real control
and protection software and hardware. Communication system
emulators, such as eXata can also be used to analyze cyber-attacks
and countermeasures as well as to evaluate the effect of
communication failures and delays on system performance.

4
KEY TAKE AWAYS
 DETAILED EMT SIMULATION IS NEEDED FOR TSA:
Modern power grids integrating large numbers of
inverter-based resources (IBR) will become very
sensitive to control system instabilities. Detailed EMT
simulation will be required to analyse transient system
stability (TSA) of critical low-inertia systems.
 GRID STABILITY IS BECOMING A CONTROL ISSUE, NOT
AN ELECTROMECHANICAL INTERACTION PROBLEM
Control systems will interact with each other as well as with
local protections, HVDC grids, interconnections and wide
area Special Protection and Control System (SPS). These
interactions will dominate the stability of low-inertia grids
 BLACKBOX CONTROLLER EMULATORS
HVDC and inverter-based resource models are now
provided as precompiled black-box models. Utilities
cannot easily develop their own models as was the case
with conventional generators.
 EMT SIMULATION IS REQUIRED FOR VERY LARGE GRIDS
Until a better method is found, TSA will require EMT
simulations to be performed over very large interconnected
territories of several thousands of kilometers and hundreds of
power electronic systems.
 MODEL INTEROPERABILITY STANDARDS
will become essential for TSA and DER interconnection
and planning studies to make sure that blackbox models
can be executed on different simulation platforms
 BLACKBOX MODEL VALIDATION BY INDEPENDENT
ORGANISATIONS WILL BE NECESSARY
to ensure that models used for connections, planning and
system operation studies behave the same as blackbox
models provided by hundreds of OEMs, even after
several controller code updates
 DIGITAL TWINS: NEW EMT-BASED ON-LINE TSA
TOOLS FOR SYSTEM OPERATORS MAY BE REQUIRED
to help system operators to evaluate the grid stability for
hundreds of contingencies
 POWERFULL EMT PARALLEL SIMULATORS WILL BE NEEDED
 IN-HOUSE AND PUBLIC CLOUD COMPUTERS WILL BE
WELCOMED

WINDOWS DLLs CAN NOW BE USED FOR HARD
REAL-TIME HiL SIMULATION AND SiL
 Windows DLLs models on LINUX-RT Container
 No additional hardware
 Microsecond performance
HIL-IN-THE-CLOUD CAN NOW BE USED TO TEST ACTUAL
EQUIPMENT AND SOFTWARE FOR WIDE AREA SPECIAL
PROTECTION AND CONTROL SYSTEMS
 Soft real-time simulation on the cloud compatible with
C37.118 PMU communication protocol
 Use case 1: Grid simulated on the cloud and SPS in the lab
 Use case 2: Grid and SPS on the cloud for SiL
GRID SIMULATION
Multi-rate EMT 50us
AND PMU
MEASUREMENT
SPS Controller Equipment in the lab
C37.118 (20 millis)
PSCAD to HYPERSIM INTERFACE AND UNIFIED MODEL
DATA BASE ARE NOW AVALAIBLE
BREAKING NEWS

6
 WINDOWS DLLs CAN
NOW BE USED FOR HIL
HARD REAL-TIME
SIMULATION WITH
MICROSECOND
PERFORMANCE
BLACKBOX PRECOMPILED OEM
CONTROLLER INTEGRATION WITH
HYPERSIM
Method 1: Direct integration using User Code
Model interface feature
Method 2: Integration using CIGRE B4 or FMU or
SIMULINK interface
Method 3: SiL and real-time HIL simulation using
Windows precompiled DLLs OEM controller models
made for PSCAD or other software using OPAL-RT
LINUX Real-Time Container

Integration of OEM controller code on real-time simulators and for SiL simulation
Method 1 and 2: HYPERSIM NATIVE INTERFACE – CIGRE B4.42 Standard
OPAL-RT provides the UCM interface to easily integrate OEM
controller codes on real-time simulators
- Allows to execute dozens/hundreds of controllers on a single
standard simulator
- Offer the best microsecond performance since our simulators
are equipped with the latest INTEL processors (4GHz). Nanos
performance can be achieved with FPGA-based simulation
- Compatible with any C/C++ and Fortran model source code
- Minimal adaptation is required to recompile the code
for real-time Linux environment
- Support real-time (HIL), offline and accelerated simulation (SIL)
- Cluster model: Deploy code locally or on a remote computer to
accelerate simulation
- User could change parameters during simulation
- Orchestra co-simulation framework with low-latency fabric can
be used to facilitate integration
- Protect and secure IP/code using black box approach
 OEM controller code integrated directly in
HYPERSIM or using standard interface such
as Simulink/CIGRE B4.82/FMU
 Require no additional hardware –
controllers could be simulated on standard
OPAL-RT OP4600 or OP5700 simulators

Thanks to our real-time container allowing to bring windows-
based code from PSCAD or other simulation software into
OPAL-RT real-time simulator.
 Very useful for users who don’t have access to the source
code of the controls
• Reuse the exact Windows and 64 bits or 32bits DLL/Library
of PSCAD or other simulation software in real-time
• No need to recompile the controller code on Linux
• Offer microsecond performance
• Save time by reusing the existing controller code developed
for other software
• OPAL-RT provide a tool to automatically import PSCAD
controls inside HYPERSIM
• HIL and SIL simulation support
• Cluster model: Deploy code locally or on a remote computer
to accelerate simulation
• Allows to execute dozens/hundreds of controllers on a
single standard simulator
Integration of OEM Blackbox Windows pre-compiled controller code (HIL)
METHOD 3: WITH HYPERSIM LINUX REAL-TIME CONTAINER FOR HIL SIMULATION
 OEM pre-compiled PSCAD Windows DLL controller code
integrated directly in HYPERSIM for real-time HIL
simulation
 Require no additional hardware: controllers can be
simulated on available cores of standard OPAL-RT
OP4610 or OP5707 simulators used to simulate the grid.

OPAL-RT invites OEM to import their existing control code to
HYPERSIM real-time simulator (if not done yet)
• OPAL-RT offers free services to help OEM to compile and
integrate controller with OPAL-RT simulation software
• Optimize real-time performance
• Develop user interface for custom block
• Help testing software integration
• Help OEM to protect their IP and codes as black box
• Offer training to learn HYPERSIM and get started with controller
code integration
• Help to compare and validate results with several simulation
software
OPAL-RT also invites customers that need to integrates OEM
controller into HS to contact us.
• OPAL-RT offers advanced support to select best approach for the
integration of controllers
• OPAL-RT could help customers to involve OEM in controller
integration.
Integration of OEM controller code on real-time simulators:
Free services offered to OEMs
OPAL-RT also supports working group CIGRE
B4.82 that intents to standardize interface of
real-time controller code with EMT simulation
software
 Facilitate OEM to reuse their code in
different simulation software
 Facilitate OEM to develop and test their
controller codes for various environment
 Facilitate to support future version of
simulation software
 Facilitate OEM to use as much as possible
actual code that is used in embedded
hardware
The use of such standard shall be
supported and specified by all utilities

SINGLE-SOURCE-OF-TRUTH CENTRALIZED MODEL DATABASE-
Schematic Editor
I/O Configuration
HYPERSIM
ePHASORSIM – CPU-based
electromechanical toolbox
eHS – FPGA-based
electrical toolbox
RT-LAB
ePHASORSIM
ARTEMiS – CPU-based
electrical toolbox
Test Scenarios
Conversion tools
Customer corporate DB interface
Phasor
EMT
Integrated algorithms for
model-to-model interfacing
PSCAD, PSS®E,
EMTP, CYME,
ETAP,
PowerFactory,
Simulink, …
Results &
Analysis PSCAD, PSSe and EMTP
available now
Power Factory and CYME
under progress
Models are a key asset for TSOs

11
 HIL-ON-THE-CLOUD
TESTING OF COMPLEX WIDE AREA SPS
USED ON VERY LARGE GRID CAN NOW
TAKE ADVANTAGE OF CLOUD
COMPUTING TECHNOLOGIES
 Large Grid simulated on the cloud
 SPS hardware in the lab
 PMU signal interfaced to actual hardware
using C37.118 protocol
 Windows or LINUX

HIL-ON-THE CLOUD
Local PC/Laptop
Cloud server
Windows
SEL
Linux
PC
Windows
Windows
PMU meas (C37.118)
PQVI meas (TCP)
HYPERSIM (network)
HYPERSIM GUI
RTlab
(controller)
RTlab GUI
Labview panel (HMI)
Remote
connection
Virtual
machine
lnitiate scenario
CB control (TCP)
P,Q,V, I measurements (TCP)
Scenario control
Industrial controller (SEL)
executing SIMULINK-BASED
wide area control
TCP
SPS CONTROLLER
IN THE LAB

13
DETAILED EMT
SIMULATION NEEDED
FOR TSA
WHY EMT simulation is required for
low-inertia grid – examples
• Evolution of Power Grids
• Transient response examples

14
Evolution of the Grid: Increasing Speed and Complexity
10 ms
50 us
5 us
- High Inertia – slow reaction
- Passive Distribution
- Unidirectional distribution
- Schedulable generation
- Local, slow protection
Pre-1970s
1 s
Relative
Complexity
Transmission Distribution
Time Scale

15
Evolution of the Grid: Increasing Speed and Complexity
10 ms
50 us
5 us
- High Inertia – slow reaction
- Passive Distribution
- Unidirectional distribution
- Schedulable generation
- Local, slow protection
- High Inertia – slow reaction
- Passive Distribution
- Unidirectional distribution
- Schedulable generation
- Fast control and protection
Machine V Regulators
HVDC, FACTS, SVC
Control & protection
- Wide-Area Control & Protection
- Operates near stability limits
- Communication Systems
Pre-1970s Through 21st Century
1 s
Transmission Distribution
Time Scale

16
Evolution of the Grid: Increasing Speed and Complexity
Time Scale
10 ms
50 us
5 us
- High Inertia – slow reaction
- Passive Distribution
- Unidirectional distribution
- Schedulable generation
- Local, slow protection
- High Inertia – slow reaction
- Passive Distribution
- Unidirectional distribution
- Schedulable generation
- Fast control and protection
Machine V Regulators
HVDC, FACTS, SVC
Control & protection
- Wide-Area Control & Protection
- Operates near stability limits
- Communication Systems
- Low inertia – fast reaction
- Active Distribution
- Bi-directional distribution
- Un-schedulable generation
- Stability relies on interaction between:
Fast Protection Systems
Power Electronic Controllers
HVDC, FACTS. SVC
- More Wide Area Control & Comms
- High dependence on communication
systems.
- Large numbers of power-electronics
based Distributed Energy Resources
(DERs)
- Microgrid penetration
Pre-1970s Through 21st Century Today and the Future
1 s
Relative
Complexity
Transmission Distribution

SIMULATION METHOD OVERVIEW
EMT-Electromagnetic
Transients Simulation
Fast EMT
Phasor
domain
Electro-
mechanical
Dynamic
Simulation
(Wide-Area)
17
RMS value
Detailed
waveforms
EMT
Phasor
Power electronic control and protection functions can react to
harmonic and fast transients only visible with EMT simulation

WHY EMT SIMULATION FOR TSA?
• The basic voltage and power control loops of power electronic
system are very simple
• But several additional non-linear control functions are added
to improve performance during large disturbances. (may be
90% of controller/protection code is related to special functions)
• Some controller include self-adaptation of controller gain to
improve performance during variation of power grid short-
circuit conditions and controller limiters if some components
are out of services (degraded mode)
• Several non-linear protection functions are added to protect
components during large disturbances. These protections can
react to fast transients, harmonic and unbalanced conditions
• Analytical method are not yet available to analyse system
stability during large disturbances
• Phase-locked-loops (PLLs) used to synchronize firing pulses,
must be tuned to follow the phase and frequency of the
voltage as fast as possible but without reacting to transients
and harmonics. Several non-linear functions are usually added
to optimise the PLL dynamic behavior during fault conditions
• The settings of these non-linear control
and protection functions are specific to
each OEM and must normally be
included in the blackbox models.
• Several control and protection functions
have time-constants and reaction time
much faster than a few milliseconds,
which is out of range of simplified RMS
models
• Consequently, the dynamic performance
and TSA of grids integrating several
power electronic system cannot be done
using only small-signal analysis. Large
balanced and unbalanced disturbances
must be analysed for several
contingencies
• Blackbox controller models or emulators
must however be validated against field
test of HIL simulation made with
controller replica

19
RMS vs. EMT: HVDC Commutation Failure
Source: AEMO system strength workshop
https://aemo.com.au/en/learn/energy-explained/system-strength-workshop
RMS
EMT
RMS
EMT
RMS
EMT –sustained
small oscillations

20
RMS vs. EMT: Response during the South Australian Blackout 2016
Courtesy of Dr. Babak Badrzadeh
EMT
RMS
RMS simulation
stopped after islanding
EMT
Actual measurement

SIMULATION RESULTS WITH BLACKBOX SVC CONTROLLER
• Sustained fast oscillations observed in the tested scenario at the POC of one SVC plant
• Results with blackbox controllers from PSCAD and HYPERSIM are superimposed
About 2.8 Hz sustained oscillation

SIMULATION RESULTS WITH BLACKBOX PV PLANT CONTROLLER
• Sustained fast oscillations observed in the tested scenario at the POC of
one PV plant
About 4-Hz sustained oscillation

23
 EMT SIMULATION TSA
FOR VERY LARGE
GRIDS IS REQUIRED
 AND IS NOW POSSIBLE
EMT SIMULATION OF LARGE
POWER GRIDS IS NOW POSSIBLE
In real-time or quasi-real-time suitable for on-
line TSA analysis
 EMT SIMULATION TSA FOR VERY LARGE GRIDS IS REQUIRED
Until a better method is found, TSA will require EMT simulations to be
performed over large interconnected territories of several thousands
kilometers and hundreds of power electronic systems.
 MODERN COMMERCIAL MULTI-CORE COMPUTER
TECHNOLOGIES ENABLE LARGE EMT GRID SIMULATION
UNDER WINDOWS OR LINUX WITH HYPERSIM
 EFFICIENT PARALLEL PROCESSING IS THE KEY ENABLING
TECHNOLOGY – Available with HYPERSIM since more than 25
years

Theoretical off-line simulation, cHIL and HIL vs Digital Twins
Theoretical
Off-Line
simulation
with Generic
control models
SiL
Software-in-the-
loop simulation
with real-code
controller
emulation
cHiL
Controller Hardware-
in-the-Loop
Simulation
with control
system replica
EMT Digital Twin
For Transient Dynamic
Stability Assessment
(TSA) with
real-code controller
emulation
System design and EMT analysis
to evaluate equipment stresses, TSA for
DER integration studies (planning)
Detailed protection
and control design
and testing
System operation
Simulator must be
initialized using
state-space estimator
Parallel HPC Processing
nice-to-have
to perform more analyses and tests in
less time to increase test coverage
must-have
to reach real-time to
interface with hardware
must-have
to enable TSA at each
5 to 10 minutes interval
Quasi or faster-than-real-time

SETUP AND RESULTS FOR THE 4000-GRID EMT SIMULATION BENCHMARK
- 30s simulation in 90s wall clock time, 500-core Windows server (LINUX can also be used)
- 50 us time step for the main grid
- 6 or 10 us for DLL controller
1x High Performance 128-core
Windows Computer
22 x High-Performance
4-GHz 18-core Computers
High-speed 5-Gbps
links between
computer
SETUP
MODEL BENCHMARK
Approximate number of components (3-phase)
Buses (3-phase) 4,000
Lines, loads, switched shunts reactors … 6,700
Transformers and machines 2,000
Inverter-based systems (DER and energy storage
systems)
150
Controllers, subsystems using real-code
(precompiled DLLs)
300+
FACTS and HVDC converters 70
Protection relays 100
500+
cores
PERFORMANCE IS
LIMITED BY DLL
CODE
About 100 cores
are used for the
4000-bus grid
simulation and 300
cores for the DLL.

CONTINIOUS RMS SIGNALS CAN BE DISPLAYED IN QUASI- REAL-TIME
EVEN FOR THIS VERY LARGE 4000-BUS GRID EMT SIMULATION
Standard tools actually used to analyse the stability of transients responses obtained
with typical phasor/RMS dynamic stability tools can be used to analyse results from EMT
simulation since fundamental (RMS) transient values computed by PMUs are available

CIGRE MMC HVDC GRID Benchmark
27
Cd-E1
Cb-C2
Ba-A0
Ba-B0
Cb-D1
DC Sym. Monopole
DC Bipole
AC Onshore
AC Offshore
Cable
Overhead line
AC-DC Converter
Station
DC-DC Converter
Station
DCS1
200
200
200
50
300
200
200
400
500
200
300
200
200
200
200
200
100
200
100
200
DCS2
DCS3
Ba-A1
Bm-A1
Bb-A1
Cm-A1
Cb-A1
Bb-C2
Bo-C2
Bo-C1
Bm-C1 Cm-C1
Bb-D1
Bb-E1
Bb-B4
Bb-B2
Bb-B1
Bb-B1x
Bm-B2
Bm-B3 Bm-B5 Bm-F1
Bm-E1
Cm-B2
Cm-B3
Cm-E1
Cm-F1
Bo-D1
Bo-E1
Bo-F1
Cd-B1
Cb-B2
Cb-B1
Ba-B1
Ba-B2
Ba-B3
This system can be simulated in real-time on CPU only at 25 us or with a
multi-rate multi CPU-FPGA simulation at 500 nanoseconds.
1
2
1000 cells per
leg
Half-leg
(or Half-
arm)
Leg/Arm
Ce
ll
C
el
l
S1
S2
R1
C
D2
D1
+
-
500 nanoseconds to simulate
all 6000 MMC cells of an
MMC converters in only one FPGA

LARGE DISTRIBUTION CIRCUITS: 1000 NODES IN 4 US USING 3 FPGA
28
: 333 3-Ph RL connections
: 204 3-Ph RL Loads
: 340 3-Ph Bus Bars
: 9 3-Ph Breaker Circuit
Total NB.
3-ph RL/RC: 537
 Simulated circuit
triple-FPGA based Solution: Six eHS-128 Solvers

29
 DIGITAL TWIN
 ON-LINE TSA TO HELP
SYSTEM OPERATOR
FAST EMT SIMULATION OF LARGE
POWER GRIDS ENABLES THE
IMPLEMENTATION OF DIGITAL
TWINS TO HELP SYSTEM OPERATORS

…
…
.
Electric Grid Digital Twins in the Control Room
STATE ESTIMATOR
Special Control
and Protection System
Dynamic Stability Assessment using
simplified models and pre-simulated
contingencies
(10 to 20 millis resolution)
Ts ≈ 5 to 10 minutes
Optimal Power Flow, Economic dispatch, TSA…
SCADA
Automatic
Control (SPS)
Operator
actions
Conventional TSA using phasor-mode simulation for high-inertia power
grids integrating large thermal and hydro generators

…
…
.
TSA using EMT simulation for low-inertia grid integrating a large quantity of
inverter-based generation system and loads.
STATE ESTIMATOR
Special Control
and Protection System
Dynamic Stability Assessment
(TSA) using simplified RMS
models
and
pre-simulated contingencies
Ts ≈ 5 to 10 minutes
Risk evaluation
Best scenarios
Operator
actions
Automatic
Control (SPS)
SCADA
HYPERSIM
EMT DIGITAL TWIN
(< 50 us resolution)
On-Line
Transient stability analysis
using
high-accuracy
EMT models
Analysis of several
contingencies in parallel
Ts ≈ 5 to 10 minutes
TSA
Initial
conditions
Prediction
EMT Digital Twins in the Control Room for Low-Inertia Grids

32
• EMT Simulation is recommended to evaluate the stability and power
transfer capability of critical low-inertia grids
• Parallel computing, multidomain simulation, and model
interoperability are invaluable for the implementation of EMT electrical
grid digital twins
- PSCAD and other simulation tool models delivered as blackboxes can now be
interfaced with HYPERSIM
- Windows DLLs can now be used as is without any modification for SiL and hard
real-time HIL simulation on the same simulator hardware used to simulate the grid
and other plants
- HPC and cloud technologies can now be used for HIL testing of wide area SPS
hardware and software controllers interfaced with large EMT grid models over the
C37.118 protocol
• Benchmarking tests have demonstrated the feasibility of
high-accuracy on-line EMT digital twins for large power grids
• Standards for model interoperability and test configurations need to be
established
- Enable transition between offline and accelerated/real-time EMT simulation as well
as RMS-EMT co-simulation
- Allow wide-area transient stability assessment considering highly dynamic behaviour
of converter-driven DERs
• Model verification by independent organisations will be needed
- During HIL factory tests, commission and continuously after code modification in the
field
• Digital twins will be useful for system planning and operation
- Enable fast prediction of grid behavior under myriad scenarios
- Enable operators to be proactive, reduce downtime and improve stability
Summary