The document provides an agenda and overview of Ulticom's Signalware SIGTRAN software as it relates to Vodafone Italy's NP project. It discusses SIGTRAN and its components SCTP and M3UA, and how Signalware implements these protocols. It describes Signalware's architecture, and how M3UA is used between the application server and signaling gateway. It also covers installation, configuration, operation and monitoring of Signalware M3UA.
4. 4www.eng.it
Ulticom’s Signalware SIGTRAN Overview
» SIGTRAN (SIGnalling TRANsport) is a Framework Architecture for Signaling
Transport defined by IETF within the SIGTRAN working group (RFC2719 – Oct
1999)
» The framework defines the architecture and the functional requirements for transport of
signalling information over IP.
» A major application of SIGTRAN framework concerns SS7 signalling; in this case, a set
of protocols (Adaptation Layers) has been defined by SIGTRAN WG
(xxUA, xxPA protocols).
» Besides, the specific needs of signalling transport over a connection-less, unreliable
packet network, have led to the introduction of a new transport protocol (SCTP)
5. 5www.eng.it
Signalware SIGTRAN Architecture
» Signalware SIGTRAN is an integrated platform
that provides an environment for the
development and deployment of SS7
applications and services over IP
» Signalware is a middleware between the
platform (HW platform/OS) and the
applications.
» In order to grant high availability and
scalability, HW platform is a cluster made up
by 1 to 8 CEs (Computer Equipments), which
loadshare traffic and minimize service
disruption.
» Applications may be configured to run on 1 to
8 LNs (Logical Nodes)
- LN:an instance of a protocol stack and a
unique Point Code (PC).
» Applications exploit an API to access
Signalware transport service (legacy
SS7/MTP or IP based transport)
HW platform/Operating System
Signalware
LN
SS7 Application
LN
SS7 Application
CE1 CEn… …
6. 6www.eng.it
Signalware architecture in NP application
» The NP application is built on
top of Signalware:
» The stack is hosted on a single
IBM x3755 server with Solaris 10
Operating System (FMP node)
» The whole NP solution consists
of several applications running
over Signalware platform
» Signalware provides
M3UA/SCCP layers, while above
layers are part of the NP
application (SCTP is embedded
in Solaris 10)
IBM x3755 / Solaris 10
Signalware
LN
SS7 Application
(Atos Platform, FMRDB, NP, SSA)
CE
xxFMP0x
8. 8www.eng.it
SCTP Overview
» Stream Control Transmission Protocol
» IETF standard
- RFC 2960
Stream Control Transmission Protocol (October 2000)
- RFC 3309
Stream Control Transmission Protocol (SCTP) Checksum Change
(September 2002)
- RFC 4960
Stream Control Transmission Protocol (September 2007)
- RFC 4460 (*)
Stream Control Transmission Protocol (SCTP) Specification, Errata
and Issues (September 2007)
(*) Signalware is compliant with RFC2960 and RFC3309, compliance with
RFC4960 is still not implemented (no compatibility issues)
9. 9www.eng.it
SCTP Overview
» SCTP is a reliable transport protocol operating specifically designed
for signalling transport on top of a connectionless packet network
such as IP
» TCP is not suited for signalling transport (why?)
» SCTP features
» Connection oriented (associations)
- Heartbeats
- Flow Control
» Message based
» Orderered and out of order message sending (within streams)
- No head-of-line blocking
» Multihoming
- Increased robustness against TCP
» DoS Protection
- Cookies
- 4-way handshake
10. 10www.eng.it
Functional view of SCTP
Association
Startup
and
Takedown
Sequenced delivery
within streams
User Data
Fragmentation
Acknowledgement and
Congestion Avoidance
ChunkBundling
PacketValidation
PathManagement
The SCTP transport service
can be decomposed into a
number of functions
11. 11www.eng.it
SCTP Packets
» An SCTP packet is composed of a common
header and one or more chunks.
» Valid chunk types:
» 0 Payload Data (DATA)
» 1 Initiation (INIT)
» 2 Initiation Acknowledgement (INIT ACK)
» 3 Selective Acknowledgement (SACK)
» 4 Heartbeat Request (HEARTBEAT)
» 5 Heartbeat Acknowledgement
(HEARTBEAT ACK)
» 6 Abort (ABORT)
» 7 Shutdown (SHUTDOWN)
» 8 Shutdown Acknowledgement
(SHUTDOWN ACK)
» 9 Operation Error (ERROR)
» 10 State Cookie (COOKIE ECHO)
» 11 Cookie Acknowledgement (COOKIE
ACK)
» 12 Reserved for Explicit Congestion
Notification Echo (ECNE)
» 13 Reserved for Congestion Window
Reduced (CWR)
» 14 Shutdown Complete (SHUTDOWN
COMPLETE)
» …
14. 14www.eng.it
M3UA Overview
» Message Transfer Part 3 (MTP3) - User Adaptation Layer
» IETF standard
- RFC 3332Signaling System 7 (SS7) Message Transfer Part 3 (MTP3) –
User Adaptation Layer (M3UA) (September 2002)
- RFC 4666Signaling System 7 (SS7) Message Transfer Part 3 (MTP3) –
User Adaptation Layer (M3UA) (September 2006)
(*) Minor changes between RFC3332 and RFC4666 (same protocol version,
i.e. 1.0)
(**)Signalware is compliant with RFC3332, compliance with RFC4666 is still
not implemented (no compatibility issues)
15. 15www.eng.it
M3UA Overview
» M3UA supports the transport of SS7 MTP3 user signaling over IP using the
services of the SCTP:
» The list of these protocol layers includes, but is not limited to, ISUP, SCCP, and
telephone user part (TUP)
» The M3UA layer provides the same interface to its upper layer as MTP3 does:
- In this way, MTP3 user is not aware that MTP3 services are offered remotely
from an MTP3 layer at an SG, and not by a local MTP3 layer.
- Similarly, MTP3 layer at the SG may also be unaware that its local users are
actually remote user parts over M3UA.
- In practice, the M3UA extends access to the MTP3 layer services to a
remote IP–based application.
16. 16www.eng.it
M3UA Network Scenarios
» Two distinct scenarios:
» AS-SG interconnection
- In the AS, M3UA replaces MTP3,
allowing ISUP, SCCP and TCAP based
application to operate without an
underlying SS7 network(*)
- In the SG, M3UA gives SS7 network
access to applications located on IP
networks (i.e. it enables signaling
transport between SS7 network and
new-generation IP networks)
» IPSP-IPSP interconnection
-M3UA enables SS7 applications in an
all-IP environment (same as an AS
without an SS7 network)
(*) The AS may or may not have an own
Signalling Point Code (e.g the PC may not be
needed if there is one-to-one correspondence
between AS and SG)
Legacy SS7
node (e.g.
MSC)
xxAP
NIF
MTP3
MTP2
MTP1
M3UA
SCTP
IP
M3UA
SCTP
IP
xxAP
TCAP
SCCP
Application
Server (AS)Signalling
Gateway
(SG)
SS
7
SS
7
IPIP
TCAP
SCCP
MTP3
MTP2
MTP1
M3UA
SCTP
IP
xxAP
TCAP
SCCP
IP Signalling
Point (IPSP)
IPIP
M3UA
SCTP
IP
xxAP
TCAP
SCCP
IP Signalling
Point (IPSP)
17. 17www.eng.it
M3UA Key Concepts
» Application Server (AS)
A logical entity handling a specific type of traffic (e.g. a virtual data base element, or the FMP
node in the NP Network Scenario). The AS contains a set of one or more unique Application
Server Processes, of which one or more is normally actively processing traffic.
» Application Server Process (ASP)
A process instance handling traffic for an AS
» Signaling Gateway (SG)
Virtual entity that acts as a bridge between IP and SS7 networks. It is perceived as an SS7
Point Code within the SS7 network.
» Signaling Gateway Process (SGP)
Process instance handling traffic within the SG
» IP Server Process (IPSP)
An ASP that interacts with other ASPs (instead of SGPs).
» Routing Key (RK)
Set of SS7 parameters that identify the type of traffic handled by an AS (DPC, SI,…). An AS
registers a RK with a SG or with another AS (there is one-to-one correspondence between an
AS and a RK)
» Routing Context (RC)
32 bits RK identifier
19. 19www.eng.it
M3UA Activation (single ASP in single AS)
» The assumption is that the SCTP
association has been already
established prior to M3UA message
exchange
» Routing Key Registration is optional
(e.g. Eagle5 does not support it,
static provisioning is needed)
» The ASP Up message is used to
indicate to the remote peer that
M3UA is ready to receive any
ASPSM/ASPTM messages for all
Routing Keys that the ASP is
configured to serve.
» The ASP Active message is sent by
an ASP to indicate to a remote M3UA
peer that it is ready to process
signalling traffic for a particular
Application Server
» As an example grey text reports
Signalware MML commands that
trigger M3UA ASPSM/ASPTM
messages.
RegReq (LRK=x,RK=y)
RegRsp (LRK=x,RC=z)
ASPUP
ASPUP ACK
ASP SGP
ASPAC (RC=z)
ASPAC ACK (RC=z)
Notify (AS-ACTIVE)
CREATE-M3UA-RKEY
ACTIVATE-M3UA-RKEY
ACTIVATE-M3UA-LSET
Notify (AS-INACTIVE)
20. 20www.eng.it
M3UA Activation (2 ASP in single AS, Loadsharing)
» The graph on the left depicts a
scenario with 2 ASPs in a single
Application Server, which:
» handle traffic in loadsharing mode
(ASPAC message):
- this means that the SGP can direct traffic
to all ASPs active at a certain moment in
time, choosing them according to an
implementation dependent algorithm.
» are configured in “1+1” sparing:
- More generally, failover model supports
an "n+k" redundancy model, where "n"
ASPs is the minimum number of
redundant ASPs required to handle traffic
and "k" ASPs are available to take over
for a failed or unavailable ASP.
ASPUP
ASPUP ACK
ASP1 SGP
ASPAC
(LoadSHR)
ASPAC ACK
Notify (AS-ACTIVE)
ASP2
ASPUP
ASPUP ACK
ASPAC
(LoadSHR)
ASPAC ACK
Notify (AS-ACTIVE)
21. 21www.eng.it
M3UA Signalling
» The graph on the left depicts a
scenario where an SS7 Legacy Node
(e.g. MSC) exchange SS7 signalling
with an Application Server (e.g. FMP
node) through a Signalling Gateway
TFP
SS7
legacy node ASSG
DUNA
DAUD
DUNA
TFA
DAVA
MSU
DATA
MSU
DATA
22. 22www.eng.it
M3UA Signalling
» The graph on the left depicts a
scenario where a local SSN on the AS
becomes prohibited
SSP
AS SG
SSA
MSU
SST
SST
MSU
SST
24. 24www.eng.it
Signalware M3UA: Installation, Configuration,
Operation & Monitoring
» Signalware product is composed of a set of packages that are automatically
installed, removed, checked and repaired through automated scripts (shipped as
part of the whole software package)
» This is the set of packages installed on Vodafone xxFMP0y nodes:
- OMNI (base system)
- OMNI-X (support for application development)
- OMNI-GUI (GUI support)
- OMNI-MAN (man pages)
- OMNI-OLU (support for on line upgrade)
- OMNI-C3 (ITU MTP and drivers)
- OMNI-C7X (ITU MTP development)
- OMNI-C4 (ITU SCCP)
- OMNI-AC7 (ANSI TCAP over ITU SCCP/MTP drivers)
- OMNI-C3M (ITU M3UA)
- OMNI-SCTP (Stream Control Transmission Protocol)
25. 25www.eng.it
Signalware M3UA: Installation
» Signalware installation requires the creation of a special user account for installation,
operation and provisioning (omni on Vodafone xxFMP0y nodes)
» A valid license file is also needed:
» this file (.lic extension) is provided by Ulticom through Ulticom portal
» it must be copied into /etc directory before installations
» it is an XML file customized for the target machine on which Signalware will be installed
(host name and host id)
<?xml version="1.0" encoding="US-ASCII" standalone="yes"?>
<LICENSE>
<CUSTID>
LSN: P507000S090028
…
<CE>
<NAME>t1fmp01</NAME>
<HOSTID>3856b52e-SunOS-t1fmp01-7D2AF259E968E1BAF756BFDBD0</HOSTID>
…
<ITEM TYPE="FEATURE">
<NAME>M3UA</NAME>
<VALUE>ENABLED</VALUE>
</ITEM>
…
-----BEGIN SIGNATURE-----
iOKdfu2H0mm1nHOIWtVlrFgmAtTwsOF/SGnxGKyFLhPXJKhVT5974wC7z+oThEhb3J9YC9EXW9/h
…
-----END SIGNATURE-----
26. 26www.eng.it
Signalware M3UA: Installation
» Installation can be done in 3 ways:
» Ulticom Menu Interface (script swsetup provided in the software bundle)
- This is strongly recommended
- Details on installation and configuration can be found in the following
document:
» Ulticom command line interface (swxxxx scripts)
» Native Solaris installation tools (pkgadd, pkgrm commands), taking
dependencies into account
SW Installation
Configuration
27. 27www.eng.it
Signalware characteristics & capabilities
» Features enabled based on the purchased license:
» In Vodafone, the following protocols/variants are available according to the
shipped license:
- M3UA
- SCCP
- TCAP
- ATM
available in C7, A7, CH7 and J7 variants (even if not all has been installed)
» The following protocols/variants are not available with Vodafone license (even if
they are part of the Ulticom Signalware suite)
- M2PA
- M2UA
- SUA
- INAP
- TUP, ISUP
- CAP
- SIP
- Diameter
and many others…
28. 28www.eng.it
Signalware characteristics & capabilities
» The main provisioning limits affecting Signalware M3UA
configuration are reported below:
- Up to 8 Logical Nodes (LN)
- Up to 256 M3UA links per CE for all nodes
- Up to 32 SCTP streams per association
- Up to 10,000 Route SETs (RSETs) per platform node
- Up to 255 Link SETs (LSETs) per platform node
- Up to 16 LSETs per RSET
- Up to 4,095 Routing Keys per platform node (default is 512)
- Outbound GTT entry number is in range [1000 - 20000] (default is
1000)
- Inbound GTT entry number defaults to 200
29. 29www.eng.it
Signalware Configuration
» There are several configuration files, located in $OMNI_HOME/conf (i.e.
/users/omni/conf on xxFMP0y nodes).
» Values specified here overwrite default configuration parameters:
- omni_conf_info
Definition of general Signalware parameters, such as:
MEAS_30MIN_TIMER=5 (interval interval by which 30-minute protocol
measurements are collected)
- usam_conf_info
Definition of USAM specific parameters, which is the Signalware SCTP association manager for
M3UA and SUA., such as:
MAX_NB_THREAD=127 (Maximum number of threads in charge of establishing
outbound SCTP associations)
- m3ua_conf_info.<SHM>
Definition of M3UA specific parameters, such as:
M3UA_SLK_ADDR_SORTED=FALSE (If FALSE, SCTP addresses are kept as
provisioned by CREATE-M3UA-SLK MML command → SCTP primary path )
SCTP_PATH_REXMIT=2 (maximun path retransmission)
SCTP_HB_INTERVAL=500000 (Heartbeat interval set to 500 ms)
» For each file there is a detailed man page which provides an explanation for all
configuration parameters
» When changes are made to those files, Signalware must be terminated and
restarted
30. 30www.eng.it
Signalware Configuration
» Signalware over Solaris 10 exploits the SCTP protocol stack
embedded in the Operating System:
» Some configuration parameters must be configured directly in the Kernel
(Solaris SCTP tunable parameters):
- /etc/init.d/ndd-set (or similar startup script)
ndd –set /dev/sctp sctp_pa_max_retr 4
Controls the maximum number of retransmissions (over all paths) for an SCTP
association. The SCTP association is closed when this number is exceeded.
» Priority order when considering SCTP Configuration Parameters
- SCTP tunable parameters
- m3ua_conf_info settings overwrite previous ones
- MML commands specific settings overwrite all the previous ones (association
specific parameters in CREATE-M3UA-SLK command)
31. 31www.eng.it
Signalware Operation
» NP application requires that Signalware is controlled through the
Solaris Service Management facility (not started manually):
» Services:
svc:/application/Ulticom/DFdaemon:default
svc:/application/Ulticom/Signalware:default
controls respectively the Ulticom Distributed File System and the core
Signalware processes
» Manifest files are stored in /var/svc/manifest/application/Ulticom/
» Signalware is started through:
/usr/sbin/svcadm enable DFdaemon
/usr/sbin/svcadm enable Signalware
and stopped through:
/usr/sbin/svcadm disable Signalware
/usr/sbin/svcadm disable DFdaemon
» Status can be checked through:
svcs DFdaemon
svcs Signalware
33. 33www.eng.it
Signalware Processes
» Solaris Service Management facility controls both DFdaemon and Signalware
execution, therefore they are started by root process:
omni 22137 1 0 16:15:40 ? 0:00 /bin/bash -p /users/omni/bin/go.omni
root 21863 1 0 16:14:32 ? 0:25 /users/omni/bin/DFdaemon -D
» The first process to be started by go.omni is the Parent Operation Process (POP),
which in turn starts all the others:
root 22844 22137 0 16:15:43 ? 0:17 /users/omni/bin/pop tmp/start.100.22137
...
omni 22850 22844 0 16:15:43 ? 0:01 /users/omni/bin/mon –start
omni 22867 22844 0 16:15:44 ? 0:03 /users/omni/bin/sw_nm -protocol C7 -nokill –start
omni 22869 22844 0 16:15:44 ? 0:02 /users/omni/bin/c7tcmg -start
...
» A detailed explanation for the main Signalware processes can be found here.
» A man page is available for each of the processes listed in the previous slide.
34. 34www.eng.it
Signalware Provisioning
» Provisioning is done through MML commands/files
» Signalware provides an enhanced terminal handler, called swmml
» swmml can be used in two different ways:
- Interactive mode:
- it provides an interactive environment where commands can be entered manually
- Available features include auto-completion (both for commands and arguments),
command history, on-line help (man pages)
+---------+ Signalware Enhanced Terminal Handler v9.41
| swmml | Copyright 1993-2008 Ulticom, Inc.
+---------+ All Rights Reserved
Welcome to swmml. To get some help, type 'man'.
Current editing mode set to: emacs mode.
N100:t1fmp01:100 >
- Batch mode:
- swmml –e ‘<MML command>’ allows the execution of a single MML command
- swmml –f <MML script> allows the execution of an MML script
- swmml –x <ext. MML script> allows the execution of an extended MML script
(i.e. a script which combines MML commands with PERL statements)
The same script can be entered in interactive mode through RUN-MML-SCRIPT
- In case of multiple logical nodes, swmml –n <node_name> is used
35. 35www.eng.it
Signalware Provisioning
» Provisioning order for a Logical Node configured as an M3UA
Application Server
» Create Own Point Code ⇒ CREATE-OSPC
» Create IP Link SETs ⇒ CREATE-M3UA-LSET
» Create IP links (i.e. associations) ⇒ CREATE-M3UA-SLK
» Create Route SETs ⇒ CREATE-RSET
» Create M3UA Routing Keys ⇒ CREATE-M3UA-RKEY
» Create remote SSNs (if needed) ⇒ CREATE-REMSSN
» Create Concerned Point Codes ⇒ CREATE-OSPC
» Allow Route SETs ⇒ ALLOW-RSET
» Activate Link SETs (or directly IP links) ⇒ ACTIVATE-M3UA-LSET or
ACTIVATE-M3UA-SLK
» Activate routing keys ⇒ ACTIVATE-M3UA-RKEY
» Define rules for GTT (if needed) ⇒ CREATE-GT
» For each command there is a detailed man page that explain how to use it
» Click here to see the configuration of MIFMP01 node
36. 36www.eng.it
Signalware Provisioning
» Similarly, there is an unprovisioning order for the LN configured as
M3UA AS
» Deactivate routing keys ⇒ DEACTIVATE-M3UA-RKEY
» Deactivate Link SETs ⇒ DEACTIVATE-M3UA-LSET
» Inhibit Route SETs ⇒ INHIBIT-RSET
» Delete Concerned Point Codes ⇒ DELETE-CPC
» Delete remote SSNs ⇒ DELETE-REMSSN
» Delete Routing Keys ⇒ DELETE-M3UA-RKEY
» Delete Route SETs ⇒ DELETE-RSET
» Delete IP links ⇒ DELETE-M3UA-LSET
» Delete IP Link SETs ⇒ DELETE-M3UA-SLK
» Delete Own Point Code ⇒ DELETE-OSPC
37. 37www.eng.it
Monitoring and Troubleshooting
» There is a set of logs in $OMNI_HOME/Logs directory (i.e. /users/omni/Logs)
» Monitor.log.<SHM>.<file_number>
- Log collected by the mon process
- Log file in displayable HEX/ASCII format. <SHM> is the shared memory value and
<file_number> is a five digit progressive number (numbering begins from a file number, when the
file reaches 2 Mbytes it is closed, the file number is increased and a new log file is created).
- Files are purged after a given time period (see DISPLAY-PURGE and SCHEDULE-PURGE
commands)
- Those files logs messages exchanged between Signalware processes
» Monitor.pcap.<SHM>.<file_number>
- Log collected by the mon process
- Log file that contains monitored SS7 Message Signaling Units (MSUs) in PCAP format (to be
read through Ethereal/Wireshark)
- The .pcap file is closed along with the corresponding .log (files are always synchronized)
- NEVER FORGET: This is a facility that logs SS7 messages as seen by the application, but
troubleshooters are encouraged to capture “real” SCTP packets directly from the network
» The MonControl and the monitor utilities can be used to handle and to configure mon
process logging
38. 38www.eng.it
Monitoring and Troubleshooting
» A set of logs in $OMNI_HOME/Logs directory (cont.):
» Logging into the previous files van be controlled through the MonControl and
the monitor utilities (see man pages for details):
- The MonControl utility controls the monitoring activity of mon process. It allows:
- Enabling/disabling monitoring of specific user space processes (or all)
- Logging in both ASCII and pcap files, or only one of them
- Force flushing or purge&open
- The monitor standalone process enables/disables monitoring in the UNIX Streams
wrapper environment.
- Monitor commands are limited to L3RT, SCCP, ISUP, and TCAP stream routers
- monitoring can be enabled at four points: message entering the module on the
write side, message exiting the module on the write side, message entering the
module on the read side, and message exiting the module on the read side.
39. 39www.eng.it
Monitoring and Troubleshooting
» A set of logs in $OMNI_HOME/Logs directory (cont.)
» usam_trace.<mmdd>.<ext>
- Log collected by the usam_daemon process (SCTP protocol log)
- The samdiag utility allows to change the trace level and also to see contents of the usam shared
memory (internal structures. etc.)
- When an error (errno) is displayed, refer to /usr/include/sys/errno.h to check the meaning
For example, if usam_trace.1023.1 contains:
2008/10/23 23:31:50 | ERROR | sam_create_connect | sctp_connect error, errno 145
then
cat /usr/include/sys/errno.h | grep 145
shows
#define ETIMEDOUT 145 /* Connection Timed Out */
» Event.<SHM>.diag.<mmdd>.<ext>
- Collected daily (from 00:00 to 24:00) from log process
- if size increases over a given limit, by default set to 10 Mbytes, the files are rotated
- The trace level can be set through MML commands (SET-M3UA-TRACE, SET-SUA-TRACE)
» Other log files:
» /data/omni/omnilog
this is the file to which Signalware output is redirected.
» $OMNI_HOME/<ce name>/tmp/Event.<SHM>.txt.<mmdd>.<ext>
(e.g. /users/omni/t1fmp01/tmp/Event.100.txt.1105.1)
this file contains an ASCII string for each logged event
40. 40www.eng.it
Monitoring and Troubleshooting
» Diagnostic tools:
» c7m3uadiag
- The c7m3uadiag tool allows the user to display the data structures and other relevant information
of all M3UA modules. It performs the following:
- Displays M3UA shared memory table
- Displays traces
- Displays M3UA kernel virtual memory
- Displays relevant information (e.g., global variables, size of data structures)
- It is a menu driven utility
- Navigating through sub-menus it allows a lot of operations (e.g. it shows last messages,
measurements, RK filters, etc.)
- See man page for details
» c7sccp_dump
- A menu driven tool similar to the previous one, it is used to examine internal SCCP runtime
information
- See man page for details
» samdiag
- The samdiag tool is a menu driven utility that performs the following
- Displays USAM shared memory (i.e. SCTP data structures, users, remote peers,
configuration data, etc.);
- Change of USAM trace level (up to single function calls)
- See man page for details
41. 41www.eng.it
Monitoring and Troubleshooting
» Diagnostic tools (cont.)
» c7MonDecode
- The c7MonDecode decodes SCCP/TCAP and ISUP messages stored in a monitor log file (i.e.
$OMNI_HOME/Logs/Monitor.log.<SHM>.<file_number>)
- It can also be used to convert those files in .pcap format if not available
» dmp
- This is the Shared Memory Dump Analysis Utility
- Shared memory contains a number of measurements and diagnostics that are useful if problems
occur. DMP also dumps/formats these variables.
- See man page for details
» omnimon
- Deprecated dump utility, replaced by the newest dumpce
» dumpce
- dumpce utility collects as much debugging information as possible about the user's system and
Signalware configurations and stores them into a .tar file for easy delivery to technical support
42. 42www.eng.it
Monitoring and Troubleshooting
» Other tools:
» snoop + Wireshark/Tshark
- snoop captures IP packets from the network and displays their contents (or
alternatively stores them into a capture file to be viewed through Wireshark)
snoop –r –d <netIf> sctp
snoop –r –d <netIf> -o <outfile.cap> sctp
- Example:
$ snoop –r –d e1000g3 sctp
10.130.34.19 -> 10.130.48.188 SCTP D=2905 S=2905 Heartbeat ACK
10.130.34.19 -> 10.130.48.188 SCTP D=2905 S=2905 SACK tsn 6072e817 win
16384 gaps/dups 0/0
10.130.34.19 -> 10.130.48.188 SCTP D=2905 S=2905 Heartbeat
» swmml commands
- Basically all the DISPLAY-* commands are useful for debugging purpose
43. 43www.eng.it
Monitoring and Troubleshooting
» An example:
» In order to see and log all messages exchanged between M3UA, SCCP and TCAP
stream routers, issue the following commands:
monitor –ln N100 –m 15 N100.L3RT
monitor –ln N100 –m 15 N100.SCCP
monitor –ln N100 –m 15 N100.TCAP
» The information will be logged into the /users/omni/Logs/Monitor.log.100.xxxxx
- To check what happens in real time, use:
tail –f /users/omni/Logs/Monitor.log.100.xxxxx
- To decode the contents of the file, use:
c7MonDecode Monitor.log.100.xxxxx
» Do not forget to disable logging when not needed:
monitor –ln N100 –m 0 N100.L3RT
monitor –ln N100 –m 0 N100.SCCP
monitor –ln N100 –m 0 N100.TCAP
44. 44www.eng.it
Events, Statistics and Measurements
» Signalware detects and reports events related to communications and
platform management.
» Event reporting is done by the log process
» Event are stored into daily log files and are also printed to stdout through the pevt tool
(see man page for details)
» An event reported on the system has the following components:
- Event code:
6 digits, the left 3 digits identify the event type, the right 3 digits the event identification
(see the manual page of omni_events for a detailed description of available events)
- Date and time
- System name
Name of the CE that reported the event
- Severity
One of the following: CRITICAL, ILLOGICAL, ERROR, INFO
(The severity can be changed through MML command CHANGE-SEVERITY)
- Logical name of the process reporting the event
- Text
Textual description of the events. They can be changed through the NLS feature
(man Event_Customization for details).
45. 45www.eng.it
Events, Statistics and Measurements
» Some examples:
…
003010 27-Oct-2008 09:04:49 t1fmp01.LOG Info Log Files Closed/Shut Down
…
…
018280 07-Oct-2008 13:22:12 t1fmp01.L3RT Error MSU discarded due to NI error
…
…
069893 12-Oct-2008 11:57:23 t1fmp01.M3UA Error SCTP <association> net status
change state NOT ACTIVE ipaddr <ipaddr>
…
Event code (type
+ identification)
Date and
time System name
Logical process
name
Severity
Textual Description
46. 46www.eng.it
Events, Statistics and Measurements
» The Node Measurements (MEAS) process collects and records measurement
data produced by the protocol components of the logical node:
» Collected measurements are stored into the following file:
$OMNI_HOME/<CE_name>/dffile/Meas.<node>.<SHM>.<mmdd>
(e.g. /users/omni/t1fmp01/dffile/Meas.N100.100.1105)
» A single file is opened at 00:00 and closed at 24:00 and contains both 5-minutes and 30-
minutes measurements for the whole day (for a single logical node).
» Measurement data are supplied by MTP, M3UA, SCCP, TCAP and MAP.
» MTP/M3UA measurements are available at 5- and 30-minute intervals, while others are
available only at 30-minute intervals.
» Measurement data can also be retrieved through the MML command RETRIEVE-MEAS
» Measurement files are kept for a given period, and are automatically purged after
expiration.
» The purge period is the same applied to logs and events and is configurable through
SCHEDULE-PURGE MML command.
47. 47www.eng.it
Events, Statistics and Measurements
» SCTP measurements are handled differently:
» In the Solaris 10 implementation, SCTP is embedded in the Operating System
- All SCTP stuff, including statistics, is directly handled by the OS itself
- The RETRIEVE-MEAS MML command does not provide SCTP statistics
- Similarly, the following MML commands:
DISPLAY-SCTP-VARIABLES
DISPLAY-SCTP-STATISTICS
DISPLAY-SCTP-ASSOCIATIONS
are not available on Solaris 10 platform.
- SCTP statistics are available through the sctp_stat shell command
48. 48www.eng.it
Events, Statistics and Measurements
» Ulticom events, measurements and alarms are integrated into the np
application:
» This allows a unique access point to statistics and alarms for the whole NP application
» Ulticom measurements are collected by the NP application and stored together with all
other statistics into the following directory:
$SDP_HOME/stat/
» Alarm management is carried out by SSA application and stored into log files located into:
$PLAT_HOME/log
» The alarms raised are forwarded to the Vodafone Network Management System using the
SNMP agent available in the SSA application.
49. 49www.eng.it
Signalware Backup
» The backup of the logical node’s configuration is performed by Signalware
automatically at periodically:
» The backup period is set by default to 5 days:
- Backup occurs at midnight
- Backup period can be changed through the MML command SCHEDULE-BKUP
(between 1 and 365 days)
- Configuration changes between two backups are stored in a recent change files
» The configuration is saved in the form of MML commands into the following files:
- archive.xxx is a single configuration file that is saved for the logical node as a whole
- db.xxx files are used for automatic recovery of the configuration when the logical
node is restarted. There is a separate db.xxx file for each protocol manager
- rc.xxx are the recent change files, which store changes applied to the configuration
since the last backup
- Files are located both in $OMNI_HOME/conf and into the DF directory
$OMNI_HOME/<CE_name>/dffile
- Old backups are purged after the purge period defined by SCHEDULE-PURGE
50. 50www.eng.it
Signalware Backup
» Backup can be forced at any moment by the BACKUP-NODE MML command,
which:
» stores an updated configuration both into archive.xxx and into db.xxx files
» deletes all recent change files
» A given configuration stored in an archive file can be restored by the
RESTORE-NODE MML command:
» The archive file overwrites the recent change files and db.xxx files are deleted, so that
configuration is reloaded as soon as Signalware is restarted
» The command is not allowed if the recent change files are not empty
52. 52www.eng.it
References
» IETF Web Site:
http://www.ietf.org
and specifically IETF SIGTRAN Working Group:
http://www.ietf.org/html.charters/sigtran-charter.html
» Some interesting tutorials from Internet Engineering Consortium (IEC):
http://www.iec.org/online/tutorials/sctp/index.asp
http://www.iec.org/online/tutorials/ss7_over/index.asp
» Ulticom Web site:
http://www.ulticom.com/
55. 55www.eng.it
Why TCP is not suited for signalling transport?
» TCP provides both reliable data transfer and strictly ordered delivery of data:
» Some applications need reliable transfer without sequence maintenance, while
others would be satisfied with partial ordering of the data.
» In both of these cases, the head-of-line blocking offered by TCP causes
unnecessary delay.
» TCP is stream-oriented, not message-oriented. Applications must:
» add their own record marking to delineate their messages, and
» make explicit use of the PSH facility to ensure that a complete message is
transferred in a reasonable time
» Poor robustness against failures:
» No support of multi-homed hosts
» Vulnerability against denial-of-service attacks:
» e.g. SYN attacks
Back
56. 56www.eng.it
Association Startup and Takedown
» An association may be inititiated either
locally by the SCTP user or by the
remote peer
» 4-way handshake with Cookie
mechanism
» Protection against SYN attacks
» Graceful (SHUTDOWN) and ungraceful
(ABORT) association termination
Back
INIT
INIT_ACK (cookie)
COOKIE (cookie)
COOKIE_ACK
SCTP (ASP) SCTP (SGP)
57. 57www.eng.it
Sequenced delivery within streams
» SCTP STREAM = Sequence of user
messages that are to be delivered in
order to the upper-layer protocol
» The SCTP user specifies the number of
streams to be supported at association
startup time. This number is negotiated
with the remote end.
» No Head-Of_Line blocking → while one
stream may be blocked waiting for the
next in-sequence user message, delivery
from other streams may proceed.
» SCTP provides also a mechanism for
unordered delivery.
Back
DATA (Stream ID=1, SSN=2, TSN=12)
SACK (CTSN=13)
SCTP SCTP
DATA (Stream ID=2, SSN=345, TSN=13)
DATA (Stream ID=1, SSN=3, TSN=14)
DATA (Stream ID=2, SSN=346, TSN=15)
DATA (Stream ID=2, SSN=347, TSN=16)
SACK (CTSN=16, GAP blocks=1,
GAP block start=15, GAP block end = 15)
58. 58www.eng.it
Acknowledgement and Congestion Avoidance
» SCTP assigns a Transmission Sequence Number (TSN) to each user data
fragment or unfragmented message.
» The TSN is independent of any Stream Sequence Number (SSN) assigned at
the stream level.
» The receiving end acknowledges all TSNs received, even if there are gaps in
the sequence.
» In this way, reliable delivery is kept functionally separate from sequenced
stream delivery.
Back
59. 59www.eng.it
User Data Fragmentation
» For transmission efficiency, SCTP defines mechanisms for bundling of small
user messages and fragmentation of large user messages:
» When converting user messages into DATA chunks, an endpoint will fragment
user messages larger than the current association Path MTU into multiple
DATA chunks. The data receiver will normally reassemble the fragmented
message from DATA chunks before delivery to the user.
» Fragmentation is performed end-to-end (IPv4 supports only hop-by-hop
fragmentation, which is less efficient)
» Path MTU can be statically provisioned
- Some implementations supports also a mechanism for Path MTU discovery.
Back
60. 60www.eng.it
Chunk Bundling
» For transmission efficiency, SCTP defines mechanisms for bundling of small
user messages and fragmentation of large user messages:
» Multiple DATA and control chunks may be bundled by the sender into a single
SCTP packet for transmission, as long as the final size of the packet does not
exceed the current path MTU.
» On the receiving side, the receiver will unbundle the packet back into the
original chunks.
Back
61. 61www.eng.it
Packet Validation
» Each SCTP Packet contains a mandatory Verification Tag field and a 32-bit
checksum field:
» The Verification Tag value is chosen by each end of the association during
association startup.
- Packets received without the expected Verification Tag value are discarded,
as a protection against blind masquerade attacks and against stale SCTP
packets from a previous association.
» The checksum is set by the sender of each SCTP packet to provide additional
protection against data corruption in the network.
- The receiver of an SCTP packet with an invalid checksum silently discards
the packet.
- RFC 2960 chose the Adler32 algorithm for packet validation, but this
algorithm suffers of poor performance with small packets
- RFC 3309 introduced the new CRC32 algorithm
Back
62. 62www.eng.it
Path Management
» The SCTP path management function is
responsible for choosing the destination
transport address for outgoing SCTP
packets, based on:
» the current reachability status of the
remote IP addresses
» the SCTP user's instructions
» Remote IP addresses’ reachability is
continuosuly monitored through SCTP
heartbeats when other packet traffic is
inadequate to provide this information
» SCTP user is notified upon remote IP
address status change
» At association startup, a primary path is
defined for each SCTP endpoint, and is
used for normal sending of SCTP
packets. Back
Multi-Homed
SCTP
End Point
Multi-Homed
SCTP
End Point
Primary Path
Alternate Paths
IP1
IP2
IP3
IP4
63. 63www.eng.it
Main Signalware Processes
» Parent Operation Process (POP)
» First process started when Signalware is launched.
» POP provides local process management functions (within a CE). It is
responsible of:
- Creating the process set’s memory mapped file
- Starting other processes
- Detecting processes’ failure or termination
- Offering services such as automatic process recovery in the event of a
failure
- Handling shutdown of local process sets and managing the deallocation of
resources
Back
64. 64www.eng.it
Main Signalware Processes
» Platform Manager (PM)
» The Platform Management (PM) process is responsible for the following:
- Configuring, unconfiguring, starting, and stopping all processes described in the
cestart.<SHM> file (created automatically during SW installation; see example below)
$ DFcat cestart.100
.CE t1fmp01
@N100 C7M3UA
- Startup and shutdown of all logical nodes
- Routing MML commands to logical nodes and other processes
- Maintenance of a per-operator command history file of MML commands
- Direct handling of Platform Management MML commands, such as:
- DISPL-PURGE
- SCH-PURGE
- DISPLAY-CE* commands
- DISPLAY-PLATFORM-STATUS
- Automatic purge of measurement and history files
- Centralizing the results of test conditions sent by various processes, computing the
state of each CE based on these conditions, and logging events.
Back
65. 65www.eng.it
Main Signalware Processes
» Signalware Node Manager (SW_NM)
» The Node Management (SW_NM) process is responsible for:
- startup and shutdown of the Signalware logical node including configuration
recovery
- routing of MML commands to the concerned protocol managers
- maintenance of a Recent Change Log of MML commands
- direct handling of node management MML commands, such as:
- BACKUP-NODE
- DISPLAY-BKUP
- RETRIEVE-MEAS
- SCHEDULE-BKUP
- STOP-NODE
- automatic backup of configurations.
Back
66. 66www.eng.it
Main Signalware Processes
» Watch Process (WATCH)
» Signalware automatically starts the watch process to monitor specified Signalware
processes.
- Watch performs two distinct types of process monitoring:
- Critical Process Watching
- List of processes to be monitored contained in a file specified with
–f option
- When the death of any critical-watch process is detected, watch will allow the process
30 seconds to restart. If the critical process does not restart within this time, watch will
shut down Signalware.
- Process Health Check Monitoring
- List of processes to be monitored contained in a file specified with
–h option
- Each process in the list will be sent a priority message (API_WATCH_CHECK) to which it
must respond (API_WATCH_RESPONSE)
- If a process fails to respond to a health check, watch will allow the process up to three such
health-cycle failures. If the process does not respond before the allowance expires, watch
will kill the process. If this fails, watch will shut down Signalware.
Back
67. 67www.eng.it
Main Signalware Processes
» USAM Daemon (USAM_DAEMON)
» The usam_daemon process runs on top of the SCTP protocol state machine
implemented in the Solaris 10 Operating System.
» It is started together with POP when Signalware is launched
» It performs the following tasks:
- waits for incoming commands from M3UA or SUA
- manages SCTP associations (i.e. opens and closes SCTP associations)
- waits for incoming calls on an opened SCTP listening port
» The TRACE_FILE argument sets the tracing filename.
- The usam_daemon does not use the same trace file as other Signalware
Logical Node processes
- Trace level can be modified by means of the samdiag utility
Back
68. 68www.eng.it
Main Signalware Processes
» C7 Protocol Managers (C7xxx processes)
» This is a set of processes that handles specific protocols of the C7 protocol stack
- The c7m3ua process manages the M3UA protocol activities for a logical node
instance that is running in the Signalware environment.
- For each logical node, a copy of the c7m3ua process is executed in each CE
- c7m3ua controls both network management requests and M3UA related MML
operator commands.
- The c7scmg process manages the SCCP protocol activities for a logical node
instance that is running in the Signalware environment.
- For each logical node, a copy of the c7scmg process is executed in each CE
- c7scmg controls both network management requests and SCCP related MML
operator commands.
- The c7tcmg process manages TCAP application-related capabilities for an instance
of a logical node running in the Signalware environment.
- For each logical node, a copy of the c7tcmg process is executed in each CE
- c7tcmg controls both network management requests and TCAP related MML
operator commands.
Back
69. 69www.eng.it
Main Signalware Processes
» Message Monitor (MON)
» MON collects information generated by other Signalware processes and stores them in
log files within the $OMNI_HOME/Logs directory (i.e. /users/omni/Logs)
» Two separate log files are created by default:
- one in displayable HEX/ASCII
- Log file name is Monitor.log.<SHM>.<file_number> where <SHM> is the shared
memory value and <file_number> is a five digit progressive number (numbering
begins from a file number, when the file reaches 2 Mbytes it is closed, the file
number is increased and a new log file is created)
- Files are purged after a given time period (see DISPLAY-PURGE and
SCHEDULE-PURGE commands)
- Those files logs messages exchanged between Signalware processes
- the other is in binary form (PCAP)
- It contains only monitored SS7 Message Signaling Units (MSUs) and can be read
through Ethereal/Wireshark
- Log file name is Monitor.pcap.<SHM>.<file_number>
- The .pcap file is closed along with the corresponding .log (files are always
synchronized)
» Use MonControl utility to select specific file-type logging and monitor to define
monitoring points
Back
70. 70www.eng.it
Main Signalware Processes
» Event Logging and Distribution Process (LOG)
» The log process records and distributes process-set-generated Events, Error
Reports, and Information Notices.
» The log process writes data to the following files:
- $OMNI_HOME/<ce name>/tmp/Event.<SHM>.txt.<mmdd>.<ext>
(e.g. /users/omni/t1fmp01/tmp/Event.100.txt.1105.1
this file contains an ASCII string of each logged event)
- $OMNI_HOME/<ce name>/tmp/Event.<SHM>.nls.<mmdd>.<ext>
(same as above, but events are logged according to the Native Language Support
feature)
- $OMNI_HOME/Logs/Event.<SHM>.diag.<mmdd>.<ext>
(e.g. /users/omni/Logs/Event.100.diag.1105.1
This file contains diagnostic information provided by the local log process to log
critical errors)
» Files are opened at 00:00 and closed at 24:00 (if size increases over a given limit, by
default set to 10 Mbytes, the files are rotated)
» The trace level can be set through MML commands (SET-M3UA-TRACE, SET-SUA-
TRACE)
Back
71. 71www.eng.it
Other Signalware Processes
» Parent Update Process (PUP)
» Used by POP in a multiple-CE environment to distribute information on table and state change to
other CEs.
» Signalware Resource Usage Monitor (RESMON)
» RESMON samples the system resources of the local CE at regular intervals to determine whether
CPU, memory, and disk utilization do not exceed given thresholds.
» If such threshold are exceeded, RESMON notifies PM, which marks the corresponding CE as not fully
operational
» Thresholds and sampling rate are configurable in omni_conf_info file
» The operator can monitor platform status through the DISPLAY-PLATFORM-STATUS MML
command
» Node Measurements Process (MEAS)
» The Node Measurements (MEAS) process collects and records measurement data produced by the
protocol components of the logical node.
» Delayed Message Delivery Process (DLY)
» The Delayed Message Delivery (dly) process collects and forwards IPC messages when a specified
day/time is reached.
» Signal Distribution Process (SIG)
» The Signal Distribution process (sig) allows a process to send a signal to another process set
member that does not operate using the same user ID or group ID. Back
72. 72www.eng.it
An example of node provisioning (MIFMP01)
» Node MIFMP01 [PC=H’0918] has 4 Ethernet
Interfaces on 2 distinct Control LANs:
» 2 ASPs, each with 2 IP addresses laying
on different Control LANs and hosted on
different PCI cards for maximum
redundancy
» ASPs load share traffic
» MIFMP01 is connected to 4 Signalling
Gateways (Eagle 5):
» BOSGW03 [PC=H’100C]
» MISGW03 [PC=H’080F]
» NASGW03 [PC=H’200C]
» RMSGW03 [PC=H’180C]
» An M3UA Link Set is defined towards each
SGW:
» Link set name is MI01YYSGW03, where
YY may be MI, NA, RM, BO
» Each Link Set contains 8 SCTP
associations, 4 from ASP 1 (A) and 4 from
ASP 2 (B) Back
BOSGW03 MISGW03
NASGW03 RMSGW03
MIFMP01
MI01BOSGW03 MI01MISGW03
MI01NASGW03 MI01RMSGW03
Primary Route
Alternate Route
73. 73www.eng.it
An example of node provisioning (MIFMP01)
» As an example, here are the 8 SCTP associations (M3UA SLK according to
Signalware terminology) included in M3UA LSET MI01BOSGW03 (to BOSGW03
SGW)
» 4 Associations from ASP A (local IP addresses 10.189.27.2 & 10.189.155.6)
» 4 Associations from ASP B (local IP addresses 10.189.155.2 & 10.189.27.6)
» Association name is MI01xxxxzw, where xxxx is the SGW, z is A or B according to local
ASP, w is an incremental number
Back
Local ASP Remote ASP M3UA SLK Primary LOCAL
IP
Secondary
LOCAL IP
Primary
REMOTE IP
Secondary
REMOTE IP
MIFMP01_A BOSGW03_W MI01BO03A1 10.189.27.2 10.189.155.6 10.189.47.232 10.189.175.232
MIFMP01_A BOSGW03_X MI01BO03A2 10.189.27.2 10.189.155.6 10.189.47.231 10.189.175.231
MIFMP01_A BOSGW03_Y MI01BO03A3 10.189.27.2 10.189.155.6 10.189.47.230 10.189.175.230
MIFMP01_A BOSGW03_Z MI01BO03A4 10.189.27.2 10.189.155.6 10.189.47.229 10.189.175.229
MIFMP01_B BOSGW03_A MI01BO03B1 10.189.155.2 10.189.27.6 10.189.175.25410.189.47.254
MIFMP01_B BOSGW03_AA MI01BO03B2 10.189.155.2 10.189.27.6 10.189.175.22810.189.47.228
MIFMP01_B BOSGW03_AB MI01BO03B3 10.189.155.2 10.189.27.6 10.189.175.22710.189.47.227
MIFMP01_B BOSGW03_V MI01BO03B4 10.189.155.2 10.189.27.6 10.189.175.23310.189.47.233
74. 74www.eng.it
An example of node provisioning (MIFMP01)
» MML commands used for MIFMP01 provisioning:
» Create Own Point Code:
CREATE-OSPC:PC=2328,NI=NAT1; # PC=H’0918=2328
» Create IP Link SETs
CREATE-M3UA-LSET:LSET=MI01BOSGW03,TYPE=ASP-SGP,
RADDR=10.189.47.232&10.189.175.232&10.189.47.231&
10.189.175.231&10.189.47.230&10.189.175.230&10.189.47.229&
10.189.175.229&10.189.175.254&10.189.47.254&10.189.175.228&
10.189.47.228&10.189.175.227&10.189.47.227&10.189.175.233&
10.189.47.233;
CREATE-M3UA-LSET:LSET=...
» Create IP links
CREATE-M3UA-SLK:SLK=MI01BO03A1,LSET=MI01BOSGW03,
LADDR=10.189.27.2&10.189.155.6,
RADDR=10.189.47.232&10.189.175.232,ASPID=1,MODE=CONNECT,
RPORT=2905,LPORT=2905; # This establishes SCTP association
CREATE-M3UA-SLK:SLK=...
» Create Route SETs
CREATE-RSET:RSET=MI01BO, PC=4108, RTES=MI01BOSGW03&MI01MISGW03,
LOADSHR=NO; # LOADSHR=NO implies first LSET is primary,
second one is alternate
CREATE-RSET:RSET=...
Back
75. 75www.eng.it
An example of node provisioning (MIFMP01)
» MML commands used for MIFMP01 provisioning (cont.):
» Create M3UA Routing Key
CREATE-M3UA-RKEY:RKEY=RKMI01,TYPE=STATIC-AS,LSET=MI01BOSGW03&
MI01MISGW03&MI01NASGW03&MI01RMSGW03,DPC=2328,CONTEXT=14,SI=SCCP;
» Create remote SSNs (not needed)
# CREATE-REMSSN not used, to avoid undesired SST messages to SGWs
» Create Concerned Point Codes
# Each SGW is also CPC
# 6 is MAP, 12 INAP, 11 and 146 provisioned for future use (SGW ITZ)
CREATE-CPC:PC=4108&2063&8204&6156,SSN=6;
CREATE-CPC:PC=4108&2063&8204&6156,SSN=11;
CREATE-CPC:PC=4108&2063&8204&6156,SSN=12;
CREATE-CPC:PC=4108&2063&8204&6156,SSN=146;
» Allow Route SETs
ALLOW-RSET:RSET=MI01BO;
ALLOW-RSET:RSET=...
Back
76. 76www.eng.it
An example of node provisioning (MIFMP01)
» MML commands used for MIFMP01 provisioning (cont.):
» Activate Link SETs/IP Links
ACTIVATE-M3UA-SLK:SLK=MI01ABOW;
ACTIVATE-M3UA-SLK:SLK=...
» Activate routing keys
ACTIVATE-M3UA-RKEY:RKEY=RKMI01;
» Define rules for inbound GTT
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="0",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="1",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="2",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="3",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="4",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="5",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="6",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="7",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="8",PC=2328,SSN=0,RI=GT;
CREATE-GT:TT=0,NP=ISDN-TEL,NA=INT,DIG="9",PC=2328,SSN=0,RI=GT;
Back