4. Distributed applications The Pouzin Society
Application protocol
Appl. Appl.
Process Application connection Process
A B
1 8
flow
Local handle to a Local handle to a
particular instance of a
particular instance of a Medium that enables applications to communicate
communication communication
• For A and B to communicate, they need:
– A means to identify each other -> Application process naming
– A medium that provides a communication service ->Flows
– A way to indicate the communication medium that they want resources to
be allocated for a particular communication to take place, with certain
quality requirements ->Communication medium API
– A shared domain of discourse ->Objects
– Optionally verify who are they talking to (authenticate), negotiate what
protocol is going to be used to carry the data they will exchange, and
what concrete encoding is to be used ->Application connection
– A method to carry their discourse (objects)->Application protocol
4
5. Distributed applications in the Internet The Pouzin Society
Many:
App name not HTTP, SMTP, FTP, Telnet, RTP, SNMP, App name not
existing, use IP SSH, XMPP, … existing, use IP
address or Application connection (not a generic address or
Domain name mechanism, partially provided through Domain name
different protocols)
1 8
TCP connection or UDP flow
Well known port (the Well-known port, the
handle is no longer local
handle is no longer Medium that enables applications to communicate
local)
• For A and B to communicate, they need:
– Application process naming: No names for applications, IP addresses and
ports is all what we have (URLs are pathnames to the applications)
– Flows: Only 2 types; TCP (some variants) or UDP, each of them with fixed
characteristics
– Communication medium API:Need to know to what PoA and port an
application is attached to in order to allocate a flow, no means to express
desired properties of the flow
– Objects: Vary depending on the application protocol used
– Application connection: Applications have to know in advance which
application protocol is going to be used; authentication is done through
separate protocols.
– Application protocol: Many protocols and encodings, tailored to different
purposes 5
6. Distributed applications in RINA The Pouzin Society
CDAP
Application Application
Application connection (generic names
names mechanism, different authentication policies)
8 8
Flows can have different QoS characteristics
0
Local handle (portId) Local handle (portId)
Medium that enables applications to communicate
• For A and B to communicate, they need:
– Application process naming: Complete application naming theory, no
communication medium internal addresses are exposed to apps.
– Flows: Flows can have a myriad of characteristics, tailored to different
application requirements.
– Communication medium API:Request allocation of flows to other
applications by name; request desired properties for each flow
– Objects: Each application decides on their contents and encoding
– Application connection: Generic application connection establishment
procedure, where different authentication policies can be plugged in
– Application protocol: A single application protocol that can have multiple
encodings: CDAP
6
7. RINA API The Pouzin Society
• The RINA API needs to be different in many ways from
conventional Internet operations (usually “sockets”)
– Application flows connect named Applications, not addresses/ports
– RINA takes responsibility for locating the destination of a flow
request, whereas current practice is to use macro-for-the-address
mapping (e.g., DNS) and then use the absolute address returned plus a
“known port”
• RINA Applications are “registered” in order to be found by their name
– Applications can be reached at multiple points (AE’s, more later)
– Applications can reject a request for a flow before it is created and
then authenticate the requestor before establishing a connection
– RINA allows a flexible specification of the requested quality/properties of
a flow
– RINA transport occurs in application-defined units (“Service Data Units”
or SDUs) vs. a stream of unstructured bytes that force applications to do
their own delimiting into meaningful units
• Current API’s don’t provide access to the full set of RINA benefits
– Though it is not strictly necessary to have a common cross-system API for
RINA, it would still be a Good Thing to have, as sockets was for IP
7
8. Naming – Points to Remember The Pouzin Society
• The Internet does not name applications
– The Internet doesn’t really name nodes/applications –
everything addressable is accessed using “absolute addresses”
(IP addresses, ports) to reach it
– DNS is a name-to-number mapping, but applications contact
other applications using the absolute addresses returned
• There is no virtual addressing (NAT is arguably a step toward it, but
interacts in complex ways with DNS)
• RINA is different
– Applications are named
• There can be multiple simultaneous executing instances of an
application, so they must be distinguished. The “application
instance” is an integral part of the application name.
• An application may contain multiple “Application Entities” (next
slide)
– Applications do not know “addresses” of each other, only
names
8
9. Application naming
The Pouzin Society
• The Application Process Model
– Application Process Name: the name of the app Application
– Application Process Instance: to differentiate specific
instances of the same app
– Application Entity Name: part of the application
Application
concerned with communication. Associated to a subset Entity
Application
Entity
of all the existing application objects
– Application Entity Instance: refers to a particular
instantiation of an application entity (that is a particular
instance of a communication associated to a concrete Gmail Server
instantiation of an application protocol and a set of application
instance 2
objects)
DRDA AE
Instance 1
Browser Application
Instance 1 Gmail Server
Application TCP connection
DRDA AE
HTTP AE HTTP AE Instance 1 Private Network Instance 2
Instance 9 Instance 1 DRDA AE DB Server
TCP connection DRDA AE
Public Internet Instance 1 TCP connection application
HTTP AE Instance 4 instance 1
TCP connection Instance 2
HTTP AE
Instance 4
Gmail app
Instance 1 9
10. Flows The Pouzin Society
• Instantiation of a communication service between applications
– A flow is locally identified by an app through the use of a port-id
– Flows transport well defined units of application data (SDUs, Service Data
Units)
• A flow has some externally visible properties:
– Bandwidth related
• Average bandwidth
• Average SDU bandwidth
• Peak BW duration
• Peak SDU BW duration
– Undetected Bit Error Rate
– Partial Delivery of SDUs allowed?
– In order delivery of SDUs required?
– Maximum allowable gap between SDUs?
– Maximum delay
– Maximum jitter
10
11. The IPC API The Pouzin Society
• Presents the service provided by a DIF: a communication flow between
applications, with certain quality attributes.
• 6 operations:
• portId_allocateFlow(destAppName, List<QoSParams>)
• void _write(portId, sdu)
• sdu_read(portId)
• void _deallocate(portId)
• void _registerApp(appName, List<difName>)
• void _unregisterApp(appName, List<difName>)
• QoSParams are defined in a technology-agnostic way
• Bandwidth-related, delay, jitter, in-order-delivery, loss rates, …
• Aid to adoption: faux sockets API.
• Presents the sockets API to the applications, but internally maps the calls to the IPC API
• Current applications can be deployed in RINA networks untouched, but won’t enjoy all
RINA features
11
12. IPC API Implementation
i2CAT/TSSG Prototype: General design The Pouzin Society
• Design goal: Portability to multiple Operating Systems (take advantage of
Java)
Java
Application Local TCP IPC Manager
connections
Socket Native IPC
s API RINA Proces
API s1
IPC
RINA App Library Proces
s2
• The RINA Library is part of the application, and provides both a Sockets and
a Native RINA API (can be part of the same library or create two
packages).
• The IPC Manager is the point of entry to the “RINA stack” running on the
computer. It hosts the IPC processes, manages its lifecycle
(creation, deletion) and acts as a broker between the RINA library and the
IPC Processes.
• Local TCP connections are the means of communication between Apps
(running the RINA Library) and the IPC Manager.
• Use of blocking I/O: one thread per each TCP connection 12
13. IPC API Implementation
i2CAT/TSSG Prototype: behavior of a “client” RINA application The Pouzin Society
RINA App IPC Manager
Library
Flow Allocation Open a new socket
Send CDAP M_CREATE Message with an FlowService object
Map App Name to DIF, find IPC
Process that is member of the
Send CDAP M_CREATE_R Message with the FlowService object DIF, invoke allocateFlow
Data transfer
Send delimited SDU (byte[]) to deliver data
Cause the IPC process to transfer
Keep data in buffer Send delimited SDU[] to deliver data the data over the flow
until read by the
app, or notify app
Flow Deallocation
Close socket (on Close socket Cause the IPC Process to
response message unallocate the flow
or timer)
Close socket Received delete flow request
13
14. IPC API Implementation
i2CAT/TSSG Prototype: behavior of a “server” RINA application (I) The Pouzin Society
RINA App IPC Manager
Library
App registration
Start a new Server
Socket at port X.
Listen for incoming Open a new socket
requests Update IDD table and related flow
allocator directorie(s). RegisterApp
Send CDAP M_START Message with a AppRegistration object contains Source App naming
info, optional list of DIF names and
socket number
Send CDAP M_START_R Message (success or not, reason)
Flow allocation Incoming Flow allocation
(for each new incoming
flow request) Open a new socket to port X request, if destination app is
registered, open a new socket
Start a new thread for Send CDAP M_CREATE Message with a FlowService object
the flow. Decide if
accept connection
Send CDAP M_CREATE_R Message with the FlowService object
Cause the IPC process to send the
allocate response back
14
15. IPC API Implementation
i2CAT/TSSG Prototype: behavior of a “server” RINA application (II) The Pouzin Society
RINA App IPC Manager
Library
Data transfer
Delimited SDU to write data (byte[])
Keep data in buffer
Cause the IPC process to transfer
until read by the Delimited SDU to read data (byte[]) the data over the flow
app, or notify app
Flow Deallocation Incoming Flow deallocation
socket closed request,
Socket close
Cause the IPC process to send the
Close socket, stop deallocate response back
thread
App unregistration
Close socket
Update IDD table and related IPC
On timer or directly: Process directorie(s)
close socket, close Close socket
serversocket On timer, if not already closed, close
socket
15
17. CACEP
Common Application Connection Establishment Phase The Pouzin Society
Once application processes have a communication flow between them, they have to
set up an application connectionbefore being able to exchange any further
information.
The application connection allows the two communicating apps to:
Exchange naming information with its apposite, optionally authenticating it
Agree on an application protocol and/or syntax version for the application data exchange
phase
1) 2)
M_CONNECT 2
Appl. (srcName, destName, credentials, proto, syntax Appl. Appl. Optional messages exchanging authentication Appl.
Process version) Process Process information Process
A 1 B A B
N
2 2 2 2
flow flow
DIF DIF
3) 4)
M_CONNECT_R (result, reason, options)
Appl. Appl. Appl. Application data transfer phase, processes Appl.
N+1
Process Process Process exchange data using an application protocol Process
A B A B
2 2 2 2
flow flow
DIF DIF
18. CDAP The Pouzin Society
• The Common Distributed Application Protocol (CDAP) is the
application protocol used by IPC Processes to exchange shared
state (IPC Processes are Application Processes)
• It is also recommended for all RINA applications to use for
exchanging shared state (when anything but an amorphous
flow of bytes is needed), legacy aps can use whatever they
want to use
• The CDAP Specification defines the complete set of operations
and messages, as well as their fields
– Connection establishment (Connect, Disconnect, authentication)
– Object operations: create, delete, read, write, start, stop
• The set of objects and meaning of operations is not dictated by
CDAP proper – that is an application concern
– IPC Processes are applications, and manipulate a set of objects, but
none of them are dictated by CDAP
• Messages can be encoded in any agreed-upon way
– As long as the applications agree, e.g., via CACEP exchange
– We currently use GPB, have experimented with JSON
18
19. CDAP operates on objects The Pouzin Society
• All objects CDAP operates on have the following attributes:
– ObjectClass
• Class (data type and representation) of an object
– ObjectName
• A identifier of an object, unique within the objects of the same class
– ObjectInstance
• An alias of objectClass + objectName, uniquely identifies an object
– ObjectValue
• The actual value of the object, encoded as desired by the application
• All CDAP operations can be applied two modifiers: scope/filter;
which enables CDAP operations to affect multiple objects that
form a hierarchy with a single message:
– Scope: An integer indicating how many levels below the selected object
the operation has to be applied.
– Filter: A predicate function that evaluates if the operation should be
applied to each individual object within the scope.
19
20. CDAP, AEs and the OIB/RIB The Pouzin Society
• All the objects an Application Process knows about are locally
“stored” in the Object Information Base/Resource Information
Base.
– The RIB may be an actual database, or just a logical representation of all the
information known by an application process.
• In RINA there’s only a single application protocol: CDAP. Then
why are there different AEs?
– Each AE is able to operate on a subset of the RIB
Application Process 2 Application Process 3
Application Process 1
AE type 1
CDAP AE type 1 AE type 2 CDAP AE type 2
Instance 1 Instance 1 Instance 1 Instance 1
OIB/RIB OIB/RIB
OIB/RIB
20
21. CDAP Implementation The Pouzin Society
• CDAP messages comprise a sequence of 1 or more fields
– The one always-present field is the message type
• Each field has an identifying name or numeric tag
(depending on encoding), and a value
• The field names or tag values (for GPB encoding), value
types, and presence/absence of particular fields in CDAP
messages of each type is defined in the CDAP
Specification
• One supported type for an object value is an embedded
message, not understood by CDAP itself, but transported
unchanged to the apposite
– The message declarations for IPC Process object values are not
part of CDAP, but part of the IPC Process Object Dictionary
definition. Other applications define their own object types
21
22. CDAP Implementation (cont.) The Pouzin Society
• For Google Protocol Buffers (GPB) syntax, a freely-available
compiler can produce code in several languages to
construct a valid CDAP message and to access the fields
of one: https://developers.google.com/protocol-buffers/
– .proto files describe the field values and types
– GPB is being used because of its simplicity, compact
representation, support (Google uses it heavily), a freely-
available high-quality tool, a simple definition language, and
general acceptance by the developer community
– XML, ASN.1, JSON, or other representations would also work as a
concrete syntax to encode CDAP messages
• i2CAT’s implementation uses Java code produced by the
Google protoc GPB compiler
• TRIA’s implementation uses a table-driven
parser/generator that also accepts/generates JSON
22
23. Example Message Definition The Pouzin Society
message qosCube_t{//a QoS cube specification
required uint32 qosId= 1;//Identifies the QoS cube
optional string name = 2;// A human-readable name for the QoS cube
optional uint64 averageBandwidth= 3;//in bytes/s, a value of 0 indicates 'don't care'
optional uint64 averageSDUBandwidth= 4;//in bytes/s, a value of 0 indicates 'don't care'
optional uint32 peakBandwidthDuration= 5;//in ms, a value of 0 indicates 'don't care'
optional uint32 peakSDUBandwidthDuration= 6;//in ms, a value of 0 indicates 'don't care'
optional doubleundetectedBitErrorRate= 7;//a value of 0 indicates 'don`t care'
optional boolpartialDelivery= 8;//indicates if partial delivery of SDUs is allowed or not
optional bool order = 9;//indicates if SDUs have to be delivered in order
optional int32 maxAllowableGapSdu= 10;//indicates the maximum gap allowed in SDUs, a gap of
N SDUs is considered the same as all SDUs delivered. A value of -1 indicates 'Any'
optional uint32 delay = 11;//in milliseconds, indicates the maximum delay allowed in this flow. A
value of 0 indicates don't care
optional uint32 jitter = 12;//in milliseconds, indicates indicates the maximum jitter allowed in this
flow. A value of 0 indicates don't care
}
23
25. SDU Protection The Pouzin Society
• Applications may have different levels of trust in the
communication mediums they use
– Need for a way to protect the SDUs they send through the flows
App A App B
SDUs
1 2
flow
Medium that enables applications to communicate
• SDU Protection module protects outgoing SDUs
App A
and unprotects incoming SDUs
Unprotected • Can perform the following functions
SDUs
(configurable through policies)
• Encryption (Integrity and confidentiality)
outbound SDUs
inbound SDUs
SDU Protection
module • Compression
Protected SDUs
• Error detection (CRCs, FECs)
1
flow • Time To Live
25
26. The Pouzin Society
IPC PROCESS
Block diagram, architecture reference vs. implementation
26
27. Levels of Abstraction
(Abstraction is Invariance)
The Pouzin Society
Reference Model
Invariant
Service
Definitions
Decreasing Levels of
Protocols
Abstraction
Procedure
Variable
s
Policies
Implementation
28. IPC Process The Pouzin Society
• The IPC Process is an entity that provides IPC services for
applications running on the same system
– It is an application, and uses RINA application operations to do
everything it does
– It may or may not be an “OS Process”
– There is no set model for how to implement it, and there can be
very different implementations – based on OS, scale, and many
other concerns
– In some implementations, it will become part of the OS, just as IP
networking is now
– In some implementations, it will operate as “middleware”, atop
the OS and its normal networking layer
• All IPC Processes do similar things – WHAT they do is
described in the Reference Architecture, but there are
many feasible Implementation Architectures for HOW
those functions get done. We’ll examine a few today
28
29. Block Diagram (Reference Arch.) The Pouzin Society
• You’ve seen the RINA Reference Architecture (RA)
partitioning of the IPC Process
– This describes the basic mechanisms and what they
communicate among themselves to perform the total
functionality ascribed to an IPC Process
• Implementers use the RA as a guide to create an
Implementation Architecture
– Driven by their particular requirements, implementation
target, and end-use
• Language, use of OS features, flow of control approach, etc., can
all be different – but they all need to implement the RA
– Modules may be different, but ALL RA functions will be present in
a complete implementation, and will communicate with the
same functions as they do in the RA
– There can also be multiple different implementations of the
same Implementation Architecture (e.g., ports to different OS’s)
29
30. Block Diagram (Reference Arch.) The Pouzin Society
Application Specific Tasks System (Host) System
System Appl. (Host)
Mgmt
Other Mgt. Tasks Appl. (Router) Process
Agemt
Process
IPC Mgt. Tasks
DIF
IPC IPC IPC IPC
Multipl
Resource Process Process Process
exing Mgmt
Mgt.
Agemt Mgmt
SDU Agemt
Inter DIF DIF DIF
Protecti IPC IPC
Directory IPC IPC
on Process Process
Process Process
IPC API
Data Transfer Data Transfer Control Layer Management
SDU Delimiting Transmission
Transmission
Transmission CACEP Enrollment
Control
State Vector
Control
State Vector
Control RIB
State Vector
Data Transfer
Data Transfer Daemon Authentication Flow Allocation
Data Transfer Retransmission
Retransmission
Retransmission
Control
Control
Control
Relaying and CDAP Resource Allocation
Multiplexing Flow Control RIB Parser/Generator
Flow Control
Flow Control Forwarding Table
SDU Protection Generator (Routing)
Increasing timescale (functions performed less often) and complexity
30
32. Overall Goals and Approach The Pouzin Society
• Provide a framework to test and debug the new protocols
– Use a single-threaded state machine model to simplify locking
and increase repeatability
– Operate entirely at user (application) level for easier debugging
• Anticipate the desire to move some portions (which ones
were as yet unknown) into the OS kernel eventually
– Coded in C
– Memory/buffering/time-management operations similar to those
available inside the UNIX/Linux OS
• Anticipate future porting to multiple targets
– Use standard POSIX/UNIX capabilities common on all or most
platforms, avoid extensions that impair portability
– Test on MacOS (Mach-based UNIX) and Linux
– Test on large and small systems (Intel and ARM-based)
32
33. Major Parts of the Implementation The Pouzin Society
• Infrastructure
– Main program, select (event) loop, state machine framework, file
management, non-blocking I/O, delimiting, pseudo-files (internal
IPC, Shim DIF), memory and message
pools, timers, startup/shutdown, configuration parsing, logging and
debug utilities, GPB and JSON utilities
• CDAP
– Table-driven CDAP msg. parse/build, connection state machine
• RIB
– Node allocation, lookup, RIB Daemon operations on nodes
– Object Manager mechanism for operations on objects
• IPC Process
– Per-DIF management (RIB
Daemon, enrollment, startup), FA, FAI, DTP/DTCP, Network Management
client interface, Shim DIF, routing, IPC Process-specific Object Managers
• RINA native API Library
• Tests, including RINABAND
33
34. High-Level Block Diagram The Pouzin Society
UNIX/Linux IDD Application NetMgr App. User Application
Processes RINA API RINA API RINA API
Authentication
Database
UNIX/Linux NetMgr Agent/ Per-DIF Manager
Process Directory Server
Flow Allocator
Flow Al. Instances
RIB
Data
Base EFCP Instances
Routing RMT
Computation
(N-1)DIF
Flows SHIM DIF RINA API
Logger
Device Driver file (N-1) FAI socket
IPCMGR Process
I/O Device RINA DIF 34
36. Overall Goals and Approach The Pouzin Society
• Provide an open source initial RINA implementation that can be
used for education and quick prototyping – as well as to
exercise and improve the RINA specs.
– Easy to develop, OS-independent language: Java
– Code structured to be modular and extendable: Use OSGi as a
component framework (Eclipse Virgo Implementation)
– Portable to different operating systems: only use Java OS-dependent
features available in most OSs (sockets)
• Enable to setup relatively complex scenarios with few hardware
resources
– Use the TINOS protocol experimentation framework –developed by
TSSG- in order to be able to emulate multiple “hardware” within the
same Java process.
• i2CAT/TSSG’s RINA implementation is part of the TINOS project, as
one of the “protocol stacks” available.
– Reuse of TINOS compile/build infrastructure
– Maximize synergies between both projects: single development
community (hosted at github)
– Integration with TINOS will be easier (not done yet)
36
37. Major Parts of the Implementation The Pouzin Society
• Infrastructure
– VIRGO OSGi core (handles the lifecycle of the different components –
bundles in OSGi parlance), single thread pool, blocking
I/O, configuration parsing (JSON library), sockets, Google Guava library
(Java has no unsigned types, thanks!), Google Java GPB
implementation, Java timers, delimiting, object encoding/decoding
• IPC Manager
– RINA-side of the IPC API, IPC Process Lifecycle Management, will host
management agent and IDD (not implemented yet), console service
(local administration)
• “Normal IPC Process”
– RIB, RIB Daemon, CDAP Parser/generator, Enrollment task, Flow
Allocator, Resource Allocator, EFCP, RMT, SDU Protection
• Shim IPC Process for IP Layers
– Setup and management of TCP and UDP flows as per the shim DIF spec
• RINA Application Library
– Native RINA API and faux sockets API
• Test applications
– RINABand, Echo server & client, simple chat application
37
38. General design (I) The Pouzin Society
Server Application 1
OS Process
Client Application 1
(Java VM
RINA Lib RINA Lib instantiation)
For the registration, Local
local TCP connection administration
For each flow, local TCP to port 32771 Listen for local TCP connections at
connection to port 32771 port X (dynamically assigned)
Listen for local TCP
For each flow to service application 1, local connections at port
Listen for local TCP TCP connection to port X 32766
connections at port 32771
IPC Manager
Application Console
Service IPC Process Lifecycle
Service
Management (“IRM”)
IDD
OS Process
(Java VM IPC Service
IPC
instantiation) Normal IPC Process Service
Components
NOTE: Could be multiple
“systems” within the same Shim IPC Process
Delimiter Encoder CDAP Session Manager for IP
Java VM once fully
integrated with TINOS GPB parser Resource
Enrollment Task Allocator
Flow
RIB Daemon Allocator
EFCP Flow Allocator
RMT SDU
Virgo OSGi Kernel Protection
Flows to/from other
Listen for TCP shim IPC Processes
connections and UDP
datagrams at IPa:portb 38
40. Why TINOS?
Larger experimentation scenarios with less infrastructure The Pouzin Society
DIF
Shim DIF Shim DIF
IP (Jnode) IP (Jnode)
Data Link Data Link Data Link Data Link Data Link
Java Virtual Machine
Java Virtual Machine • With TINOS multiple nodes can be created within
the same Java JVM, with different network
connectivity with each other and other JVMs
(TINOS uses adapted IP stack from JNode and
XMPP for this)
DIF
Shim DIF Shim DIF
Java Virtual IP (Jnode) IP (Jnode)
Machine
Data Link Data Link Data Link Data Link Data Link
Java Virtual Machine
40
42. RINA in the OS Kernel The Pouzin Society
• Make RINA a “native” networking API
– New/Extended OS system calls provide full RINA capability
– Move (at least) DTP/DTCP into the OS kernel for speed
App IPC
App
Process-
Application Space
OS Kernel New/Extended OS API Calls
DTP/DTCP Flow State
Forwarding
RMT Table
“Network Device”
Might be a Shim DIF
Network Device 1 Network Device 2 or a RINA DIF
42
43. RINA Split Between H/W and S/W The Pouzin Society
• RINA RMT/DTP performed in hardware
– Software still does DTCP and remainder of IPC Process fn’s
– Transiting PDUs need not be processed by software
App IPC
App
Process-
Application Space
OS Kernel New/Extended OS API Calls
DTCP Flow State
DTP
Hardware/Firmware
Forwarding
RMT Table
Network Interface 1 Network Interface 2
43
45. RIB and RIB Operations The Pouzin Society
• The Resource Information Base (RIB) is a virtual object
database
– Each AE projects a view over the underlying objects
– The RIB holds the shared state of the communication
instances between applications
• The IPC Processes communicate by exchanging
operations on RIB objects
– The only operations are:
create, delete, read, write, start, and stop
– These operations cause objects to perform appropriate
actions (defined in an object dictionary)
– There is a particular tree of RIB objects defined for IPC Process
use (any other application can define its own tree)
45
46. A Few Thoughts on the RIB Daemon The Pouzin Society
• A generalization of Event Management and Routing Update
– Elsewhere (circa 1988) I said Event Management is the
hypothalamus of network management and looks like this:
To Other Management Applications
Add/Delete
Subscription
Filter control
Subscription
Service Subscript
Def
File
Logging Rcv
Events
47. A Few Thoughts on the RIB Daemon The Pouzin Society
• Generalizing routing update adds a capability for managing periodic
and/or event driven a data distribution and replication strategy.
To Other Management Applications
Add/Delete
Subscription
Filter control
Subscription
Service Subscript
Def
File
Replication
Optimizer
Logging Rcv
Does this imply Events
an opportunity Write
for a journaling Subscriptions
RIB for some
data?
48. A Few Thoughts on the RIB Daemon The Pouzin Society
• So re-arranging and re-labeling for our current problem.
Add/Delete
To Requesting Tasks Subscriptions
From Tasks
Event
Reads and Writes to an Subscriptions Subscript
Subscription
actual store or to other Def
Service
tasks or task data File
Write
structures, e.g. DT-state
Subscriptions
vector.
Logging CDAP
An opportunity Processing Replication
for a journaling Optimizer
RIB for some
data?
Incoming CDAP PDUSs
49. RIB Implementation The Pouzin Society
• Our protocol exchanges refer to objects by name and/or
object-id (a number)
– We haven’t started using object-id’s yet, but the intent was to
make the protocol exchanges more compact
– We will standardize the object names/id’s that need to be the
same for consistent RINA implementations through PSOC
• The RIB appears as a tree-structured database with
objects at its leaf nodes. Leaves are named with the full
absolute pathname from the root to the leaf.
• We operate on an object by sending the operation and
the operand object’s name/id (and a value, if
appropriate)
– The reference model has a “RIB Daemon” that performs the
operation; in practice, this may be subsumed into other entities
49
50. Naming conventions for IPC Processes
The Pouzin Society
• Application names:
– Can be whatever, probably would be useful to give some
kind of indication of its physical location (to facilitate
management, for no other purposes).
• Application instances:
– Not used in principle, since in normal operation there should
be no need to connect to a concrete instance of an
application process (default to 1).
• Two Application Entities:
– Management AE: Flows established to/from here are used to
establish application connections to neighboring IPC
Processes and exchange layer management information
using CDAP.
– Data Transfer AE: Flows established to/from here are used by
the RMT to transport “data transfer SDUs”. 50
51. Current tree of objects The Pouzin Society
/daf/management/operationalstatus
/daf/management/naming /daf/management/naming/applicationprocessname
/daf/management /daf/management/naming/address
/daf/management/naming/whatevercastnames
/daf
/daf/management/neighbors
/ /dif/ipc/datatransfer /dif/ipc/datatransfer/constants
/dif/ipc
/dif
/dif/management/flowallocator /dif/management/flowallocator/qoscubes
/dif/management
/dif/management/flowallocator/directoryforwardingtableentries
/dif/resourceallocation /dif/resourceallocation/flowallocator /dif/resourceallocation/flowallocator/flow
s
/dif/resourceallocation/nminus1flowmanager /dif/resourceallocation/nminus1flowmanager/nminus1flows
/dif/resourceallocation/pduforwardingtable
51
53. Enrollment The Pouzin Society
• Enrollment is the process by which an IPC Process
communicates with another IPC Process to join a DIF
– And acquires enough information to start operating as a member of
the DIF
– After enrollment, the newly-enrolled IPC Process is able to create
and accept flows between it and other IPC Processes in the DIF
• Enrollment on the Internet
– For TCP/IP mostly inexistent or by ad-hoc/manual means (DHCP
provides a bit of the required functionality)
– In IEEE 802.11 the procedure for joining a network is almost identical
to what RINA predicts. The BSSID is a DIF-name.
– Similarly, there is enrollment in 802.1q (VLANs).
– Done independently, confirmation of the theory.
53
54. Start at the Beginning
Joining a DIF The Pouzin Society
IPC Process A wants to join IPC Process B is a member of a DIF beta
DIF of which B is a member.
DIF Management
DIF Management
Establish conn
authenticate
Initialization information
(N-1)-DIF
• A Wants to join DIF beta of which B is a member. First it needs to establish communication with beta. So A’s
DIF Management task using DIF A’s IPC Manager (not shown) does an allocate(beta, as good QoS as it can
get).The name beta is a whatevercast name for the set containing the addresses of all members of beta that
the rule returns the address of an IPC Process with a common (N-1)-DIF. The whatevernme is resolved by the
(N-1)-DIF.
• The Allocate creates a flow between A and B. They exchange CDAP connect requests, followed by
whatever authentication is required to establish an application connection between A and B. Actually
between A and beta. B is acting as an agent or representative for beta.
• Then A and B exchange initialization information. Primarily B is telling A what its DIF internal name (address) is
and populating A’s RIB with the current information on the DIF. We will come back to this.
54
55. A is now a member of beta
The Pouzin Society
IPC Management
RIB Daemon
IPC Management
RIB Daemon
A B
Application
Connection
(N-1)-DIF
• There is now an application connection between the IPC management components of A
and B.
– All connections between members of a DAF are managed by their IPC management
component.
– Any management component can send on the flows managed by IPC.
– All incoming PDUs are delivered to the RIB Daemon.
– The RIB Daemon is a subscription service, essentially a generalization of both routing
update and event management. When any CDAP PDU arrives, it is logged and
distributed to the tasks that have subscribed to be notified.
– The Flow Allocator subscribes to Create/Delete Flow Req. (The Flow Allocator will
update the RIB after processing the request.)
55
56. Enrollment Exchange The Pouzin Society
• There are several enrollment situations that IPC
Processes encounter when connecting, for example:
– An IPC Process that is not enrolled connects to an IPC Process
that is not enrolled in a DIF – the two form a DIF
– An IPC Process that is not enrolled connects to an IPC Process
that is already enrolled in a DIF – it joins the DIF
– An IPC Process that is enrolled makes a connection to a
neighbor that is enrolled – they now have a new route for
flows
• An IPC Process can be in either role, as initiator or
target
• The information exchanged in some cases can be
reduced to minimize enrollment time
56
57. Enrollment Procedure I The Pouzin Society
• When the New Member receives the M_Connect Response, the New Member
copies Current_Address to Saved_Address, it sends
– M_StartEnrollment(address, Address_expiration_time, other data about New
Member)
• /* The New Member is telling the Existing Member what it knows. Primarily
this is derived from the address (NULL or not), and the expiration life-time of
the address if non-NULL. Since addresses are generally assigned for hours or
minutes, tight time synchronization is not required. (Even for DIFs with fast
turnover, fairly long assignment times are still prudent.)*/
• The Member sends
– M_Start_REnrollment(address (potentially different), Application Process
Name, Current_Address, Address_Expiration).
58. Enrollment Procedure II The Pouzin Society
• Using the information, provided by the New Member, the Existing Member
sends
– M_Create (zero or more) to initialize the Static and Near Static information
required. When finished and the New Member has sent all necessary
– M_Create_Rs
• The Existing Member sends a
– M_Stop Enrollment (Immediate:Boolean)
• The New Member may Read any additional information not provided by the
Existing Member.
– M_Read (zero or more)
– M_Stop_R Enrollment
• If the Immediate Boolean is True, the New Member is free to transition to the
Operational state.
• If the Boolean Immediate is False, then the New Member can not transition to
the Operational state until an M_Start Operation is received.
59. Enrollment Procedure III The Pouzin Society
• The New Member is free to Read any information not provided by the
Existing Member. Once these are completed, the Existing Member sends:
– M_Start Operation
• The New Member sends
– M_Start_R Operation
• Invoke RIB Update of dynamic information which will cause others to send
data to the New Member.
60. Example Message Sequence
Skipping Application connection setup (CACEP) The Pouzin Society
• One IPC process is a member of a DIF, another one is not
Joining Member
IPC Process IPC Process
M_START (Enrollment_Info_object{address=null})
1
The joining IPC Process has no
2 address, not a member of the DIF.
M_START_R (ok, Enrollment_Info_object{address=25})
Assign a valid address and reply
Got a positive response and 3
an address. Wait for STOP Send DIF static info (whatevercast
4
Enrollment response, RIB 4 names, data transfer
Daemon processes the M_CREATE (DIF_info1) constants, qos cubes, supported
M_CREATE messages policy sets) and dynamic info
5
(neighbours, directory forwarding
… table entries) through a series of
M_CREATE (DIF_infoN) M_CREATE messages
5
Check if I got enough data to Once all the information is sent,
start. If more info is required 6 send stop enrollment request
send M_READ requests on (informing the enrollee has to
specific objects (not the M_STOP (Enrollment{allowed_to_start_early=true}) wait for START operation request)
case). I’m Enrolled! and wait for response
Now, if I have a DIF in 7
8 common with one or more of
the neighbors (I’m M_STOP_R (ok)
multihomed) I could enroll
with them as well (next slide) 9
10 Got STOP response. He’s enrolled!
Ignore if started earlier, or M_START (operationalStatus)
12 Send M_START message (no
start now (consider enrolled 11 answer required)
now)
60
61. Example Message Sequence
Skipping Application connection setup (CACEP) The Pouzin Society
• Both IPC Processes are members of the same DIF
Joining Member
IPC Process IPC Process
also a member
M_START (Enrollment_Info_object{address=25})
1
The joining IPC Process has a valid
2 address, he is a member of the
M_START_R (ok, Enrollment_Info_object{address=25})
DIF. Reply
Got a positive response and 3
my address is still valid. Send DIFs dynamic info only
4
Wait for M_STOP 4 (neighbours, directory forwarding
enrollment request, RIB M_CREATE (DIF_info1) table entries) through a series of
Daemon processes the M_CREATE messages
M_CREATE messages 5
…
M_CREATE (DIF_infoN)
5
Once all the information is
6
sent, send stop enrollment
Check if I got enough data to request
M_STOP (Enrollment{allowed_to_start_early=true})
start. If more info is required
send M_READ requests on 7
8 specific objects (not the
case). The member I’ve
talked to is now my M_STOP_R (ok)
neighbor!
9
Ignore if started earlier, or M_START (operationalStatus) 10 Got STOP response. He’s my
12
start now (consider enrolled 11 neighbor! Send start message, no
now) response required
61
63. Flow Allocator The Pouzin Society
Allocate(Dest-Appl-Name, QoS parameters)
Flow
Allocator
Dir
Local
Forwarding
Dir Cache
Table
• When Application Process generates an Allocate request, the
Flow Allocator creates a flow allocator instance to manage each
new flow.
• The Instance is responsible for managing the flow and
deallocating the ports
– DTP/DTCP instances are deleted automatically after 2MPL with no
traffic,
• When it is given an Allocate Request it does the following:
63
64. Details of the Allocation Data Flow: I
The Pouzin Society
Allocate(dest-appl, desired_flow_properties)
Appl-names Next Place
Create Flow(dest-appl, stuff)
FAI
Directory
Subscribe Forwarding
Create/delete Table
Flow objects EFCP
RIB
Daemon
IPC/RMT
• Upon initialization, the FA subscribes to create/delete flow objects.
• The FAI is handed an allocate request. After determining that it is well formed it must find the
destination application.
• It consults the Directory Forwarding Table (dotted arrow). The table maps the dest-appl in the
request to a “Next Place” to look for it (IPC Process @)
• That points to either a nearest neighbor management flow (if it is multihomed there will be more
than one) or a connection allocated that does not go to a nearest neighbor, but uses the data
transfer AE. This connection was created by the management task and is available to all tasks
within the IPC Process.
64
65. Details of the Allocation Data Flow: II
The Pouzin Society
Appl-names Next Place
FA Create Flow(dest-appl, stuff)
RIB Directory
Daemon Forwarding
Table EFCP
Create Flow(dest-appl, stuff)
IPC/RMT
• When a Create Flow Request arrives, the RIB Daemon forwards it to the FAI for
inspection.
• If the FA determines that dest-appl is not here, then it consults the Directory
forwarding table as before to determine where to send it next.
• If dest-appl is here, then . . .
65
66. Details of the Allocation Data Flow: III
The Pouzin Society
Allocate Dest
Allocate
Indicate Appl
Confirm Read/Write
Appl-names Next Place
Create Flow Resp
FAI
RIB Directory
Daemon Forwarding
Table
EFCP
Create Flow Req(dest-appl, stuff)
IPC/RMT
• When the Create Flow Req arrives it is passed to the Flow Allocator.
• The Flow Allocator looks it up in the table and determines that dest-appl is here. It
determines whether or not the requestor has access to dest-appl.If it does,
• then dest-appl is instantiated if necessary, and given an allocate indicate.
• It responds with an allocate confirm, if positive then data transfer can commence.
• In either case or earlier of access was denied, a Create Flow Resp is sent back with the
appropriate response.
66
67. Implementation of the Flow Allocator
Application registration and DirectoryForwardingTable The Pouzin Society
• DirectoryForwardingTable maps ApNames to the @ of IPC processes
where they are currently registered.
– Updated by local application registration events (through IPC API)
– Updated by remote application registration events (through remote CDAP
messages processed by the RIB Daemon)
– Updated by timers (to discard stale entries)
• Distributed database, several strategies for implementation (the
larger the DIF, the more complex it becomes)
– Compromise between load of messages to update the database vs. the
timeliness of the data in each DB
– Fully replicated vs. partially replicated
• Current implementation: simple, only for small DIFs.
– Fully replicated Database (all the IPC Processes know about all the registered
applications in the DIF)
– Each time a local application registers/unregister, the FA sends CDAP M_CREATE
message to all its nearest neighbors
– Each time a new mapping is learned (from a remote update), if the value of that
mapping changed, the FA sends CDAP M_CREATE message to all its nearest
neighbors – except for the one that notified the update -
67
68. Implementation of the Flow Allocator
i2CAT: Management of flows and Interaction with EFCP The Pouzin Society
Appl. Appl.
Process Process
Allocate allocation_requested(srcApName)
Response(result) Allocate Request (destAPName, QoS Params)
1 allocation_response(
9 5 6 result)
Map request into M_CREATE(Flow object)
IPC Process 2 Check access IPC Process
policies, see if is feasible. 3
Search dest app. at the 4 control and
Flow policies to see if Flow
directory. Allocator
Allocator flow is feasible
8
Create DTP/DTCP M_CREATE_R(Flow object) 7 Create
2 Create FAI 2 4 Create FAI
instance DTP/DTCP
instance
FAI DTP/D DTP/D
FAI
TCP DIF TCP
• When the flow has been established, 1
incoming and 1 outgoing queue are
allocated at the layer boundary by the
FAI
• Also, a new EFCP StateVector for the
connection (1 per flow right now) is
instantiated at the DataTransferAE; as well
as 2 queues for queuing PDUs to/from the
RMT
68
70. EFCP: Error and Flow Control Protocol The Pouzin Society
Data
State Vector
Transfe
Data Transfer r
Protocol Control
• Based on delta-t with mechanism and policy separated.
– Naturally cleaves into Data Transfer and Data Transfer Control
• Data Transfer consists of tightly bound mechanisms
– Roughly similar to IP+UDP
• Data Transfer Control, if present, consists of loosely bound mechanisms.
– Flow control and retransmission (ack) control
• One or more instances per flow; policies driven by the QoS
parameters.
– The Flow Allocator translates the QoS parameters into suitable policies.
– In parallel, might be used for things like p2p [sic] do.
– Used serially, avoids the need for a separate security connection as in IPsec.
• Comes in several syntactic flavors based on the length of
(address, connection-endpoint-id and sequence number)
• Addresses: 8, 16, 32, 64, 128, variable.
• CEP-id: 8, 16, 32, 64
• Sequence: 4, 8, 16, 32, 64
70
71. EFCP: separation of port allocation from
synchronization The Pouzin Society
Port-
id
Port Allocation (FA dialogue, IPC
Connection Process
Endpoint management)
Synchronization (EFCP state machines, data transfer)
Connection
• Separatingportallocationfromsynchronization–unlike TCP- has
interestingsecurityimplications– more onthislater.
• Port Allocationstateiscreated/deletedbasedonexplicitrequests
• Local applicationsthroughthe IPC API (allocate/deallocateflows)
• Remote CREATE/DELETE Flowrequestsfromother IPC Processes
• Synchronizationstateisrefreshedevery time a DTP/DTCP
packetissent/received
• If no packetisreceivedafter a certainamountof time stateisdiscarded
71
72. Intro to Delta-T
Timer-based connection management The Pouzin Society
• All connections exist all the time, the protocol just needs to keep
caches of the state for those that have carried traffic recently
– When a PDU is received for a certain connectionId, the state of the
connection is refreshed
– After a certain amount of time with no traffic, the state is discarded
• What amount of time with no traffic is necessary to be able to
safely discard the send/receive state and ensure that:
– No packets from a previous connection are accepted in a new
connection
– The receiving side doesn’t close until it has received all the
retransmissions of the sending site and can unambiguously respond
to them
– A sending side must not close until it has received an Ack for all its
transmitted data or allowed time for an Ack of its final retransmission
to return before reporting a giveup failure.
72
73. Intro to Delta-T (II)
Timer-based connection management The Pouzin Society
• MPL: Max time to traverse a network
Sender Receiver • A: Max time the receiver will wait
before sending an acknowledge
MPL
• R: Max time a sender will keep
A
retransmitting a packet
• deltaT = MPL + A + R
R • Watson showed that send state can
be safely discarded after a period of
3deltaT with no traffic, and receive
state can be discarded after a
period of 2deltaT with no traffic
• No SYNs are FYNs are necessary (compared to TCP) -> simpler, more robust
• Implication of Watson’s results:
• If MPL cannot be bound, then there is no wayto have a reliable data
transport, therefore it cannot be IPC
73
74. Data Transfer Protocol (DTP) The Pouzin Society
• Notice that the flow is a
InboundQ straight shot, very little
Delimit SDU
processing and if there is
anything to do, it is moved to
Sequencing/ Fragment/
the side. The most complex
Strip Concatenate thing DTP does is
Delimiting
reassembly and ordering.
Reassmb/SeqQ Reassembly/ Sequence/
Separation Address
CRC • If there is a DTCP instance
ClsdWinQ for this flow:
– If the flow control window closes,
PDUs are shunted to the flow
controlQ.
RexmsnQ
– If the flow does retransmission, a
DTCP copy of the PDU is put on the
PDUs rexmsnQ.
RMT
CRC
• These PDUs are now
DTCP ’ s responsibility to
send when appropriate.
74
75. Data Transfer PDU Contents The Pouzin Society
• Version: 8 Bit (optionally used, absent in current prototypes)
• Destination-Address: Addr-Length
• Source-Address: Addr-Length
• Flow-id: Struct
– QoS-id: 8 Bit
– Destination-CEP-id: Port-id-Length
– Source-CEP-id: Port-id-length
• PDUType: 8 bits
• Flags: 8 bits
• PDU-Length: LengthLength
• SequenceNumber: SequenceNumberlength
• Sequence User-Data{DelimitedSDU* | SDUFrag}
75
76. DTP PDU Parsing Example (DEMO) The Pouzin Society
intpolicy = dtc.EFCPEncodingPolicyType;
switch ( policy ) {
case PDUVERSION_DEMOPROFILE:
NEXT16(destAddr);
NEXT16(srcAddr);
NEXT16(destCEPID);
NEXT16(srcCEPID);
NEXT8(qosID);
NEXT8(pduType);
NEXT8(flags);
NEXT32(pduSeqNumber);
break;
76
Hinweis der Redaktion
IMPORTANT: RINA DIFs transport any kind of data (independent of the application protocol), here we’re just showing that there’s only a need for a single application protocol, and there are benefits if everyone sticks to just one.
A major point to make in this slide is that using the existing sockets API over RINA is not going to be a satisfactory way to make them “compatible”. Possibly a good time to mention “Microsoft-style compatibility” vs “Apple-style compatibility”.This transitions to the next slide, that’s a similar reminder about naming.****Sockets doesn’t let you at all of the capability of AEs and AEIs or Application Instances. You can’t connect to the same process instance for different reasons with sockets.
This is repeating things they’ve already heard, but in the context of the API and comparing to Internet it becomes more clear what the implications of that are.File names ARE a part of a URL, by convention, but the first part of the URL is a FQDN or IP address, so the first point is still true even if the “application” is an HTML file or CGI script. *****The URL is a *path* to the application, not a name of the application.
CACEP is used to create an application connection for any application protocol. CACEP has the hooks to provide an authenticated connection. For pure RINA DAFs, CACEP is always wrapped around CDAP. However, it might be used with any legacy application protocol, or a legacy application protocol could use RINA directly. These are provided primarily for backward compatibility. We believe the advantages of adopting the DAF approach to distributed computing has sufficient advantages that it will overtime replace the myriad of legacy application protocols. Over time use of CACEP without CDAP will be rare.
Mention it is similar to CMIP. Why CMIP? It’s a management protocol.– Not really, it is a distributed object-oriented intermediate language– That does create/delete, read/write (get/put), action (start/stop)– With OO-support in Scope and Filter:• Scope selects the base object alone, the nth level of the base object,the base object and all its subordinates to and including the nth level,or the base object and all its subordinates.– This can be quite powerful in minimizing traffic.• Filter is an expression of assertions grouped using AND, OR, andNOT to determine equality, ordering, presence, or set comparison.– Extensions to scope and filter should be considered.• CMIP does everything required and nothing more.Note in passing, we use little-endian application protocol encoding byte order because most modern computers do, so it avoids multiple byte-flops at both ends to accommodate a big-endian network byte order convention.NOTE: Should we talk about scope and filter?
Reasons for choosing GPB:Provides efficient encoding (bitsize and time to parse/generate)Being used in production by Google in massive scaled distributed systemsFree, open source tools for developers in many programming languages
Scope/Fileter not implemented yet
Use of this message as the example also makes the QoS cube concept more concrete.
Implementation described with IPC Process, relationship with RMT
Remember, this is the architecture!DAF Support Tasks: The IPC Management (and other management: memory, storage, CPU) tasks are usually implemented as OS functionality.IPC Resource Management: Creation/Deletion of IPC processesMultiplexing (Usually inverse multiplexing, an application flow into multiple DIF flows, for example: 1 for video, 1 for audio, 1 for text, …)SDU Protection (CRCs, encryption, TTL, …)IDD (Inter DIF Directory, find out in what DIF the destination application process is executing)
TRIA current code size: about 40K lines of source files (counting blank lines and comments, so probably not much over ¾ of that is code)
This is from TRIA’s earliest implementation document, updated by adding the Shim DIF box. The “DIF Manager” incorporates the RIB Daemon, Enrollment, and a few other IPC Process functions, as well as overall IPC Process life cycle (startup, shutdown).Bold-face boxes are State Machines. Other boxes are libraries.
i2CAT current code size: about 45K lines of source files (counting blank lines, comments and Javadoc, so probably not much over ¾ of that is code)
These pictures are not intended to be especially complete or accurate, but to show the basic idea.
The “IPC Process-” is an IPC Process that does not need to perform EFCP processing for application flows. It controls the content of the flow table (for allocating/deallocating) and the forwarding table. Only one is shown, but there could be as many as desired, each with an RMT, and possibly sitting above other EFCP/RMT instances in the kernel rather than a raw network interface.In the next slide, we’ll comment on the size of the Forwarding Table since it’s in the OS space here (which is not virtual, and size is a big issue). We can make an even stronger point when it’s in hardware.No Shim DIF is shown, RINA is the native network interface in this picture, as it’s easier to understand this way.
No attempt to be terribly accurate in this picture – just trying to capture the idea. You could replicate the RMT/DTP/Forwarding in the hardware layer for multiple DIFs if desired.This is a good place to point out that if the address space of the version of DTP in use by the hardware is small, the forwarding table is also small. For hardware tables, that’s really important since the memories are often implemented in an expensive technology. Compare that with the present situation with IPv6.The IPC Process- is shown as connecting to DTP – that’s for its management flows.
This may or may not be a good idea, but it’s easy to delete. It’s intended to show how simple DTP PDU parsing is.This shows the DTP form adopted for the demo. Error checking (e.g., short SDU) has been deleted. The length of the PDU is implicitly present, as the PDU is carried in a SDU that is received as a unit with a known length.
The pushback strategy described has implications – e.g., you have to know when the (N-1) can take it. This is ok if you have a queue there (you can see if it has space in it), an an opportunity to run a policy to decide which of the flows with outbound data is the most worthy to take a PDU from when the queue develops empty space.Note that the RMT is the main entity responsible for implementing QoS on outbound PDU’s. The policies it uses to select which flow’s PDU to accept for output is the heart of QoS.
We can allow different protection on different (N-1) flows, but that is mainly a Shim DIF issue.
Note that we are currently using TCP for reliable data flows, but for Data Transfer (as opposed to management traffic) it does potentially generate the need for unlimited buffering (since there’s no way to individually flow control the multiplexed flows). As we get DTCP more fully implemented, this need will go away and we’ll use UDP for everything.