[erlang-questions] : : : sctp_peeloff() support in OTP

[Valentin Micic]

One of the problems that I see with SCTP is that, unlike TCP where you have
no choice but to send the reply over the same stream you've received request
through; SCTP AFAIK does not regulate this -- you could receive a request
over, say, stream 6 and send a reply using stream 54. Whilst this problem
exists irrespective on how one implement stream separation, using separate
file descriptor may look a bit wasteful from local resources point of view.

If it was up to me, I would invest a bit more time in attempt to avoid usage
of multiple file descriptors. Could one have a number of "pseudo-ports"
mapped to the same "real" port/file descriptor?

Best
V. 

-----Original Message-----
From: 'Raimo Niskanen' [mailto:raimo+erlang-questions@REDACTED] 
Sent: 22 April 2009 10:36 AM
To: Valentin Micic; erlang-questions@REDACTED
Subject: Re: : : [erlang-questions] : sctp_peeloff() support in OTP

On Tue, Apr 21, 2009 at 05:58:47PM +0200, Valentin Micic wrote:
> 
> >> implementing it without defeating the purpose behind SCTP?
> 
> > Could you elaborate...?
> 
> Let me try by saying that TCP was not adequate for "multiplexing", a
traffic
> case where a bunch of things on one machine wants to talk to a bunch of
> things on another using the same connection. The SCTP addressed this issue
> on a number of levels, but most importantly -- by eliminating a
head-of-line
> blocking bottleneck (someone mentioned a year or two ago that SCTP would
be
> far better protocol for Erlang distribution).
> 
> The reasoning behind SCTP is to provide for a minimal latency, where
> processing of one stream should not create a processing delay on another.
> Having the same port owner responsible for all the processing is at odds
> with this design goal. In other words, {active, once} would effectively
> reintroduce head-of-line blocking, by serializing all the traffic and
> forcing it through a single process.
> 

Thank you for the explanation. The term "head-of-line blocking"
was new to me.

Unfortunately I just realized that when the driver calls recvmsg()
it can not choose which stream to receive from. So {active,once}
must introduce head-of-line blocking in some form. And introducing
{active,StreamNo,Pid} would have to imply {active,true},
just dispatching the streams.

I think the only thing that solves the head-of-line blocking
problem is branched-off associations (sctp_peeloff()) because
then you get a new filedescriptor to poll() and recvmsg() as
you choose from, in a new port driver instance (Port).
Correct me if I am wrong.

> Not so well thought through approach to active socket processing:
> -----------------------------------------------------------------
> Maybe one may mitigate this by creating a number of Erlang processes as
> there are streams (call it stream processors), and route upstream traffic
to
> a particular stream processor? Then, if there is an "external process"
> requesting {active, [once|true, StramNo, PID]} that relationship may be
> remembered by that particular "stream processor". The stream processor may
> also be responsible for queuing, etc. If there no "external process",
> requesting the traffic, it may be routed to the port-owner...
> 
> Down-stream traffic may always be forced through port-owner, thus avoiding
> port locking problems (See below).
> 
>  +-------------------------------+
>  |   +-----------+               |
>  |   |           |               |
>  V   V           |               |
> +-----+        +--+            +--+
> |Owner| <--+   |P1|<----+      |Pn|<------+
> +-----+    |   +--+     |      +--+       |
>  |         |            |                 |
>  |         |            |                 |
>  |     +----------+ +----------+ +----------+ 
>  |     |StrmProc#1| |StrmProc#2| |StrmProc#n|
>  |     +----------+ +----------+ +----------+
>  |       A               A             A
>  V       |               |             |
> +-----------+            |             |
> |Port Driver|============+-------------+
> +-----------+
> 
> Where:
>  Owner      - Port Owner
>  StrmProc#n - Stream Processor n
>  Pn         - External Process n
> 
> Kind Regards,
> V.

I see no obvious gain with respect to the simple approach
(as you suggested first (as I understood it)), that would be:

 +------+   +----+          +----+
 |      |   |    |          |    |
+-----+ |   |  +--+         |  +--+
|Owner| |   |  |P1|         |  |Pn|
+-----+ |   |  +--+         |  +--+
 |      A   A    |          A    |
 V      |   |    V          |    V
+---------------------------------+
|         Port Driver             |
+---------------------------------+

The port lock is just an efficiency problem. All processes
simultaneously sending to a port will compete for the same lock.
Passing all downstream data through the port owner will only
introduce copying.

The port driver will in both cases have to know
(have a lookup table) of where to deliver messages
on a given stream. This is a currently missing feature.

Having erlang processes in the upstream data path will
also introduce copying. The only advantage I see with 
the StrmProc processes is that they can be created and
maintained through library code that can solve the
problem of notifying them if the port gets killed.
This problem can maybe just as well be solved by
documenting and by e.g an API function that when
registering a PID for a given stream will also set a
monitor on the port.

Would not sctp_peeloff() solve the right problems?
Or would it waste filedescriptors? Is it necessary
to have the possibility to multiplex streams on one
SCTP file descriptor even though it implies {active,true}.

 +------+      +----+        +----+
 |      |      |    |        |    |
+-----+ |      |  +--+       |  +--+
|Owner| |      |  |P1|       |  |Pn|
+-----+ |      |  +--+       |  +--+
 |      A      A    |        A    |
 V      |      |    V        |    V
+-----------+ +-----------+ +-----------+
|Port Driver| |Port Driver| |Port Driver|
+-----------+ +-----------+ +-----------+
   |     A       |  A          |  A
   V     |       V  |          V  |
+-----------+  +--------+    +--------+
| Main FD   |  | FD 1   |    | FD n   |
+-----------+--+--------+----+--------+
|           SCTP endpoint             |
+-------------------------------------+

Also, stream multiplexing with passive sockets
would only be possible with sctp_peeloff().

-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB