[erlang-questions] : : : : sctp_peeloff() support in OTP

['Raimo Niskanen']

On Wed, Apr 22, 2009 at 06:00:25PM +0200, Valentin Micic wrote:
> One of the problems that I see with SCTP is that, unlike TCP where you have
> no choice but to send the reply over the same stream you've received request
> through; SCTP AFAIK does not regulate this -- you could receive a request
> over, say, stream 6 and send a reply using stream 54. Whilst this problem
> exists irrespective on how one implement stream separation, using separate
> file descriptor may look a bit wasteful from local resources point of view.
> 
> If it was up to me, I would invest a bit more time in attempt to avoid usage
> of multiple file descriptors. Could one have a number of "pseudo-ports"
> mapped to the same "real" port/file descriptor?

Possible. But what I think is needed is a solution for the
head-of-line blocking problem. That is: flow control per stream.

In what way is it useful to dispatch messages from the
driver level onto a PID dedicated for a streams if
you do not get flow control per stream?

> 
> Best
> V. 
> 
> -----Original Message-----
> From: 'Raimo Niskanen' [mailto:raimo+erlang-questions@REDACTED] 
> Sent: 22 April 2009 10:36 AM
> To: Valentin Micic; erlang-questions@REDACTED
> Subject: Re: : : [erlang-questions] : sctp_peeloff() support in OTP
> 
> On Tue, Apr 21, 2009 at 05:58:47PM +0200, Valentin Micic wrote:
> > 
> > >> implementing it without defeating the purpose behind SCTP?
> > 
> > > Could you elaborate...?
> > 
> > Let me try by saying that TCP was not adequate for "multiplexing", a
> traffic
> > case where a bunch of things on one machine wants to talk to a bunch of
> > things on another using the same connection. The SCTP addressed this issue
> > on a number of levels, but most importantly -- by eliminating a
> head-of-line
> > blocking bottleneck (someone mentioned a year or two ago that SCTP would
> be
> > far better protocol for Erlang distribution).
> > 
> > The reasoning behind SCTP is to provide for a minimal latency, where
> > processing of one stream should not create a processing delay on another.
> > Having the same port owner responsible for all the processing is at odds
> > with this design goal. In other words, {active, once} would effectively
> > reintroduce head-of-line blocking, by serializing all the traffic and
> > forcing it through a single process.
> > 
> 
> Thank you for the explanation. The term "head-of-line blocking"
> was new to me.
> 
> Unfortunately I just realized that when the driver calls recvmsg()
> it can not choose which stream to receive from. So {active,once}
> must introduce head-of-line blocking in some form. And introducing
> {active,StreamNo,Pid} would have to imply {active,true},
> just dispatching the streams.
> 
> I think the only thing that solves the head-of-line blocking
> problem is branched-off associations (sctp_peeloff()) because
> then you get a new filedescriptor to poll() and recvmsg() as
> you choose from, in a new port driver instance (Port).
> Correct me if I am wrong.
> 
> > Not so well thought through approach to active socket processing:
> > -----------------------------------------------------------------
> > Maybe one may mitigate this by creating a number of Erlang processes as
> > there are streams (call it stream processors), and route upstream traffic
> to
> > a particular stream processor? Then, if there is an "external process"
> > requesting {active, [once|true, StramNo, PID]} that relationship may be
> > remembered by that particular "stream processor". The stream processor may
> > also be responsible for queuing, etc. If there no "external process",
> > requesting the traffic, it may be routed to the port-owner...
> > 
> > Down-stream traffic may always be forced through port-owner, thus avoiding
> > port locking problems (See below).
> > 
> >  +-------------------------------+
> >  |   +-----------+               |
> >  |   |           |               |
> >  V   V           |               |
> > +-----+        +--+            +--+
> > |Owner| <--+   |P1|<----+      |Pn|<------+
> > +-----+    |   +--+     |      +--+       |
> >  |         |            |                 |
> >  |         |            |                 |
> >  |     +----------+ +----------+ +----------+ 
> >  |     |StrmProc#1| |StrmProc#2| |StrmProc#n|
> >  |     +----------+ +----------+ +----------+
> >  |       A               A             A
> >  V       |               |             |
> > +-----------+            |             |
> > |Port Driver|============+-------------+
> > +-----------+
> > 
> > Where:
> >  Owner      - Port Owner
> >  StrmProc#n - Stream Processor n
> >  Pn         - External Process n
> > 
> > Kind Regards,
> > V.
> 
> I see no obvious gain with respect to the simple approach
> (as you suggested first (as I understood it)), that would be:
> 
>  +------+   +----+          +----+
>  |      |   |    |          |    |
> +-----+ |   |  +--+         |  +--+
> |Owner| |   |  |P1|         |  |Pn|
> +-----+ |   |  +--+         |  +--+
>  |      A   A    |          A    |
>  V      |   |    V          |    V
> +---------------------------------+
> |         Port Driver             |
> +---------------------------------+
> 
> The port lock is just an efficiency problem. All processes
> simultaneously sending to a port will compete for the same lock.
> Passing all downstream data through the port owner will only
> introduce copying.
> 
> The port driver will in both cases have to know
> (have a lookup table) of where to deliver messages
> on a given stream. This is a currently missing feature.
> 
> Having erlang processes in the upstream data path will
> also introduce copying. The only advantage I see with 
> the StrmProc processes is that they can be created and
> maintained through library code that can solve the
> problem of notifying them if the port gets killed.
> This problem can maybe just as well be solved by
> documenting and by e.g an API function that when
> registering a PID for a given stream will also set a
> monitor on the port.
> 
> Would not sctp_peeloff() solve the right problems?
> Or would it waste filedescriptors? Is it necessary
> to have the possibility to multiplex streams on one
> SCTP file descriptor even though it implies {active,true}.
> 
>  +------+      +----+        +----+
>  |      |      |    |        |    |
> +-----+ |      |  +--+       |  +--+
> |Owner| |      |  |P1|       |  |Pn|
> +-----+ |      |  +--+       |  +--+
>  |      A      A    |        A    |
>  V      |      |    V        |    V
> +-----------+ +-----------+ +-----------+
> |Port Driver| |Port Driver| |Port Driver|
> +-----------+ +-----------+ +-----------+
>    |     A       |  A          |  A
>    V     |       V  |          V  |
> +-----------+  +--------+    +--------+
> | Main FD   |  | FD 1   |    | FD n   |
> +-----------+--+--------+----+--------+
> |           SCTP endpoint             |
> +-------------------------------------+
> 
> Also, stream multiplexing with passive sockets
> would only be possible with sctp_peeloff().
> 
> -- 
> 
> / Raimo Niskanen, Erlang/OTP, Ericsson AB

-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB