[erlang-questions] node to node message passing

[Tony Rogvall]

Hi Morten!

On 12 sep 2010, at 12.48, Morten Krogh wrote:

> Hi Erlangers.
> 
> During some test with node to node communication, I sent a large binary from a process on node A
> to a process on another node, node B. I also sent some smaller messages from other processes on node A to other
> processes on node B. It turned out that the large message blocked the later messages. Furthermore, it even blocked
> the net tick communication, so node A and B disconnected from each other even though the large message was being transferred!
> 

This is one of the things that should have been fixed a long time ago (I think)
May be I am also the one that should have done it while I had the chance ;-)


> After looking a bit around, I have come to the understanding that Erlang uses one tcp connection between two nodes, and messages are sent
> sequentially from the sending node A to the receiving node.
> 
> If that is correct, I think some improvements are needed.
> 

I can only agree here.

> The problem to solve is basically that small messages, including the net tick, should get through more or less independently of
> the presence of large messages.
> 
> The simplest would be to have several connections, but that doesn't fully solve the problem. A large message will still take up
> a lot of the hardware bandwidth even on another tcp connection.
> 
> My suggestion is something like the following.
> 
> For communication between node A and node B, there is a process (send process) on each node, that coordinates all messages. The send process
> keeps queues of different priorities around, e.g., a high priority, medium priority and low priority. Messages are split up into fragments of
> a maximum size. The receiver(node B) send process assembles the fragments into the original message and delivers it locally to the
> right process. The fragments ensure that no single transfer will occupy the connection for very long.
> There will be a function send_priority where the user can specify a priority. The usual send will default to medium, say.
> Net tick will use high priority, of course. Small messages that are needed to produce a web application response can have high priority. File transfers
> for backup purposes can have low priority.
> The send process then switches between the queues in some way, that could be very similar to context switching priorities.

Yes, multiplexing over one TCP channel I think is a reasonable way to go. 

> 
> More advanced, the send processes could occasionally probe the connection with packets to estimate latency and bandwidth. Those figures could then be used
> to calculate fragment sizes. High bandwidth, high latency would require large fragments. Low bandwidth, low latency small fragments for instance.
> There could even be a function send_estimated_transfer_time that sends a message and has a return value of estimated transfer time, which could be used in
> a timeout in a receive loop.
> 

Why not take one step at a time, Erlang/OTP is and should be a process of improvements. Trying to do too much
is just going to delay the implementation finding it's way into the main branch.

> 
> I have actually implemented my own small module for splitting messages into fragments, and it solves the issues; net tick goes through, and small
> messages can overtake large ones.
> 
Can't wait to see the implementation, do you have a git somehere ?

> There is of course an issue when the sending and receiving process is the same for several messages. Either the guaranteed message order should be given up, or the
> coordinators should keep track of that as well. Personally, I think guaranteed message order should be given up. Erlang should model the real world as
> much as possible, and learn from it. In the real world, two letters going from person A to person B, can definitely arrive in the opposite order
> of the one in which they were sent. And as node to node communication will be over larger and larger distances, it is totally unnatural to require
> a certain order.

Lets try to stick to the existing "semantics" here, then the implementation will have a chance to get realized.

> 
> I am relatively new to Erlang and I really enjoy it. Kudos to all involved!
> 
Well you certainly stumbled into the right place ;-)

Regards

/Tony