[erlang-questions] node to node message passing

[Joe Armstrong]

Interesting - I think what you want is a message passing layer that "just works"
this should survive virtually all failures. What happens if you send a
large message
between A and B and B is "totally destroyed" during the sending?

Think B is a laptop and has just been crushed with a sledge hammer.

You buy a new B - reinstall the software and ... should the data
transfer proceed?

I think you need an introspection layer for things like this - you
need to start the
transfer, then go back later and ask how it it doing? - you might get the answer
"B is broken - waiting to be repaired" - but having said "transfer X
from A to B"
the system should "try forever" to do this.

You might want "transfer X from A to B and tell Y if you can't do this
within time T"
which is completely different.

Exactly what is "built-in" to the standard inter-process message
passing semantics
is a mute point. I think what you want belongs to a separate library function.

Actually I have > 1 macs at home - with bad wireless connectivity between
certain rooms - so this is a not uninteresting problem.

Some kind of "bit-torrent" algorithm might be better - suppose
connectivity between A and
C is bad - but between A and B and B and C is good - what then?

A very interesting problem ... some kind of gossip algorithm with
bit-torrent type
fragment sharing seems in order. Since it's your own network, bit-torreent type
fairness is not relevant ...

You have twiched my programming nerve ...

/Joe






On Mon, Sep 13, 2010 at 11:38 PM, Tony Rogvall <tony@REDACTED> wrote:
> Hi Morten!
>
> On 12 sep 2010, at 12.48, Morten Krogh wrote:
>
>> Hi Erlangers.
>>
>> During some test with node to node communication, I sent a large binary from a process on node A
>> to a process on another node, node B. I also sent some smaller messages from other processes on node A to other
>> processes on node B. It turned out that the large message blocked the later messages. Furthermore, it even blocked
>> the net tick communication, so node A and B disconnected from each other even though the large message was being transferred!
>>
>
> This is one of the things that should have been fixed a long time ago (I think)
> May be I am also the one that should have done it while I had the chance ;-)
>
>
>> After looking a bit around, I have come to the understanding that Erlang uses one tcp connection between two nodes, and messages are sent
>> sequentially from the sending node A to the receiving node.
>>
>> If that is correct, I think some improvements are needed.
>>
>
> I can only agree here.
>
>> The problem to solve is basically that small messages, including the net tick, should get through more or less independently of
>> the presence of large messages.
>>
>> The simplest would be to have several connections, but that doesn't fully solve the problem. A large message will still take up
>> a lot of the hardware bandwidth even on another tcp connection.
>>
>> My suggestion is something like the following.
>>
>> For communication between node A and node B, there is a process (send process) on each node, that coordinates all messages. The send process
>> keeps queues of different priorities around, e.g., a high priority, medium priority and low priority. Messages are split up into fragments of
>> a maximum size. The receiver(node B) send process assembles the fragments into the original message and delivers it locally to the
>> right process. The fragments ensure that no single transfer will occupy the connection for very long.
>> There will be a function send_priority where the user can specify a priority. The usual send will default to medium, say.
>> Net tick will use high priority, of course. Small messages that are needed to produce a web application response can have high priority. File transfers
>> for backup purposes can have low priority.
>> The send process then switches between the queues in some way, that could be very similar to context switching priorities.
>
> Yes, multiplexing over one TCP channel I think is a reasonable way to go.
>
>>
>> More advanced, the send processes could occasionally probe the connection with packets to estimate latency and bandwidth. Those figures could then be used
>> to calculate fragment sizes. High bandwidth, high latency would require large fragments. Low bandwidth, low latency small fragments for instance.
>> There could even be a function send_estimated_transfer_time that sends a message and has a return value of estimated transfer time, which could be used in
>> a timeout in a receive loop.
>>
>
> Why not take one step at a time, Erlang/OTP is and should be a process of improvements. Trying to do too much
> is just going to delay the implementation finding it's way into the main branch.
>
>>
>> I have actually implemented my own small module for splitting messages into fragments, and it solves the issues; net tick goes through, and small
>> messages can overtake large ones.
>>
> Can't wait to see the implementation, do you have a git somehere ?
>
>> There is of course an issue when the sending and receiving process is the same for several messages. Either the guaranteed message order should be given up, or the
>> coordinators should keep track of that as well. Personally, I think guaranteed message order should be given up. Erlang should model the real world as
>> much as possible, and learn from it. In the real world, two letters going from person A to person B, can definitely arrive in the opposite order
>> of the one in which they were sent. And as node to node communication will be over larger and larger distances, it is totally unnatural to require
>> a certain order.
>
> Lets try to stick to the existing "semantics" here, then the implementation will have a chance to get realized.
>
>>
>> I am relatively new to Erlang and I really enjoy it. Kudos to all involved!
>>
> Well you certainly stumbled into the right place ;-)
>
> Regards
>
> /Tony
>
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>