[erlang-questions] node to node message passing

Jesper Louis Andersen <>
Tue Sep 14 12:41:40 CEST 2010


On Tue, Sep 14, 2010 at 11:04 AM, Joe Armstrong <> wrote:
>
> Some kind of "bit-torrent" algorithm might be better - suppose
> connectivity between A and
> C is bad - but between A and B and B and C is good - what then?

Bittorrent does nothing to protect against a killed TCP connection. If
the protocol gets a disconnect, it will simply grab another peer from
a pool of pending peers and try to connect to it. It may be the same
peer, or it may not. If the line is merely slow, the fragmentation of
the data ensures a grab eventually where the faster connections pass
more data around.

> Since it's your own network, bit-torreent type fairness is not relevant ...

Most of the bittorrent protocol is there to solve a specific problem:
most people are on slow, copper based xDSL type lines. These are
asymmetric and congest easily in the upstream path. Hence, a lot of
the algorithm is devoted to meticulously manage the limited upstream
bandwidth to balance the somewhat unlimited downstream bandwidth and
maximize it (until the file has been downloaded in which case we
change strategy). The algorithm is not that much about fairness. It
doesn't really care if anybody gets a "fair" share of the file. Rather
it tries to optimize the downstream bandwidth for the client - but to
do that the protocol is designed such that you must send data back.

If you ignore the sending, you will eventually get your file. The
point is just that you will get it later than everybody else. Another
interesting property is that clients "sort themselves" into classes
where a class is determined by its upload speed. Thus the fast peers
tend to get the file fast and then distribute it to other peers.

The brilliance of the protocol lies in swarm intelligence. Each peer
knows nothing -- but the information it has gathered itself. Each peer
use no historic data older than 10 seconds. Yet, data is transferred
at speeds that rival or surpass any other P2P network of critical mass
on the internet.


>
> You have twiched my programming nerve ...
>
> /Joe
>
>
>
>
>
>
> On Mon, Sep 13, 2010 at 11:38 PM, Tony Rogvall <> wrote:
>> Hi Morten!
>>
>> On 12 sep 2010, at 12.48, Morten Krogh wrote:
>>
>>> Hi Erlangers.
>>>
>>> During some test with node to node communication, I sent a large binary from a process on node A
>>> to a process on another node, node B. I also sent some smaller messages from other processes on node A to other
>>> processes on node B. It turned out that the large message blocked the later messages. Furthermore, it even blocked
>>> the net tick communication, so node A and B disconnected from each other even though the large message was being transferred!
>>>
>>
>> This is one of the things that should have been fixed a long time ago (I think)
>> May be I am also the one that should have done it while I had the chance ;-)
>>
>>
>>> After looking a bit around, I have come to the understanding that Erlang uses one tcp connection between two nodes, and messages are sent
>>> sequentially from the sending node A to the receiving node.
>>>
>>> If that is correct, I think some improvements are needed.
>>>
>>
>> I can only agree here.
>>
>>> The problem to solve is basically that small messages, including the net tick, should get through more or less independently of
>>> the presence of large messages.
>>>
>>> The simplest would be to have several connections, but that doesn't fully solve the problem. A large message will still take up
>>> a lot of the hardware bandwidth even on another tcp connection.
>>>
>>> My suggestion is something like the following.
>>>
>>> For communication between node A and node B, there is a process (send process) on each node, that coordinates all messages. The send process
>>> keeps queues of different priorities around, e.g., a high priority, medium priority and low priority. Messages are split up into fragments of
>>> a maximum size. The receiver(node B) send process assembles the fragments into the original message and delivers it locally to the
>>> right process. The fragments ensure that no single transfer will occupy the connection for very long.
>>> There will be a function send_priority where the user can specify a priority. The usual send will default to medium, say.
>>> Net tick will use high priority, of course. Small messages that are needed to produce a web application response can have high priority. File transfers
>>> for backup purposes can have low priority.
>>> The send process then switches between the queues in some way, that could be very similar to context switching priorities.
>>
>> Yes, multiplexing over one TCP channel I think is a reasonable way to go.
>>
>>>
>>> More advanced, the send processes could occasionally probe the connection with packets to estimate latency and bandwidth. Those figures could then be used
>>> to calculate fragment sizes. High bandwidth, high latency would require large fragments. Low bandwidth, low latency small fragments for instance.
>>> There could even be a function send_estimated_transfer_time that sends a message and has a return value of estimated transfer time, which could be used in
>>> a timeout in a receive loop.
>>>
>>
>> Why not take one step at a time, Erlang/OTP is and should be a process of improvements. Trying to do too much
>> is just going to delay the implementation finding it's way into the main branch.
>>
>>>
>>> I have actually implemented my own small module for splitting messages into fragments, and it solves the issues; net tick goes through, and small
>>> messages can overtake large ones.
>>>
>> Can't wait to see the implementation, do you have a git somehere ?
>>
>>> There is of course an issue when the sending and receiving process is the same for several messages. Either the guaranteed message order should be given up, or the
>>> coordinators should keep track of that as well. Personally, I think guaranteed message order should be given up. Erlang should model the real world as
>>> much as possible, and learn from it. In the real world, two letters going from person A to person B, can definitely arrive in the opposite order
>>> of the one in which they were sent. And as node to node communication will be over larger and larger distances, it is totally unnatural to require
>>> a certain order.
>>
>> Lets try to stick to the existing "semantics" here, then the implementation will have a chance to get realized.
>>
>>>
>>> I am relatively new to Erlang and I really enjoy it. Kudos to all involved!
>>>
>> Well you certainly stumbled into the right place ;-)
>>
>> Regards
>>
>> /Tony
>>
>>
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:
>>
>>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:
>
>



-- 
J.


More information about the erlang-questions mailing list