message passing between nodes using custom TCP

[Roberto Ostinelli]

dear all,

as some of you may already know i'm very interested in increasing  
message passing speed between erlang nodes. therefore, i started a  
simple test to see whether increasing the number of persistent TCP  
connections [i.e. pipes] between the nodes could speed things up.

the idea is simple.

- on the recipient node B, i start up a gen_server managing a  
listening TCP server and a series of recv processes which are spawned  
upon gen_tcp:accept/1;
- on the sending node A, i start up a gen_server managing a series of  
clients processes which create a number of connections to the TCP  
server on B.

this creates a certain number of TCP 'pipes' between A and B.

processes sending messages from node A to processes on node B pass  
through the gen_server on A, by using a function, instead of the  
native erlang sending functionality, such as:

send(Recipient, Message).

this function converts the erlang terms Recipient and Message to  
binary, then sends this binary through TCP using one of the available  
clients on A. the binary data sent is received by the associated recv  
process on B, which converts the received binary data to the original  
erlang term and sends it to the destination process.


the code i've developed works fine, and when i use just one TCP pipe  
the speed is only marginally slower to the native erlang sending  
functionality. however, when i use multiple TCP connections, the speed  
linearly degrades, and for instance when i set up something like 50  
parallel TCP channels it gets down by 40%.

i'm asking for a clue to have an understanding of the wrong  
assumptions i'm making, or what i could try more to achieve my  
objective. i do understand that this can be quite hard without seeing  
my code, but i have no problem in posting it if someone can help me  
out. i believe this can become quite interesting.

cheers,

r.