[erlang-questions] Copy big files over network as fast as possible

[Tom Samplonius]

----- yerl@REDACTED wrote:
> Hi All!
> 
> I've a bunch of big files (~500.000 files, each one >100Mb). 
> I want to copy them as fast as possible from one machine to another. 
> Both machines are in the same 1Go network.
> 
> It's a research problem in physic fields. A team of scientists from
> CERN come with FDT:
> 
> http://monalisa.cern.ch/FDT
> 
> the fastest solution available at this time, but in Java :-[ .
> 
> So, I start thinking to implement my solution in Erlang.
> 

  FDT looks pretty complicated.  But you could write a C driver for sendfile() (which is available on Linux and several OSes), and use that to control file transmission.

  I would stick with TCP.  Just increase the window size.  There is no need to hack something up in UDP.

  The problem with sendfile() is reception of the files.  A more general solution would be a C driver for erlang that can malloc and mmap() a region of memory.  Writing from an mmap()'ed file is basically the same as sendfile anyways, and works on more OSes than sendfile().  And you can use the same mechanism to receive data too.

  And mmap() sounds a lot like what FDT is doing with its "Directly mapped buffers".


Tom