[erlang-questions] Copy big files over network as fast as possible

[Christian S]

> I've a bunch of big files (~500.000 files, each one >100Mb).
> I want to copy them as fast as possible from one machine to another.
> Both machines are in the same 1Go network.

> Has anybody faced this problem before ?
> Any advice will be very welcome.

I know people dealing with warez in my univ days gave up on TCP for asocial
UDP protocols that transmitted at full speed, never throttling at increased
packet collisions. It caused bandwith starvation for other users, but the
warez people sustained near 10Mbit which was the teoretical maximum
on that lan.

It might be worthwile to take this route, to build your own stream protocol
over UDP, one that is better suited for your extreme needs. Of course these
warez people were mostly out to take other's bandwidth share. You're alone
on your Gbit LAN?

It might be worthwile to implement a driver for linux's sendfile
syscall, if not only
just to avoid stressing erlang's garbage collector.