Sendfile in erlang

[Sean Hinde]

On Friday, November 14, 2003, at 11:27  am, Miguel Barreiro wrote:

> This draft patch adds a sendfile() interface to Erlang. Sendfile(2) is 
> a
> system call present in Linux 2.2, AIX 5 and later kernels, and similar
> interfaces are present in recent versions of Solaris and possibly other
> Unices. The usual loop of read()ing from a file and then send()ing to a
> socket or write()ing to another file has an unnecesarily large 
> overhead:
> copying data from kernel space to user space on read, and then back to
> kernel space again on write() or send(). Besides, if we are reading 
> from
> Erlang, that means getting all those data chunks into the erlang 
> runtime
> memory management system only to get them out again immediately and 
> then
> GC them sometime in the future. Very often (think of a web or file 
> server)
> our program has no use for that read data except sending it out again.
>
> Sendfile(f,t,o,c) simply instructs the kernel (the OS kernel, not
> $ROOTDIR/lib/kernel) to read c bytes at offset o of file descriptor f 
> and
> write them again to file descriptor t. No data is moved to/from user
> space.

This looks like an excellent addition to Erlang. I'd fully support this 
being adopted by the OTP team.

> ObPerfData: a cycle of file:read() and gen_tcp:send() moving 4KB chunks
> over 1000Base-T between 1GHz Pentium3 machines sustains a throughput of
> about 55Mbps. A cycle of file:sendfile() calls sustains over 410Mbps 
> down
> the pipe. Make sure you have a well supported network card before 
> trying.

The 55Mbps matches well with my measurements on a 1GHz PPC. We would 
use this tomorrow if it were beefed up with some of your suggestions 
and the normal OTP extra safe semantics.

Brilliant!

Sean