Sendfile in erlang

Sean Hinde sean.hinde@REDACTED
Fri Nov 14 14:45:38 CET 2003


On Friday, November 14, 2003, at 11:27  am, Miguel Barreiro wrote:

> This draft patch adds a sendfile() interface to Erlang. Sendfile(2) is 
> a
> system call present in Linux 2.2, AIX 5 and later kernels, and similar
> interfaces are present in recent versions of Solaris and possibly other
> Unices. The usual loop of read()ing from a file and then send()ing to a
> socket or write()ing to another file has an unnecesarily large 
> overhead:
> copying data from kernel space to user space on read, and then back to
> kernel space again on write() or send(). Besides, if we are reading 
> from
> Erlang, that means getting all those data chunks into the erlang 
> runtime
> memory management system only to get them out again immediately and 
> then
> GC them sometime in the future. Very often (think of a web or file 
> server)
> our program has no use for that read data except sending it out again.
>
> Sendfile(f,t,o,c) simply instructs the kernel (the OS kernel, not
> $ROOTDIR/lib/kernel) to read c bytes at offset o of file descriptor f 
> and
> write them again to file descriptor t. No data is moved to/from user
> space.

This looks like an excellent addition to Erlang. I'd fully support this 
being adopted by the OTP team.

> ObPerfData: a cycle of file:read() and gen_tcp:send() moving 4KB chunks
> over 1000Base-T between 1GHz Pentium3 machines sustains a throughput of
> about 55Mbps. A cycle of file:sendfile() calls sustains over 410Mbps 
> down
> the pipe. Make sure you have a well supported network card before 
> trying.

The 55Mbps matches well with my measurements on a 1GHz PPC. We would 
use this tomorrow if it were beefed up with some of your suggestions 
and the normal OTP extra safe semantics.

Brilliant!

Sean




More information about the erlang-patches mailing list