[erlang-questions] fast file sending - erlang & nginx

Morten Krogh mk@REDACTED
Sun Nov 21 12:43:04 CET 2010


Robert, what throughput do you get by sending a preloaded binary instead?

In other words, read the file into a binary once, and then just serve 
the replies from that binary.

Morten.


On 11/21/10 11:52 AM, Roberto Ostinelli wrote:
> dear list,
>
> i've just made a basic test to measure the sending of a small file
> (5080 bytes) via http, using erlang and nginx. code for erlang's file
> sending is appended here below. the results from ab benchmarks on an
> ubuntu box show that the throughput of nginx is around 5 times the one
> i can reach with erlang.
>
> i've developed a fastcgi backend in erlang [soon to be released on
> github], and using it i can reach comparable [if not faster]
> throughtputs on it than the ones i can reach with nginx. so, the fact
> that erlang is slower in sending static files cannot be due to the
> http handling itself.
>
> i decided to investigate if this was due to I/O reading speed.
>
> i've seen that joel has done some research on this using NIF to
> replace the file reading module on couchdb with the fd.erl module and
> it's related NIFs
> [https://github.com/wagerlabs/couchdb/tree/23527eb8165f81e63d47b230f3297d3072c88d83/src/couchdb],
> however if i'm not wrong i've seen that he is not using this module
> anymore, so i decided not test it.
>
> instead, i tried using the sendfile system call developed by steve
> vinosky sendfile_drv, packaged by tuncer
> [https://github.com/tuncer/sendfile], since in this way the file
> reading is anyway done by a fast C system call and i would be able to
> see if there are any differences. the results are that raw throughput
> remains around the same.
>
> therefore, i believe that the difference in throughput between nginx
> and this erlang code is not due to I/O issues. i'm wandering if nginx
> is nmapping files into memory, which would allow it to outperform in
> this way, or if i am missing a point here.
>
> i'd be glad to have pointers on how to improve fast file sending over http.
>
> cheers,
>
> r.
>
> =========================================
> -module(send_file).
> -include_lib("kernel/include/file.hrl").
> -compile(export_all).
>
> % sending of a file
> file_send(Socket, FilePath, Headers) ->
> 	% get file size
> 	case file:read_file_info(FilePath) of
> 		{ok, FileInfo} ->
> 			% get filesize
> 			FileSize = FileInfo#file_info.size,
> 			% do the gradual sending
> 			file_open_and_send(Socket, FilePath);
> 		{error, Reason} ->
> 			{error, Reason}
> 	end.
> file_open_and_send(Socket, FilePath) ->
> 	case file:open(FilePath, [read, binary]) of
> 		{error, Reason} ->
> 			{error, Reason};
> 		{ok, IoDevice} ->
> 			% read portions
> 			case file_read_and_send(Socket, IoDevice, 0) of
> 				{error, Reason} ->
> 					file:close(IoDevice),
> 					{error, Reason};
> 				ok ->
> 					file:close(IoDevice),
> 					ok
> 			end
> 	end.
> file_read_and_send(Socket, IoDevice, Position) ->
> 	% read buffer
> 	case file:pread(IoDevice, Position, ?FILE_READ_BUFFER) of
> 		{ok, Data} ->
> 			% file read, send
> 			gen_tcp:send(Socket, Data),
> 			% loop
> 			file_read_and_send(Socket, IoDevice, Position + ?FILE_READ_BUFFER);
> 		eof ->
> 			% finished
> 			ok;
> 		{error, Reason} ->
> 			{error, Reason}
> 	end.
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>



More information about the erlang-questions mailing list