[erlang-bugs] file_server processes very large numbers of messages slowly

[Alexandru Scvorţov]

Hi,

If you try to do a lot of IO concurrently (e.g. 100 000 calls to
file:read_file_info/1), you notice that the calls take longer and longer
as the number of concurrent requests rises.

As far as we (RabbitMQ) can tell, the bottleneck seems to be
file_server.

For instance, running the attached program (100 000 calls to file:r_f_i) with:
  erlc breaky.erl && erl +P 1048576 -s breaky break_fileserver -s init stop
you can see this happening clearly.  The first calls, which happen when
file_server is congested with a large number of messages on its queue,
take a long time to run; as the messages are processed, the processing
speed increases as well (the first column shows the time to serve 1000
requests).

Our guess is that this happens because of the selective receive in
prim_file:drv_get_response/1.  That seems to cause cause a full scan of the
mailbox (which is obviously takes longer when there are a lot of messages).

Does this sound right?  Is there any workaround other than not doing a lot
of file IO concurrently?

Cheers,
Alex

-------------- next part --------------
-module(breaky).
-compile([export_all]).

break_fileserver() ->
    N = 100000,
    Self = self(),
    [spawn(fun () ->
                   {ok, _} = file:read_file_info("breaky.erl"),
                   Self ! done
           end) || _ <- lists:seq(0,N)],
    gather(erlang:now(), N).

gather(_, 0) ->
    ok;
gather(Now, N) ->
    Now2 = if N rem 1000 =:= 0 ->
                   Now1 = erlang:now(),
                   io:format("~p: ~p ~p~n", [timer:now_diff(Now1, Now) / 1000,
                                             N,
                                             process_info(whereis(file_server_2),
                                                          message_queue_len)]),
                   Now1;
              true -> Now
    end,
    receive
        done -> gather(Now2, N-1)
    end.