[erlang-questions] Increase the handle limit

[Martin Dimitrov]

The error returned on Windows after allocating 1200 handles is system_limit.

Regards,

Martin

On 10/21/2011 5:49 PM, Peer Stritzinger wrote:
> On Fri, Oct 21, 2011 at 9:35 AM, Joel Reymont <joelr1@REDACTED> wrote:
>> I don't think you can.
>>
>> I bump against this all the time on the Mac.
>>
>> Erlang uses the FD_SETSIZE setting from the system header files when compiled, normally 1024.
> I took the time and wrote a little test because this sounded too
> unbelievable, but nobody disagreed.
>
> TL;DR Version: This is not true.  Tested on MacOSX 10.5.8 on PowerPC
> with Erlang R14B03
> (used the oldest version of Mac OSX anybody sane would run and the
> Erlang is also not cutting edge version):
>
> Could open about 56000 files the I hit some OS system limit.  Didn't
> try to increase further since this already 50 times the amount of
> FD_SETSIZE which is indeed set to 1024 on my system.
>
> In order to be able to do this I had to do this:
>
> Set ERL_MAX_PORTS (see erlang manpage) to increase the number of ports.
>
> Set ulimit -n to allow more open files in the process
>
> Start Erlang with +P to allow enough processes
>
> With this settings I get up to about 10000 open files since I can't
> set the ulimit above the system limit.  In order to get more I had to
>
>       sysctl -w kern.maxfiles=120000
>       sysctl -w kern.maxfilesperproc=100000
>
> The increase ulimits -n and rerun the test:
>
>     {enfile,56016}
>
> Error enfile means from open(2):
>
>     [ENFILE]           The system file table is full.
>
> Since I set maxfiles to 120000 there should have been more files
> possible, don't know what limit I hit, maybe number of files in a
> directory?
>
> Definitely not a Erlang limit though.
>
> The program I used is below.
>> Editing this value and re-compiling Erlang causes epmd to stop working.
> This is no wonder.  Editing the header doesn't change the system call
> internals, probably epmd uses select() which fails because the system
> call  won't handle sets with increased set size.
>
> Editing the system headers is definitely not to recommend.  It may
> break all kinds of libraries in a subtle way.
>
>> My solution is to save the old epmd and recompile after editing the system header files.
>>
>> On Oct 21, 2011, at 7:25 AM, Martin Dimitrov wrote:
>>
>>> Hi all,
>>>
>>> I am developing on a Windows 7 machine, Erlang R14B04. The limit of open
>>> handles seems to be 1200. But this is not imposed by the operating
>>> system (Windows has a hard coded limit of 16 million handles). So my
>>> question is how can I increase the handle limit in Erlang?
> All this probably won't help you solving the Windows problem in a
> systematic way, unfortunately (actually fortunately I think ;-) I'm no
> Windows expert.
>
> Looking the the corresponding source files of ERTS might help:
>
> * find out how Erlang finds out about file descriptor changes
>
> * find out where the limit comes from
>
> Also what might lead you to the reason of the limit is the exact error
> message you get when you hit the limit.
>
> Cheers
> -- Peer
>
> Little program used to find limit and reason:
>
> -module(test_fds).
> -compile([export_all]).
>
> count() ->
>     count(1, []).
>
> count(N, Fds) ->
>     case file:open(integer_to_list(N), [write]) of
> 	{ok, F} ->
> 	    count(N+1, [F| Fds]);
> 	{error, Err} ->
> 	    [ file:close(F) || F <- Fds ],
> 	    {Err, N}
>     end.
>
>
>> --------------------------------------------------------------------------
>> - for hire: mac osx device driver ninja, kernel extensions and usb drivers
>> ---------------------+------------+---------------------------------------
>> http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
>> ---------------------+------------+---------------------------------------
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>