Max open files and erlang memory

[Vincent de Phily]

Hi list,

I've got some enormous system memory usage whenever I run erlang (32bit linux 
system) :

> $ erl
> Erlang R13B01 (erts-5.7.2) [source] [rq:1] [async-threads:0]
> [kernel-poll:false] 
>
> Eshell V5.7.2  (abort with ^G)
> 1> erlang:memory().
> [{total,302822080},
>  {processes,489036},
>  {processes_used,483292},
>  {system,302333044},
>  {atom,255645},
>  {atom_used,254795},
>  {binary,306280},
>  {code,1775473},
>  {ets,104668}]

That's close to 300MB for a bare shell. Too much, if you ask me :p

I've traced it back to the maximum number of open files allowed by the system. 
I use a big number here, to allow that many tcp connections to my server. If 
I reduce this limit, erlang memory usage returns to manageable levels:

> $ for i in 999999 900000 800000 700000 600000 500000 400000 300000 200000
> 100000 50000 10000 5000 1000 500 100 50; do ulimit -n $i; echo -n $i;
> erl -noinput -eval 'io:format(" ~p~n", [erlang:memory(system)])' -s erlang
> halt; done   
> 999999 301852400
> 900000 296652448
> 800000 291452448
> 700000 286252464
> 600000 281052464
> 500000 152120480
> 400000 146920480
> 300000 141720496
> 200000 74654496
> 100000 38521504
> 50000 20455024
> 10000 6775136
> 5000 4581840
> 1000 2682176
> 500 2656176
> 100 2634088
> 50 2629488

I understand that opening more files uses more memory, but I haven't opened 
anything yet. I shouldn't be paying the price for a system limit I haven't 
reached yet.
Also, note that the memory usage in the above example is not linear : there 
are jumps in the graph.

I am not 100% sure, but I think this is a regression : I tested my software 
with a big max-open-file limit before and hadn't noticed the memory usage.


For the immediate future, I'll size the system limit more conservatively. But 
I'm interested in a fix or workaround on the erlang side.


Thanks.

-- 
Vincent de Phily
Mobile Devices
+33 (0) 666 301 306
+33 (0) 142 119 325

Warning
This message (and any associated files) is intended only for the use of its
intended recipient and may contain information that is confidential, subject
to copyright or constitutes a trade secret. If you are not the intended
recipient you are hereby notified that any dissemination, copying or
distribution of this message, or files associated with this message, is
strictly prohibited. If you have received this message in error, please
notify us immediately by replying to the message and deleting it from your
computer. Any views or opinions presented are solely those of the author
vincent.dephily@REDACTED and do not necessarily represent those of 
the
company. Although the company has taken reasonable precautions to ensure no
viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments.