[erlang-questions] System limit bringing down rex and the VM

[Musumeci, Antonio S]

And what if the needs are dynamic in nature? Or if I *want* the limit and need to take some action when it occurs? There is no technical reason not to handle the error. It's purely a design decision or something that was overlooked. Limits are for limiting. Crashing is not an appropriate action to a definite and controllable problem. If you didn't care about whether or not something crashed there would be no limits at all. This is no different from a whole OS crashing because some process consumed too much memory or file handles.

I shouldn't have to build my own spawn wrapper to keep track of the number of processes. The VM already does this. Besides, this problem couldn't be fully addressed that way. This is caused by mnesia reacting to some event causing mnesia_recover to send a rpc. I'm not in control of that.

-----Original Message-----
From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Mark Scandariato
Sent: Wednesday, September 08, 2010 2:00 PM
To: erlang-questions@REDACTED
Subject: Re: [erlang-questions] System limit bringing down rex and the VM

Last time I looked, the process limit was there to allow the process table to be preallocated at startup. The default limit is 32768.

It is not at all ridiculous to determine the needs of your application and increase the process limit to match.

On Wed, Sep 8, 2010 at 1:46 PM, Musumeci, Antonio S < Antonio.Musumeci@REDACTED> wrote:

> I'm sorry but that's ridiculous. Limits exist to *limit* not to crash. 
> I see no reasons that rex should cause the VM to fail because it can't 
> spawn a child to handle the rpc request. When a node is down it 
> returns {badrpc,nodedown}. If a process can't be spawned shouldn't it 
> return a similar error? At least be configured to not have the 
> supervisor exit and therefore bring down the entire system?
>
> [1] That's not what is being argued. Obviously spawn will fail if 
> there are too many processes. The problem is that rex exits when it 
> happens which causes the entire system to fail. Does any modern OS 
> suddenly reboot when the same type of limit is reached? Do they shut 
> down when virtual memory is exhausted? No, they return errors and 
> allow the developer to handle the issue as well as the situation allows.
>
> [2] The Linux kernel for example keeps a certain amount of memory for 
> itself. Besides, just because you can't acquire some resource at T0 
> does not mean 1) you need the same resource to handle the error and/or 
> 2) that it won't be available at time T1.
>
> Increasing the process count isn't a solution. It's a hack that pushes 
> the problem out further.
>
> -----Original Message-----
> From: Jachym Holecek [mailto:freza@REDACTED]
> Sent: Wednesday, September 08, 2010 1:08 PM
> To: Musumeci, Antonio S (Enterprise Infrastructure)
> Cc: Igor Ribeiro Sucupira; erlang-questions@REDACTED
> Subject: Re: [erlang-questions] System limit bringing down rex and the 
> VM
>
> # Musumeci, Antonio S 2010-09-08:
> > OS limits? This is the BEAM process limit. [...]
>
> Which you can adjust with '+P <number>' emulator flag to a 
> sufficiently large value and be done with it, in the spirit of what Igor suggests.
>
> You're supposed to provide the runtime system with enough resources 
> (be it OS or emulator settings) to handle the expected load. Dealing 
> with this kind of errors "more gracefully" would be too much pain[1] 
> or simply impossible[2], AFAIU.
>
> Regards,
>        -- Jachym
>
> [1] Process table limit -- every single call to any variant of spawn() 
> may fail
>    this way. There's plenty of those. There's a similar limit on # of 
> ETS tabs.
>
> [2] Out of memory -- if there's no memory, where do you get the memory 
> to deal
>    with OOM error?
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>