[erlang-questions] Why Beam.smp crashes when memory is over?

[Ulf Wiger]

Richard O'Keefe wrote:
> 
> On Nov 11, 2009, at 11:26 AM, Ulf Wiger wrote:
> 
>> Richard O'Keefe wrote:
>>
>>> I am not denying the *need* for limits.  (Anyone else remember "engines"
>>> in Scheme?)  I've had enough functions in enough languages go into
>>> infinite recursion that I can see the point of stopping them.
>>> However, Ulf Wiger has not addressed these points:
>>> - if the tagging scheme is changed (and it has in the past),
>>>   memory requirements may change.
>> > ...
>>
>> Perhaps I'm colored by having worked on systems where every
>> product release was preceded with months of relatively
>> thorough testing, where monitoring memory usage during
>> important operating conditions was routine.
> 
> Let's see if we can reach consensus here.

I think we do agree. Hopefully the volley will clarify things
to the rest of the crowd - let's go another round and see if
people will start groaning. :)

> What I'm saying is that you have to *KEEP* doing it,
> as indeed you did ("every product release"), and that
> you have to keep doing it even if YOUR code didn't
> change at all.

Indeed. For high-availability products, this ought to
be a given. But as Erlang is also used in many other settings,
I would like to add an observation - perhaps obvious to all:

Erlang has some limits /today/ that programmers have to live
with. In many cases, the limits are high enough that they
are no cause for concern. Some of the better-known limits are:

- the number of simultaneous processes in the system
   (default: 32768, but can be raised up to 268435456)
- the number of ETS tables (default: 1400, can be raised considerably)
- the number of open ports and file descriptors (basically
   an OS limit)

Other system limits can be found in
http://www.erlang.org/doc/efficiency_guide/part_frame.html
and most are implementation details and subject to change.

What happens if you run into a system limit, e.g. if you
try creating too many ets tables, is that the operation
fails with an exception. Out of memory is a rather special
case, in that it brings down the entire VM.

Adding a few more limits that affect individual processes
rather than suffering the node-global disaster of OOM,
is thus nothing new, and in that sense shouldn't be
terribly controversial in itself. The reasonable /default/ is
of course that processes behave as today, i.e. the heap
and message queue length limits et al would be /optional/.

Having said this, it is certainly important to discuss which
such limits would actually be useful in practice.

> How you do that thorough testing is something that needs
> to be written up clearly in some new Erlang book, which
> I wish you would write.  I would certainly buy it.

Ok, at least one potential customer. I'll think about it. :)

BR,
Ulf W
-- 
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com