[erlang-questions] Why Beam.smp crashes when memory is over?

[Richard O'Keefe]

On Nov 9, 2009, at 10:53 PM, Kenneth Lundin wrote:
> What we could do is to make it easier for the user to prevent OOM
> situations and also to
> let him take the decision when it occurs or rather before it occurs.
>
> One way would be to let the user set a memory quota on a process with
> options at spawn time. When the process reaches it quota it can be
> automatically killed or the user can
> be notified in some way and take actions.

One of the reasons this hasn't been done is, I presume, the fact that
it is quite difficult for a programmer to determine what the memory
quota should be.  It depends on
  - the nature of the code being run
  - the data it is processing (data loads tend to increase with the  
years)
  - the cleverness of the compiler
  - the data representation used by the Erlang system (e.g., whether
    pointers are 32 bits or 64, whether there are special tags for small
    arrays
  - the data representation used by library modules the code calls
    (change the representation of ordered sets and a bound that might
    have worked might not any more)
  - a system policy on whether it's better to be tight on memory and
    tolerate some processes crashing and needing to be restarted or
    whether it's best to keep more processing running at the cost of
    using more memory, a thing that might change at run time.

I have long bemoaned the (measured!) fact that the size of a stack
frame in C can vary by a factor of 10, so that determining the sizes
for POSIX thread stacks is a game of Russian Roulette.

You can determine suitable sizes for a particular release by running
experiments, but the sizes thus determined are ONLY reliable for the
specific release or releases you tried on the machine or machines
that you tried it or them on.  Set a limit, and you can *EXPECT*
working processes to be crashed as a *NORMAL* outcome.

Once some feature goes in, I expect to see messages in this mailing
list "My program gets lots of killed processes due to Out-Of-Memory
but it used to work and I haven't changed it."