[erlang-questions] Why Beam.smp crashes when memory is over?

[Richard O'Keefe]

On Nov 11, 2009, at 2:13 AM, Kenneth Lundin wrote:
> If you for example have processes representing subscribers och
> half-calls or mobile phones
> it is pretty easy to know a reasonable size of such a process by  
> measurement.

For  THIS version of the program
with THIS version of the Erlang libraries, compiler, VM
and  THIS build (because C space requirements can depend on C
           compiler settings)
on   THIS hardware
with THIS workload.

Measurement is a good thing, but when any of these factors changes,
it's advisable to measure again.  In particular, if I develop something
on my box and measure and set an appropriate bound, and I send you the
code, you have no reason to believe that the bound is appropriate for
your circumstances until you have measured.

Only the other day I was explaining to some students that the reason
the SPARK subset of Ada can be used safely in embedded systems is that
the SPARK subset excludes
  - recursion
  - pointers and dynamically allocated objects
  - dynamic array sizes
so that the compiler can be sure everything fits.  Erlang has all
the things that make this hard.

> It should be easy to say if it can grow up to 100 kwords, 1Mwords, 10
> Mwords etc.

I can't do that for small C programs.  I've seen C programs where
the stack space varied by a factor of 10 between different hardware/
compiler/options choices.  Come to that, with some C compilers
supporting tail call optimisation and some not, it's quite easy to
construct artificial examples where the memory needed by a thread
could be as low as 1 KB or as high as 1 GB.

> Of course you should not try to set quota as tight as possible. The
> quota on process
> size is just a way to protect the system from total disaster if one of
> these processes
> happens to grow over all reasonable limits because of a bug.

There are two separable issues.
  A. How do we detect a runaway process and stop it ruining things
     for everyone else?  Loose quotas are just fine for this.  Even
     a factor of 10 too large is good enough.  (Maybe.)
  B. How do we handle memory exhaustion without killing the VM
     *when no process has exceeded its quota*.  Loose quotas actually
     create this problem.  (In the present system, the quotas are as
     loose as they could possibly be...)

I have no quarrel with optional loose quotas as a debugging/safety
measure, a solution to problem A.  It's just that we shouldn't
mistake them for a solution to problem B, which is what I thought
this thread was about.  Tackling problem B should make a nice PhD
for someone.  (While there are some, um, questionable decisions in
the design of the Java _language_, a lot of smart people have
worked on Java _implementation_, and if there were a good solution
to problem B I think some Java vendor would be boasting about it.)