[erlang-questions] Setting maximum amount of data allocated by an Erlang node.‏

[Jesper Louis Andersen]

On Sat, Sep 5, 2015 at 1:53 AM, Akash Chowdhury <achowdhury918@REDACTED>
wrote:

> b)      for maximum amount of data allocated by an Erlang node


One thing that has not been touched yet is that the Erlang runtime is a bit
different in its allocation policy compared to many other programming
languages with garbage collectors. In the other category, the runtime wants
as much memory as possible, and you can make the system run faster by
giving it more memory to play with. The thing here is that it improves
productivity of the system because garbage collection happens more rarely
than it noramlly would do. So you can make a trade-off: Lower the available
memory the GC has available. This lowers productivity, but makes the
program stay within bounds.

The BEAM Erlang runtime system on the other hand works differently. It
usually doesn't allocate data unless it needs it, and it tends to grow
processes slowly as they need more and more memory. Hence, the BEAM has
much smaller overhead in memory compared to a GC solution.

This suggests that if a system uses 100 Gigabytes of memory, then this is
almost all taken up by something in the program. Thus, it will not help
setting a system limit, which will just crash the VM. You will have to make
sure your system doesn't overallocate memory by looking at the amount of
data it is storing and limiting itself. The first step should be, as
Theepan writes, to investigate what takes up memory in the system, and then
firgure out hwo to limit that memory usage.

Alternatively, if you can't compress memory more, is to look at
distribution. But this open s up a nother can of worms because distribted
systems have their own set of complications you then have to handle.
However, if you can't keep your data resident on one machine, you either
have to look into storing data temporarily on disk. Or to employ more than
a single machine containing the data in memory.



-- 
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150905/9c513697/attachment.htm>