[erlang-questions] How can we increase multiblock carrier utilization for binary_alloc?

[Jesper Louis Andersen]

On Tue, May 1, 2018 at 3:06 PM Gerhard Lazu <gerhard@REDACTED> wrote:

>
> During this exploration, there's a specific thing that's been bugging us:
> why is RSS smaller than the allocated memory?
>
>
This seems fairly obvious to me, but perhaps I am missing something. The
Erlang system has allocated memory from the kernel, but the kernel has not
yet handed that memory out to the process, and hence it is not in the RSS
(Resident Set Size). As you hit new pages, there should be kernel traps, a
page is allocated to the process (bumping RSS) and the program is resumed.
If you allocate a larger carrier the "inner parts" of it might not be
allocated before the first access.

The other situation is that you have excessive memory pressure and the
system starts removing pages which are possible to remove (they either bear
no data, have already been written to the page/swap file on disk, have
libraries in them, or madvise(2) have been called and told the OS that the
pages are not needed[0]).

In general, most systems with a GC doesn't give memory back to the OS
straight away. They either keep it indefinitely (perhaps with an madvise(2)
on the areas they don't need), or they have a reaper later on which can
give back fully unused "spans" of memory (Go does this, for instance, but
it isn't instant).


[0] Be *very* cautious with madvise(2) since its implementation semantics
are slightly different on Linux/BSD/Illumos. especially around
DONTNEED/FREE and friends, which have different semantics. In particular,
if a don't need a page right now, is the operating system allowed to hand
me a zeroed page later if I start going for that memory again. Bryan
Cantrill has some interesting pointers w.r.t. Lx-branded Illumos Zones and
this :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180502/faf81eb9/attachment.htm>