[erlang-questions] How can we increase multiblock carrier utilization for binary_alloc?

[Gerhard Lazu]

Hi Jesper,

This is a great explanation, I was able to connect the missing dots.

Thank you very much, Gerhard.

On Wed, May 2, 2018 at 1:51 PM, Jesper Louis Andersen <
jesper.louis.andersen@REDACTED> wrote:

> On Tue, May 1, 2018 at 3:06 PM Gerhard Lazu <gerhard@REDACTED> wrote:
>
>>
>> During this exploration, there's a specific thing that's been bugging us:
>> why is RSS smaller than the allocated memory?
>>
>>
> This seems fairly obvious to me, but perhaps I am missing something. The
> Erlang system has allocated memory from the kernel, but the kernel has not
> yet handed that memory out to the process, and hence it is not in the RSS
> (Resident Set Size). As you hit new pages, there should be kernel traps, a
> page is allocated to the process (bumping RSS) and the program is resumed.
> If you allocate a larger carrier the "inner parts" of it might not be
> allocated before the first access.
>
> The other situation is that you have excessive memory pressure and the
> system starts removing pages which are possible to remove (they either bear
> no data, have already been written to the page/swap file on disk, have
> libraries in them, or madvise(2) have been called and told the OS that the
> pages are not needed[0]).
>
> In general, most systems with a GC doesn't give memory back to the OS
> straight away. They either keep it indefinitely (perhaps with an madvise(2)
> on the areas they don't need), or they have a reaper later on which can
> give back fully unused "spans" of memory (Go does this, for instance, but
> it isn't instant).
>
>
> [0] Be *very* cautious with madvise(2) since its implementation semantics
> are slightly different on Linux/BSD/Illumos. especially around
> DONTNEED/FREE and friends, which have different semantics. In particular,
> if a don't need a page right now, is the operating system allowed to hand
> me a zeroed page later if I start going for that memory again. Bryan
> Cantrill has some interesting pointers w.r.t. Lx-branded Illumos Zones and
> this :)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180503/943bf142/attachment.htm>