[erlang-questions] Garbage Collection, BEAM memory and Erlang memory

[Roberto Ostinelli]

Here's something I've tried which is successful in avoiding the memory
increase for binaries.

Inside a loop, I used to have:

    [...]
    <<Body:Len/binary, "\r\n", Rest/binary>> = Data,
    loop(Body, Rest);

Now I force a binary copy to ensure that the reference to the original full
binary is easily removed:

    [...]
    <<Body0:Len/binary, "\r\n", Rest0/binary>> = Data,
    Body = binary:copy(Body0),
    Rest = binary:copy(Rest0),
    loop(Body, Rest);

This seems to have stabilized the memory usage reported by erlang:memory/0.

However:

   - I believe this can only work if the copied binary are *heap* and not
   *ref-c*, is this correct?
   - Unfortunately, the BEAM process reported RES memory sill keeps growing.

Any other ideas?
r.




On Thu, Jan 22, 2015 at 6:11 PM, Roberto Ostinelli <roberto@REDACTED>
wrote:

> Thank you Robert.
> I'm going to try a selective fullsweep_after.
>
> Could this also justify the process memory increase (which is more
> significant)?
>
>
>
> On Thu, Jan 22, 2015 at 6:00 PM, Robert Virding <rvirding@REDACTED>
> wrote:
>
>> One thing you can see is that the size of the binary data is growing.
>> This space contains the large binaries (> 64 bytes) which are sent in
>> messages between processes. While this means that the messages become
>> (much) smaller and faster to send it takes a much longer time to detect
>> that they are no longer alive and can be reclaimed. Basically it takes
>> until all the processes they have passed through does a full garbage
>> collection. Setting fullsweep_after to 0 and doing explicit garbage
>> collects speeds up reclaiming the binaries.
>>
>> You could be much more selective in which processes you set
>> fullsweep_after to 0 and which ones you explicitly garbage collect.
>>
>> I don't know if the is *the* problem but it is *a* problem you have.
>>
>> Robert
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150122/7dee04c8/attachment.htm>