[erlang-questions] Get to know your Erlang memory allocation

[Fred Hebert]

On 11/07, Bob Ippolito wrote:
> Did you try using binary:copy/1 at all? Last time I had this issue, that
> function wasn't around, but it seems like it might be useful for your use
> case.
> 

Yeah the problem with it is that our binary messages were often >64b,
and we wanted them in full. If we were to copy these messages, they'd
end up just on the binary heap again.

As far as I can tell, binary:copy/1 is useful whenever the piece of
binary you want to keep around is a subset of the large one you
initially loaded. In our cases, that doesn't happen and there's little
to gain there with that.

> Some of the other strategies we've used to work around these kinds of
> issues are:
> 
> * Use a temporary process for operations that will generate a lot of garbage
> * When that doesn't or can't work, include a manual hibernate or
> erlang:garbage_collect/0
> 

Right. In our case the processes are long-lived -- they're buffers that
have to be alive for a while. It's part of the entire problem.

Manual hibernation is something we've injected all over the place on top
of the last-resort global GC. Things are mostly stable since then, and
on top of the memory allocation rework, we also added a dynamic buffer
resizing based on whether we're able to ever empty the buffers or not to
reduce the amount of very long-lived information stored on the node to
free up as many carriers as possible.

That seemed to work well for now, and we've got rid of most alerts we
could have related to memory to this day.