Garbage collection of binaries

[Sean Hinde]

On Tuesday, October 14, 2003, at 03:59  pm, Jesper Wilhelmsson wrote:

> On Mon, 13 Oct 2003, James Hague wrote:
>
>> If I write a tail recursive function that loads a large file as a
>> binary--say, 10MB--on each iteration, then the emulator quickly runs 
>> out of
>> memory and dies.  This is apparently because binaries are only garbage
>> collected when the regular garbage collector runs, and with large 
>> binaries
>> it's easy to run out of memory before then.  It's easy to get this to 
>> happen
>> in a real application.
>>
>> One hacky solution is to call erlang:garbage_collect all over the 
>> place,
>> but...yuck.  Is this being addressed in the run-time system at some 
>> point?
>> Any other workarounds?
>
> Binaries are referenced counted and are removed when their conter reach
> zero. This can happen in the garbage collector as you say, but it can 
> also
> happen when the process owning it dies.
> So, I guess a workaround could be to let a new process handle the 
> binary
> in each iteration, and let it die afterwards.
>
> I would of course be possible (and a bit more efficient) to add a BIF 
> that
> removes the binary explicitly, but from what I have seen in this list,
> introducing new functionality (and in particular one that destructively
> removes things from memory) is not that popular right now ;-)
>


Is there not some mechanism which could be introduced to specifically 
check for processes growing in this way - I don't know, perhaps every 
time a very large binary is assigned on the heap check to see if it 
might be a good time do do a "survival" GC? Maybe a running total could 
be kept of binaries assigned over a certain size? It would seem to have 
relatively little overhead compared to the actual operation of making a 
very large binary, and it would protect against this case quite a bit 
of the time.

Sean