[erlang-questions] Garbage collecting binaries

[Anders Nygren]

On Thu, Oct 7, 2010 at 10:07 AM, tom kelly <ttom.kelly@REDACTED> wrote:
> Hi Nicholas,
>
> This was just run in the erlang shell, A was a 100000 byte binary, "f(A)."
> told the shell to forget it, ie. set the reference count to zero. There were
> no other references to A in this case nor were other binaries created from
> it. It was gc'd after a while, presumably after 65535 reductions of the
> process that stored it (if that makes sense). My question is how can I
> reduce the fullsweep_after or explicitly force a gc on the heap where it is
> stored.
>
> //Tom.
>

Hi
The shell keeps a history also. So even after f(A) the binary is still
stored in the
history.
As an example

7> A=lists:seq(1,10).
[1,2,3,4,5,6,7,8,9,10]
8> f(A).
ok
9> A.
* 1: variable 'A' is unbound
10> v(7).
[1,2,3,4,5,6,7,8,9,10]

The useful shell functions for this is

h()        -- history
history(N) -- set how many previous commands to keep
results(N) -- set how many previous command results to keep

/Anders

>
>
> On Thu, Oct 7, 2010 at 3:57 PM, Nicholas Frechette <zeno490@REDACTED>wrote:
>
>> A bit hard to understand what is happening without seeing f(A)'s code.
>> AFAIK, even if you don't keep a reference to a binary, if you keep a
>> reference to a binary created from it, it'll keep it in full in memory.
>> ie:
>> A = << some large binary>>.
>> <<B:32/binary, _/binary>> = A.
>> If you keep no references to A but you do to B, A will remain in memory
>> because B will have been created as a pointer offset from A (as an
>> optimization).
>>
>> Nicholas
>>
>>
>> On Thu, Oct 7, 2010 at 10:07 AM, tom kelly <ttom.kelly@REDACTED> wrote:
>>
>>> Hello List,
>>>
>>> I need some help on garbage collecting binaries.
>>> I have an application that handles large binaries and under load it eats
>>> up
>>> all the available memory then falls over, even when I start all the
>>> data-handling processes with "{spawn_opt,[{fullsweep_after, 20}]}".
>>>
>>> Reading point 5.15 on the page:
>>> http://www.erlang.org/faq/how_do_i.html
>>> leads me to think that calling the garbage collector on the process that
>>> created a binary will clean it up but my shell experiment below shows me
>>> I'm
>>> wrong. It's now 10 minutes since I've done this experiment and the large
>>> binary is still in memory.
>>>
>>> Maybe I have to call the garbage collector of the process whose heap is
>>> storing the binary? If so, which process is it?
>>>
>>> Any pointers as to what I'm doing wrong or mis-understand will be greatly
>>> appreciated.
>>>
>>> I'm on a legacy R12B5 system.
>>>
>>> //Tom.
>>>
>>>
>>>
>>> 18> process_info(self(),total_heap_size).
>>> {total_heap_size,4181}
>>>
>>> 19> A = L(100000).
>>> [140,161,41,128,44,96,215,43,15,164,88,107,1,167,4,125,118,
>>>  180,121,181,160,124,244,140,169,215,31,82,43|...]
>>>
>>> 20> process_info(self(),total_heap_size).
>>> {total_heap_size,1149851}
>>>
>>> 21> f(A).
>>> ok
>>>
>>> 22> erlang:garbage_collect().
>>> true
>>>
>>> 23> process_info(self(),total_heap_size).
>>> {total_heap_size,3194}
>>>
>>>
>>> 24> memory(binary).
>>> 6536
>>>
>>> 25> A = B(100000).
>>> <<232,241,55,171,218,35,86,122,211,185,232,1,203,249,181,
>>>  218,176,33,88,131,102,56,102,82,158,114,200,174,253,...>>
>>>
>>> 26> memory(binary).
>>> 106704
>>>
>>> 27> f(A).
>>> ok
>>>
>>> 28> memory(binary).
>>> 106704
>>>
>>> 29> erlang:garbage_collect().
>>> true
>>>
>>> 30> memory(binary).
>>> 106560
>>>
>>
>>
>