[erlang-questions] Memory usage

[Ulf Wiger]

It's late, and since you're still up and about (and the ERTS experts perhaps
aren't), I'll take a wild stab and suggest that it has something to do with
generational GC etc. The process_info(P,heap_size) only looks at the
youngest heap, so it misses all data on the old heap.

Perhaps you can play around a bit with GC parameter tuning if
you're planning to stay up all night?  ;-)

BR,
Ulf W

2008/9/9 Francesca Gangemi <francesca@REDACTED>:
> Hi Ulf,
>
> thanks for your suggestion, we already knew about this article but it didn't
> help.
> In our case the process parsing the xml does not use a huge amount of memory
> and we don't have so many requests at the same time.
> Furthermore I can't see which process is using all the memory.
> The memory information says that processes_used memory is 2060858920
> (~1965MB) while the process information reports a total of  24318346 bytes
> (~231 MB), adding up the stack+heap values for all the processes.
>
> What is included in the calculation of the processes_used value?
>
> Kind regards
> Francesca
>
> Ulf Wiger wrote:
>>
>> Hi Francesca,
>>
>> It sounds a bit similar to this:
>>
>> http://www.erlangatwork.com/2008/07/hunting-bugs.html
>>
>> In that case, xmerl was used, and was happily gobbling up memory
>> (now, that was a 64-bit system, which of course aggravated the problem.
>> OTOH the XML bits were fairly small.)
>>
>> BR,
>> Ulf W
>>
>> 2008/9/9 Francesca Gangemi <francesca@REDACTED>:
>>
>>>
>>> Hi,
>>>
>>> we notice that nodes crash during a load test because they cannot
>>> allocate more memory.
>>> The slogan from the crash dump says
>>> "eheap_alloc: Cannot allocate 3328160 bytes of memory (of type
>>> old_heap")."
>>>
>>> Memory information reports the following values
>>> total              2468460696
>>> processes     2060871072
>>> processes_used     2060858920
>>> system              407589624
>>>
>>> If then I look at the process information and I add up the Stack+heap
>>> values for all the processes I get 243183460 bytes. This number is quite
>>> less than the processes_used one.
>>> I wonder if something else is added up to the process_used memory value.
>>>
>>> We run the test using R11B-3 and R12B-4.
>>> There are 6000 processes running under a supervisor. They generate xml
>>> requests and parse xml responses. The XML structure is quite complex and
>>> its size is up to 80KB.
>>>
>>> The following lines report information about allocated areas
>>>
>>> sys_misc     3422757
>>> static     14417920
>>> atom_space     163860    162413
>>> atom_table     91901
>>> module_table     5768
>>> export_table     48008
>>> register_table     852
>>> fun_table     3250
>>> module_refs     4096
>>> loaded_code     4553244
>>> dist_table     555
>>> node_table     299
>>> bits_bufs_size     0
>>> bif_timer     40604
>>> link_lh     0
>>> proc     3995708    3988028
>>> atom_entry     316564    316276
>>> export_entry     312364    312172
>>> module_entry     12088    11888
>>> reg_proc     6320    6104
>>> monitor_sh     253288    252248
>>> nlink_sh     362060    358844
>>> proc_list     28    28
>>> fun_entry     60724    60220
>>> db_tab     12848    10416
>>> driver_event_data_state     28    28
>>> driver_select_data_state     1000    328
>>>
>>>
>>> Kind regards
>>> Francesca
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://www.erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>
>