[erlang-questions] Memory growing

[Alex P]

It seems that I've found the issue.

I have a "process_killer" gen_server where processes can subscribe for 
periodic GC or kill. Its subscribe functions are called on each message 
received by some processes to postpone the GC/kill (something like re-arm).

This process performs an erlang:monitor if not already monitored to catch a 
dead process and remove it from watch list. If I comment our the 
re-subscription line on each handled message, "system" area seems to behave 
normally. That means it is a bug in my process_killer that does leak 
monitor refs (remember you can call erlang:monitor multiple times and each 
call creates a reference).

I was lead to this idea because I've tested a simple module which was 
calling erlang:monitor in a loop and I have seen ~13 bytes "system" area 
grow on each call. 

The workers themselves were OK because they would die anyway taking their 
monitors along with them. There is one long running (starts with the app, 
stops with the app) process that dispatches all the messages to the workers 
that was calling GC re-arm on each received message, so we're talking about 
tens of thousands of monitors spawned per hour and never released.

I'm writing this answer here for future reference.

TL;DR; make sure you are not leaking monitor refs on a long running 
process. 

On Thursday, February 7, 2013 6:21:45 PM UTC+2, Matthew Evans wrote:
>
>
> Can you kill the VM with a SIGUSR1 and look at the resulting crash dump? 
>
> Start an Erlang shell, start crashdump viewer and poke around: 
>
> crashdump_viewer:start(). 
>
> ________________________________ 
> > Date: Thu, 7 Feb 2013 08:08:58 -0800 
> > From: sig...@REDACTED <javascript:> 
> > To: erlang-pr...@REDACTED <javascript:> 
> > CC: erlang-q...@REDACTED <javascript:> 
> > Subject: Re: [erlang-questions] Memory growing 
> > 
> > I've tested NIFs from lists module and they seem to be fine. 
> > 
> > So, coming back to square one, how can I debug what is allocated in the 
> > "system" section of erlang:memory/0 ? 
> > 
> > Alex 
> > 
> > On Saturday, February 2, 2013 10:52:04 PM UTC+2, Alex P wrote: 
> > Hello, 
> > 
> > If somebody could help me with an issue, I would be grateful. I'm 
> > pulling my hair off trying to figure out why an erlang app keeps 
> > growing in memory usage. The erlang:memory() after 3 days shows this: 
> > 
> > [{total,325732292}, 
> > {processes,54520269}, 
> > {processes_used,54492115}, 
> > {system,271212023}, 
> > {atom,932793}, 
> > {atom_used,912937}, 
> > {binary,2348080}, 
> > {code,8288354}, 
> > {ets,487392}, 
> > {maximum,398189416}] 
> > 
> > The interesting part is "system" section. All other data is fine 
> > (processes goes to few MBs during the night as all the started procs 
> > are either hibernated - for long running or they die because of 
> > inactivity). 
> > 
> > There are two issues here: 
> > - "system" keeps growing (around 800MB/week) 
> > - OS (Linux) shows a significantly different amount of memory usage 
> > (almost 2x, 610MB resident in this case) 
> > 
> > The only case where I've managed to make "system" grow is by calling 
> > erlang:monitor() multiple times. It grows with ~13K/call. 
> > 
> > Erlang release used is R15B03. 
> > 
> > The questions are: 
> > - is there any hidden function to inspect all the active process 
> > monitors? (as the single place where I use erlang:monitor looks fine. 
> > The spawned kid does its job and dies, so the ref should be garbage 
> > collected). 
> > - is there any way to inspect "system" section and find out what it 
> > contains? Looking at memory chunks in process map got me nowhere 
> > 
> > Thank you, 
> > Alex 
> > 
> > 
> > 
> > _______________________________________________ erlang-questions 
> > mailing list erlang-q...@REDACTED <javascript:> 
> > http://erlang.org/mailman/listinfo/erlang-questions 
>                                                 
> _______________________________________________ 
> erlang-questions mailing list 
> erlang-q...@REDACTED <javascript:> 
> http://erlang.org/mailman/listinfo/erlang-questions 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130209/c99b1321/attachment.htm>