[erlang-questions] investigating memory issues

[Caragea Silviu]

Hello,

I have an erlang application which is reading jobs from a gearman queue and
call some HTTP url's (send's notifications over http).
Beside this also saves some states in a redis/mysql database.
Everything performs very nice excepting the fact that memory is increasing
like crazy and get never released.

The project is using the following deps:

1. Lager
2. hackeny - for the HTTP requests
3. econfig
4. folsom
5. emysql
6. mero
7. jsonx
8. tempo
9. erlang_gearman

When I start the app it's starting with several hundred of MB but in 20
days is going to over 1.5 GB and still increasing if I don't restart the
process.

recon_alloc:memory:
    allocated -> shows at this moment 1454477752
    used -> 1351474280

>From what I see 95 % of memory is on the binary_alloc. Fragmentation for
this allocator is looking like:

recon_alloc:fragmentation(current).

[{{binary_alloc,1},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.9404540505595212},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,691837288},
   {mbcs_carriers_size,735641776}]},
 {{binary_alloc,2},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.9441229925677996},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,567818272},
   {mbcs_carriers_size,601424048}]},

I also tried to run the GC over all processes but nothing changing :(

erlang:memory(total). => 1350813872

[erlang:garbage_collect(Pid) || Pid <- processes()].

erlang:memory(total). => 1347613584

I have no idea on how to identify where all this memory goes.. The sum of
all processes as displayed by Observer app in the Process tab is
insignificant.

Any advice ?

Silviu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20151016/c6f10c9d/attachment.htm>