[erlang-questions] investigating memory issues

[Doug Rohrer]

What version of Erlang are you using?  One thing that comes to mind is a console handling binary memory leak in 18 before 18.03, which can cause problems if you're using lager's console handler.

Updating to 18.03 or greater (18.1) should resolve the issue.

Doug Rohrer

> On Oct 16, 2015, at 3:34 PM, Caragea Silviu <silviu.cpp@REDACTED> wrote:
> 
> Hello,
> 
> I have an erlang application which is reading jobs from a gearman queue and call some HTTP url's (send's notifications over http).
> Beside this also saves some states in a redis/mysql database. 
> Everything performs very nice excepting the fact that memory is increasing like crazy and get never released.
> 
> The project is using the following deps:
> 
> 1. Lager
> 2. hackeny - for the HTTP requests
> 3. econfig
> 4. folsom 
> 5. emysql
> 6. mero
> 7. jsonx
> 8. tempo
> 9. erlang_gearman
> 
> When I start the app it's starting with several hundred of MB but in 20 days is going to over 1.5 GB and still increasing if I don't restart the process. 
> 
> recon_alloc:memory: 
>     allocated -> shows at this moment 1454477752
>     used -> 1351474280
> 
> From what I see 95 % of memory is on the binary_alloc. Fragmentation for this allocator is looking like:
> 
> recon_alloc:fragmentation(current).
> 
> [{{binary_alloc,1},
>   [{sbcs_usage,1.0},
>    {mbcs_usage,0.9404540505595212},
>    {sbcs_block_size,0},
>    {sbcs_carriers_size,0},
>    {mbcs_block_size,691837288},
>    {mbcs_carriers_size,735641776}]},
>  {{binary_alloc,2},
>   [{sbcs_usage,1.0},
>    {mbcs_usage,0.9441229925677996},
>    {sbcs_block_size,0},
>    {sbcs_carriers_size,0},
>    {mbcs_block_size,567818272},
>    {mbcs_carriers_size,601424048}]},
> 
> I also tried to run the GC over all processes but nothing changing :(
> 
> erlang:memory(total). => 1350813872
> 
> [erlang:garbage_collect(Pid) || Pid <- processes()].
> 
> erlang:memory(total). => 1347613584
> 
> I have no idea on how to identify where all this memory goes.. The sum of all processes as displayed by Observer app in the Process tab is insignificant.
> 
> Any advice ? 
> 
> Silviu
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions