[erlang-questions] investigating memory issues

[dmkolesnikov@REDACTED]

Hello,

Some of library or your code leaks binary.
This usually happened went large binary blob is not de-referenced during processing. GC would not help here at all. You need to find and fix code that does it. 

Dmitry 

Sent from my iPhone

> On 16 Oct 2015, at 22:34, Caragea Silviu <silviu.cpp@REDACTED> wrote:
> 
> Hello,
> 
> I have an erlang application which is reading jobs from a gearman queue and call some HTTP url's (send's notifications over http).
> Beside this also saves some states in a redis/mysql database. 
> Everything performs very nice excepting the fact that memory is increasing like crazy and get never released.
> 
> The project is using the following deps:
> 
> 1. Lager
> 2. hackeny - for the HTTP requests
> 3. econfig
> 4. folsom 
> 5. emysql
> 6. mero
> 7. jsonx
> 8. tempo
> 9. erlang_gearman
> 
> When I start the app it's starting with several hundred of MB but in 20 days is going to over 1.5 GB and still increasing if I don't restart the process. 
> 
> recon_alloc:memory: 
>     allocated -> shows at this moment 1454477752
>     used -> 1351474280
> 
> From what I see 95 % of memory is on the binary_alloc. Fragmentation for this allocator is looking like:
> 
> recon_alloc:fragmentation(current).
> 
> [{{binary_alloc,1},
>   [{sbcs_usage,1.0},
>    {mbcs_usage,0.9404540505595212},
>    {sbcs_block_size,0},
>    {sbcs_carriers_size,0},
>    {mbcs_block_size,691837288},
>    {mbcs_carriers_size,735641776}]},
>  {{binary_alloc,2},
>   [{sbcs_usage,1.0},
>    {mbcs_usage,0.9441229925677996},
>    {sbcs_block_size,0},
>    {sbcs_carriers_size,0},
>    {mbcs_block_size,567818272},
>    {mbcs_carriers_size,601424048}]},
> 
> I also tried to run the GC over all processes but nothing changing :(
> 
> erlang:memory(total). => 1350813872
> 
> [erlang:garbage_collect(Pid) || Pid <- processes()].
> 
> erlang:memory(total). => 1347613584
> 
> I have no idea on how to identify where all this memory goes.. The sum of all processes as displayed by Observer app in the Process tab is insignificant.
> 
> Any advice ? 
> 
> Silviu
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions