[erlang-questions] Erlang node crashes due to message queue despite the large amount of free memory.

[Valentin Micic]

Have you considered switching from 32-bit erlang to 64-bit?

V/

On 17 Jan 2014, at 12:37 PM, Saltanov, Alexey wrote:

> I have simple Erlang echo server:
> loop() ->
>     receive
>         {From, Request} ->
>             erlang:send(From, Request),
>             loop()
>     end.
> This echo server started on the Server1.
>  
> And I have some client application (Basho Bench test):
> run(send, _KeyGen, ValueGen, State) ->
>     Msg = ValueGen(),
>     erlang:send(State#state.address, {{State#state.reg_name, node()}, {self(), Msg}}),
>     {ok, State}.
> Wihich sends many messages to the echo server.
>  
> When I started client application on the Server2 I've found the crash of the echo server.
> =erl_crash_dump:0.2
> Wed Jan 15 17:04:26 2014
> Slogan: eheap_alloc: Cannot allocate 949152844 bytes of memory (of type "heap").
> System version: Erlang R16B02 (erts-5.10.3) [source] [64-bit halfword] [smp:8:8] [async-threads:10] [kernel-poll:false]
> Compiled: Wed Dec 11 16:19:29 2013
> Taints: crypto
> Atoms: 16134
> =memory
> total: 4131605472
> processes: 2386433264
> processes_used: 2386409264
> system: 1745172208
> atom: 463441
> atom_used: 446760
> binary: 501624
> code: 11870105
> ets: 274696
> low: 3429068696
>  
> But the Server1 had 30 Gb free RAM and 16 Gb free swap space before I started the echo server,
> and I saw with the "top" command that my echo server allocated only 5 Gb RAM and really used only about 2 Gb.
> Why the message queue cannot use more server memory?
>  
> Some information about the Server1 configuration.
> $ uname -a
> Linux 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
>  
> $ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 6.3 (Santiago)
>  
> $ top
> top - 12:27:01 up 2 days, 22:14,  1 user,  load average: 0.00, 0.00, 0.00
> Tasks: 310 total,   1 running, 309 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem:  33011268k total,   868152k used, 32143116k free,   132708k buffers
> Swap: 16777208k total,        0k used, 16777208k free,   172772k cached
>  
> I also tested memory allocation for the big binary data on the same Server1 node:
> Erlang R16B02 (erts-5.10.3) [source] [64-bit halfword] [smp:8:8] [async-threads:10] [kernel-poll:false]
> Eshell V5.10.3  (abort with ^G)
> 1> Z1=fun(N)-> X=lists:map(fun(I)-> binary:copy(<<1>>,1073741824) end, lists:seq(1,N)), io:format("mem: ~p~n", [memory()]), length(X) end.
> #Fun<erl_eval.6.80484245>
> 2> Z1(10).
> mem: [{total,10758494488},
>       {processes,4659176},
>       {processes_used,4659176},
>       {system,10753835312},
>       {atom,194289},
>       {atom_used,174684},
>       {binary,10737518424},
>       {code,4012558},
>       {ets,172744},
>       {low,4057440}]
> 10
> 3>Z2=fun(N)-> X=lists:map(fun(I)-> binary:copy(<<1>>,1024) end, lists:seq(1,N)), io:format("mem: ~p~n", [memory()]), length(X) end.
> #Fun<erl_eval.6.80484245>
> 4>Z2(10485760).
> 4> Z2(10485760).
> mem: [{total,11973197304},
>       {processes,795606312},
>       {processes_used,795606312},
>       {system,11177590992},
>       {atom,194289},
>       {atom_used,174744},
>       {binary,11161273768},
>       {code,4012558},
>       {ets,172744},
>       {low,4057440}]
> 10485760
>  
> And this test completed successfully. But on the same server my echo server crashes when it is used only 4 Gb of the memory.
> Has the messages queue its own limit?
>  
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140117/f0ec9d30/attachment.htm>