[erlang-questions] Erlang node crashes due to message queue despite the large amount of free memory.

[Saltanov, Alexey]

I have simple Erlang echo server:
loop() ->
    receive
        {From, Request} ->
            erlang:send(From, Request),
            loop()
    end.
This echo server started on the Server1.

And I have some client application (Basho Bench test):
run(send, _KeyGen, ValueGen, State) ->
    Msg = ValueGen(),
    erlang:send(State#state.address, {{State#state.reg_name, node()}, {self(), Msg}}),
    {ok, State}.
Wihich sends many messages to the echo server.

When I started client application on the Server2 I've found the crash of the echo server.
=erl_crash_dump:0.2
Wed Jan 15 17:04:26 2014
Slogan: eheap_alloc: Cannot allocate 949152844 bytes of memory (of type "heap").
System version: Erlang R16B02 (erts-5.10.3) [source] [64-bit halfword] [smp:8:8] [async-threads:10] [kernel-poll:false]
Compiled: Wed Dec 11 16:19:29 2013
Taints: crypto
Atoms: 16134
=memory
total: 4131605472
processes: 2386433264
processes_used: 2386409264
system: 1745172208
atom: 463441
atom_used: 446760
binary: 501624
code: 11870105
ets: 274696
low: 3429068696

But the Server1 had 30 Gb free RAM and 16 Gb free swap space before I started the echo server,
and I saw with the "top" command that my echo server allocated only 5 Gb RAM and really used only about 2 Gb.
Why the message queue cannot use more server memory?

Some information about the Server1 configuration.
$ uname -a
Linux 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.3 (Santiago)

$ top
top - 12:27:01 up 2 days, 22:14,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 310 total,   1 running, 309 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  33011268k total,   868152k used, 32143116k free,   132708k buffers
Swap: 16777208k total,        0k used, 16777208k free,   172772k cached

I also tested memory allocation for the big binary data on the same Server1 node:
Erlang R16B02 (erts-5.10.3) [source] [64-bit halfword] [smp:8:8] [async-threads:10] [kernel-poll:false]
Eshell V5.10.3  (abort with ^G)
1> Z1=fun(N)-> X=lists:map(fun(I)-> binary:copy(<<1>>,1073741824) end, lists:seq(1,N)), io:format("mem: ~p~n", [memory()]), length(X) end.
#Fun<erl_eval.6.80484245>
2> Z1(10).
mem: [{total,10758494488},
      {processes,4659176},
      {processes_used,4659176},
      {system,10753835312},
      {atom,194289},
      {atom_used,174684},
      {binary,10737518424},
      {code,4012558},
      {ets,172744},
      {low,4057440}]
10
3>Z2=fun(N)-> X=lists:map(fun(I)-> binary:copy(<<1>>,1024) end, lists:seq(1,N)), io:format("mem: ~p~n", [memory()]), length(X) end.
#Fun<erl_eval.6.80484245>
4>Z2(10485760).
4> Z2(10485760).
mem: [{total,11973197304},
      {processes,795606312},
      {processes_used,795606312},
      {system,11177590992},
      {atom,194289},
      {atom_used,174744},
      {binary,11161273768},
      {code,4012558},
      {ets,172744},
      {low,4057440}]
10485760

And this test completed successfully. But on the same server my echo server crashes when it is used only 4 Gb of the memory.
Has the messages queue its own limit?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140117/3d868da2/attachment.htm>