[erlang-questions] Erlang node crashes due to message queue despite the large amount of free memory.

[Lukas Larsson]

Hello,

As Xiao Jia pointed out, the problem is that you are running the halfword
emulator. In the halfword emulator a tradeoff is made where all term
pointers are shrunk to 32 bit and thus the memory footprint shrinks,
however as a consequence of this the total memory of all process heaps
cannot be greater than 4GB. It is this limit that you are hitting.

The halfword emulator is mainly useful when you have a lot of data in ets
tables, and not so much in the process heap.

Lukas


On Fri, Jan 17, 2014 at 11:37 AM, Saltanov, Alexey <
Alexey.Saltanov@REDACTED> wrote:

>  I have simple Erlang echo server:
> loop() ->
>     receive
>         {From, Request} ->
>             erlang:send(From, Request),
>             loop()
>     end.
> This echo server started on the Server1.
>
> And I have some client application (Basho Bench test):
> run(send, _KeyGen, ValueGen, State) ->
>     Msg = ValueGen(),
>     erlang:send(State#state.address, {{State#state.reg_name, node()},
> {self(), Msg}}),
>     {ok, State}.
> Wihich sends many messages to the echo server.
>
> When I started client application on the Server2 I've found the crash of
> the echo server.
> =erl_crash_dump:0.2
> Wed Jan 15 17:04:26 2014
> Slogan: eheap_alloc: Cannot allocate 949152844 bytes of memory (of type
> "heap").
> System version: Erlang R16B02 (erts-5.10.3) [source] [64-bit halfword]
> [smp:8:8] [async-threads:10] [kernel-poll:false]
> Compiled: Wed Dec 11 16:19:29 2013
> Taints: crypto
> Atoms: 16134
> =memory
> total: 4131605472
> processes: 2386433264
> processes_used: 2386409264
> system: 1745172208
> atom: 463441
> atom_used: 446760
> binary: 501624
> code: 11870105
> ets: 274696
> low: 3429068696
>
> But the Server1 had 30 Gb free RAM and 16 Gb free swap space before I
> started the echo server,
> and I saw with the "top" command that my echo server allocated only 5 Gb
> RAM and really used only about 2 Gb.
> Why the message queue cannot use more server memory?
>
> Some information about the Server1 configuration.
> $ uname -a
> Linux 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64
> x86_64 x86_64 GNU/Linux
>
> $ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 6.3 (Santiago)
>
> $ top
> top - 12:27:01 up 2 days, 22:14,  1 user,  load average: 0.00, 0.00, 0.00
> Tasks: 310 total,   1 running, 309 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Mem:  33011268k total,   868152k used, 32143116k free,   132708k buffers
> Swap: 16777208k total,        0k used, 16777208k free,   172772k cached
>
> I also tested memory allocation for the big binary data on the same
> Server1 node:
> Erlang R16B02 (erts-5.10.3) [source] [64-bit halfword] [smp:8:8]
> [async-threads:10] [kernel-poll:false]
> Eshell V5.10.3  (abort with ^G)
> 1> Z1=fun(N)-> X=lists:map(fun(I)-> binary:copy(<<1>>,1073741824) end,
> lists:seq(1,N)), io:format("mem: ~p~n", [memory()]), length(X) end.
> #Fun<erl_eval.6.80484245>
> 2> Z1(10).
> mem: [{total,10758494488},
>       {processes,4659176},
>       {processes_used,4659176},
>       {system,10753835312},
>       {atom,194289},
>       {atom_used,174684},
>       {binary,10737518424},
>       {code,4012558},
>       {ets,172744},
>       {low,4057440}]
> 10
> 3>Z2=fun(N)-> X=lists:map(fun(I)-> binary:copy(<<1>>,1024) end,
> lists:seq(1,N)), io:format("mem: ~p~n", [memory()]), length(X) end.
> #Fun<erl_eval.6.80484245>
> 4>Z2(10485760).
> 4> Z2(10485760).
> mem: [{total,11973197304},
>       {processes,795606312},
>       {processes_used,795606312},
>       {system,11177590992},
>       {atom,194289},
>       {atom_used,174744},
>       {binary,11161273768},
>       {code,4012558},
>       {ets,172744},
>       {low,4057440}]
> 10485760
>
> And this test completed successfully. But on the same server my echo
> server crashes when it is used only 4 Gb of the memory.
> Has the messages queue its own limit?
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140117/212f54d7/attachment.htm>