[erlang-questions] Erlang node crashes due to message queue despite the large amount of free memory.

[Saltanov, Alexey]

Thank you!
I will try to compile Erlang without halfword and try the my test again.
But I have one more question. How could I allocate 10 Gb and 20 Gb memory for binary data on the same node?

From: garazdawi@REDACTED [mailto:garazdawi@REDACTED] On Behalf Of Lukas Larsson
Sent: Friday, January 17, 2014 5:02 PM
To: Saltanov, Alexey
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] Erlang node crashes due to message queue despite the large amount of free memory.

Hello,

As Xiao Jia pointed out, the problem is that you are running the halfword emulator. In the halfword emulator a tradeoff is made where all term pointers are shrunk to 32 bit and thus the memory footprint shrinks, however as a consequence of this the total memory of all process heaps cannot be greater than 4GB. It is this limit that you are hitting.

The halfword emulator is mainly useful when you have a lot of data in ets tables, and not so much in the process heap.

Lukas

On Fri, Jan 17, 2014 at 11:37 AM, Saltanov, Alexey <Alexey.Saltanov@REDACTED<mailto:Alexey.Saltanov@REDACTED>> wrote:
I have simple Erlang echo server:
loop() ->
    receive
        {From, Request} ->
            erlang:send(From, Request),
            loop()
    end.
This echo server started on the Server1.

And I have some client application (Basho Bench test):
run(send, _KeyGen, ValueGen, State) ->
    Msg = ValueGen(),
    erlang:send(State#state.address, {{State#state.reg_name, node()}, {self(), Msg}}),
    {ok, State}.
Wihich sends many messages to the echo server.

When I started client application on the Server2 I've found the crash of the echo server.
=erl_crash_dump:0.2
Wed Jan 15 17:04:26 2014
Slogan: eheap_alloc: Cannot allocate 949152844 bytes of memory (of type "heap").
System version: Erlang R16B02 (erts-5.10.3) [source] [64-bit halfword] [smp:8:8] [async-threads:10] [kernel-poll:false]
Compiled: Wed Dec 11 16:19:29 2013
Taints: crypto
Atoms: 16134
=memory
total: 4131605472
processes: 2386433264
processes_used: 2386409264
system: 1745172208
atom: 463441
atom_used: 446760
binary: 501624
code: 11870105
ets: 274696
low: 3429068696

But the Server1 had 30 Gb free RAM and 16 Gb free swap space before I started the echo server,
and I saw with the "top" command that my echo server allocated only 5 Gb RAM and really used only about 2 Gb.
Why the message queue cannot use more server memory?

Some information about the Server1 configuration.
$ uname -a
Linux 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.3 (Santiago)

$ top
top - 12:27:01 up 2 days, 22:14,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 310 total,   1 running, 309 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  33011268k total,   868152k used, 32143116k free,   132708k buffers
Swap: 16777208k total,        0k used, 16777208k free,   172772k cached

I also tested memory allocation for the big binary data on the same Server1 node:
Erlang R16B02 (erts-5.10.3) [source] [64-bit halfword] [smp:8:8] [async-threads:10] [kernel-poll:false]
Eshell V5.10.3  (abort with ^G)
1> Z1=fun(N)-> X=lists:map(fun(I)-> binary:copy(<<1>>,1073741824) end, lists:seq(1,N)), io:format("mem: ~p~n", [memory()]), length(X) end.
#Fun<erl_eval.6.80484245>
2> Z1(10).
mem: [{total,10758494488},
      {processes,4659176},
      {processes_used,4659176},
      {system,10753835312},
      {atom,194289},
      {atom_used,174684},
      {binary,10737518424},
      {code,4012558},
      {ets,172744},
      {low,4057440}]
10
3>Z2=fun(N)-> X=lists:map(fun(I)-> binary:copy(<<1>>,1024) end, lists:seq(1,N)), io:format("mem: ~p~n", [memory()]), length(X) end.
#Fun<erl_eval.6.80484245>
4>Z2(10485760).
4> Z2(10485760).
mem: [{total,11973197304},
      {processes,795606312},
      {processes_used,795606312},
      {system,11177590992},
      {atom,194289},
      {atom_used,174744},
      {binary,11161273768},
      {code,4012558},
      {ets,172744},
      {low,4057440}]
10485760

And this test completed successfully. But on the same server my echo server crashes when it is used only 4 Gb of the memory.
Has the messages queue its own limit?


_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED<mailto:erlang-questions@REDACTED>
http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140117/2ed35421/attachment.htm>