[erlang-questions] Out of memory troubles.

[Alex Shneyderman]

Hi, all!

I have an out of memory exception I do not understand.

I posted yesterday about not being able to open more than certain
number of persistent connections towards my server. That turned out to
be due to ulimit not being high enough on my mac's shell. I increased
that to 8192 and things started to work better. I was able to open 1K
connections towards the server with no problems and memory footprint
of the OS process that keeps those 1K connections while exchanging
data between the server and  the processes on erlang VM is about
150MB. The next step was to increase the number of persistent
connections from 1K to 2K.

The number of connections where not allowed to go further than 1024
that as I understood was due to the way Erlang was compiled on Mac. I
searched on this list and found a suggestion by Joel Raymont on how to
up that number by increasing __DARWIN_FD_SETSIZE
(http://erlang.org/pipermail/erlang-questions/2011-December/063119.html).
Then I recompiled erlang and ran the same application trying to
request 2K connections and here is where I get out of memory error.
Here is what I see in the shell:

09:08:42.229 [error] erts_poll_wait() failed: einval (22)


beam.smp(71174,0xb032e000) malloc: *** mmap(size=730857472) failed
(error code=12)

           *** error: can't allocate region

                                           *** set a breakpoint in
malloc_error_break to debug


crash reports are basically identical to what I see on the console
except for the malloc business.

>From the footprint that this simulation leaves on my server, I can see
that there were 1008 successful connections. I was trying to monitor
memory consumption of the erlang process and it jumps very high very
fast after around 1K connections. It goes from 150M to about 1.5GB
really fast and then VM crashes.

As I mentioned 1K connections only has about 150M of memory footprint
on my system and this laptop has about 1.5G of spare memory at the
time I start the process - so, I'd expect 8-10K connections possible.
I can certainly run several VMs with 1K connections per VM - as the
matter of fact that is what I do now to avoid out of memory exceptions
and crashes - but I would like to be able to have more agents
(connections) on the same VM which simplifies deployment for me.

Any ideas on what might I be doing wrong?

Thanks,
Alex.