[erlang-questions] Out of memory troubles.

[Gordon Guthrie]

Alex

I had a similar problem on Linux - not sure how relevant is to a Mac
but I thought I would mention it.

On Linux you can increase the limits but only subject to the
system-level restrictions placed on your username by a file called
/etc/security/limits.conf

To get my open-file limit up on Linux I had to bump that up as well..

Mebbies there is something like that on OSX as well?

Gordon

On 11 June 2012 08:30, Alex Shneyderman <a.shneyderman@REDACTED> wrote:
> Hi, all!
>
> I have an out of memory exception I do not understand.
>
> I posted yesterday about not being able to open more than certain
> number of persistent connections towards my server. That turned out to
> be due to ulimit not being high enough on my mac's shell. I increased
> that to 8192 and things started to work better. I was able to open 1K
> connections towards the server with no problems and memory footprint
> of the OS process that keeps those 1K connections while exchanging
> data between the server and  the processes on erlang VM is about
> 150MB. The next step was to increase the number of persistent
> connections from 1K to 2K.
>
> The number of connections where not allowed to go further than 1024
> that as I understood was due to the way Erlang was compiled on Mac. I
> searched on this list and found a suggestion by Joel Raymont on how to
> up that number by increasing __DARWIN_FD_SETSIZE
> (http://erlang.org/pipermail/erlang-questions/2011-December/063119.html).
> Then I recompiled erlang and ran the same application trying to
> request 2K connections and here is where I get out of memory error.
> Here is what I see in the shell:
>
> 09:08:42.229 [error] erts_poll_wait() failed: einval (22)
>
>
> beam.smp(71174,0xb032e000) malloc: *** mmap(size=730857472) failed
> (error code=12)
>
>           *** error: can't allocate region
>
>                                           *** set a breakpoint in
> malloc_error_break to debug
>
>
> crash reports are basically identical to what I see on the console
> except for the malloc business.
>
> From the footprint that this simulation leaves on my server, I can see
> that there were 1008 successful connections. I was trying to monitor
> memory consumption of the erlang process and it jumps very high very
> fast after around 1K connections. It goes from 150M to about 1.5GB
> really fast and then VM crashes.
>
> As I mentioned 1K connections only has about 150M of memory footprint
> on my system and this laptop has about 1.5G of spare memory at the
> time I start the process - so, I'd expect 8-10K connections possible.
> I can certainly run several VMs with 1K connections per VM - as the
> matter of fact that is what I do now to avoid out of memory exceptions
> and crashes - but I would like to be able to have more agents
> (connections) on the same VM which simplifies deployment for me.
>
> Any ideas on what might I be doing wrong?
>
> Thanks,
> Alex.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions