[erlang-questions] Memory allocation HOWTO from Fred

[Fred Hebert]

Hi Max,

Thanks for the good comments. Have you found any pattern that helped in
your case? I remember seeing Lukas recommending a few tweaks here and
there and wondered if you ever found suitable settings.

I'm guessing just the churn is gonna hurt a lot in your case -- would it
be worth having a middle-man process that does blocking receives with a
bound size and messages them to your main process?

    [ recv(Sock, 50000) ] --> Bin --> [ gen_server ]

It sounds like having ~100 of these messaging refc bins would be a huge
gain overall given the alternatives and the low cost of message passing
in that case.

Regards,
Fred.

On 11/08, Max Lapshin wrote:
> https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole
> 
> Fred, at first: thank you for this tutorial.
> 
> It is not just a blogpost, it looks more like a paper into journal =)
> 
> 
> 
> I've experiencing similar problems in erlyvideo, because in some places
> reallocating memory on gigabits per second speed creates lots of binaries.
> 
> For example: if I read from tcp socket with active,once mode, I get 1400
> byte blocks, which are big enough to get into big heap, but too small to
> handle them and it ends in reallocation and copying hundreds times per
> second.
> 
> There is no  {active, 100K} mode that will send messages of 100K bytes, so
> the only way out is something like:
> 
> handle_info({tcp, Sock, Bin}, State) ->
>   {ok, Bin2} = gen_tcp:recv(Sock, 50000),
>   handle(<<Bin/binary, Bin2/binary>>)
> 
> Without such primitive patch I get about 500 new binaries per second and
> CPU is melting out.
> I don't know why, but it is clear: when there are 100 processes in system
> that creates about 500 small binaries per second, beam.smp is consuming lot
> of CPU power.

> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions