[erlang-questions] Inter-node communication bottleneck

[Gokhan Boranalp]

Hello,

I think the main problem is docker here. You should test it without docker.
If you still insist to go with docker please check the items below:

- How docker container is configured?
- What kind of network setup you use within docker? Did you use Open
switch etc.?
- Did you do general tweak on machine, before you run the docker
containers? Like incrementing file descriptors, tweaking TCP stack in
sysconfig etc.

See my example sysctl.conf below. Please do not use these values directly :)

#kunthar tuning values

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 16384 16777216
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_congestion_control = cubic

#Disable ipv6 for now
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

SNIP from /etc/init/docker.conf
description "Docker daemon"

start on (local-filesystems and net-device-up IFACE!=lo)
stop on runlevel [!2345]

This part should be added
>>>  limit nofile 524288 1048576
>>> limit nproc 524288 1048576

You also should pass needed params when run docker containers properly.
Please read:
http://docs.docker.com/reference/run/#runtime-constraints-on-cpu-and-memory
http://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration
http://docs.docker.com/reference/run/
http://stackoverflow.com/questions/16084741/how-do-i-set-resources-allocated-to-a-container-using-docker

Also, i strongly advise you to change your base docker image to this:
https://github.com/phusion/baseimage-docker

Good luck.


On Mon, Sep 8, 2014 at 4:25 PM, Jihyun Yu <yjh0502@REDACTED> wrote:
> I didn't tested, but AFAIK {active, once} is useful when process wants
> to handle messages other than TCP packet without blocking.
>
> On Mon, Sep 08, 2014 at 06:14:02AM -0700, Dror Mein wrote:
>> active false worked better than active once?
>>
>>
>> On Monday, September 8, 2014 3:12 PM, Jihyun Yu <yjh0502@REDACTED> wrote:
>>
>>
>>
>> Hi,
>>
>> I couldn't spend much time after last
>>  update. It seems that using
>> multiple TCP sockets is not that scalable as well. I ran a test with
>> following configurations.
>>
>> - 12 cores, 24 threads server (Intel L5640 x 2)
>> - Two Erlang VM on two docker containers
>> - Connects with 12 TCP connections and send/recv packets
>> - [{delay_send,true}, {high_watermark,1024000}]
>>
>> With single integer term, total bandwidth is capped on about ~10MB/s.
>> With single binary term <<"hello">>, total bandwidth is capped on about
>> 20~30MB/s. Two Erlang VM consume almost all CPU resources (2400% total),
>> so bandwidth per CPU usage is much worse than Erlang messaging case. I
>> didn't run perf yet, so I don't know where is an bottleneck.
>>
>> I attached test source code so you can
>>  reproduce the result. Please tell
>> me if there is an error on configurations/test codes/...
>>
>>
>> [1] http://www.erlang.org/doc/man/inet.html
>>
>>
>> On Mon, Sep 08, 2014 at 03:59:04AM -0700, Dror Mein wrote:
>> > I encountered the same problem. any updates or new successes? the binary_to_term over tcp sounds promising.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions



-- 
BR,
\|/ Kunthar