[erlang-questions] Amazon AWS benchmarking of erlang-diameter not upto the mark

[Arshad Ansari]

Hello Dmitry,

On my localbox I had a simple lan configuration with a single
gateway/switch to internet, so I'm not sure what else there is to mention
about the lan config. However; the machine configurations are as below:
1. Server-1 [4 core 4 GB]
2. Server-2 [4 core 4 GB]
3. Relay [Works as a load Balancer] [4 core 4 GB]
4. Client 1 [4 core 4 GB]
5. Client 2 [4 core 4 GB]

Each machine runs a single node as mentioned. There is a single tcp socket
connection between peers where alll Clients connect to Relay and Relay in
turn connects with the 2 servers and acts a proxy while also balancing the
request between the two servers using round robin algorithm. This
connection is used to send all the request using the gen_server:call
(blocking) method. Following is the erl vm params I'm using for all of the
nodes:

erl +A 1024 +P 134217727 +Q 134217727 -env ERL_MAX_PORTS 134217727 +K true

I'm able to get around 26K req/sec on relay which divides the requests
between two servers around 13K req/sec. Each and every node was utilizing
near about *99%* of all the cores available on the system. This I was able
to get on my local network.

As for AWS, I had the following setup:

1. Server - 1 [4 core 16GB]
2. Server - 2 [4 core 16GB]
3. Relay [16 core 30GB]
4. Client [16 core 30GB]

I had launched four of them as part of single placement group to remove
bandwidth restriction. When I did run the nodes, the processors were *not*
being utilized more than *60% *and yet the relay was getting around 12K
req/sec which it was dividing between servers with each one getting around
6K req/sec. Erlang VM args were the same as mentioned above. I was using
internal private IPs on AWS to connect the peers with one another to avoid
bandwidth limitation that comes with public IP usage. Each of the aws
instance has EBS volume attached to it. The only disk access I'm doing is
to write log at the per second rate, besides accessing the ETS table for
getting counter instances from each instance.

I hope I've covered all there is to the configurations and also that I was
clear enough.

Thanks again,
Regards,
Arshad

On Tue, Aug 30, 2016 at 12:34 AM Dmitry Kolesnikov <dmkolesnikov@REDACTED>
wrote:

> Hello,
>
> You’ve experienced difference due to network configuration. CPU factor is
> important but I’ll spent more time to study network config.
>
> You have not defined to us your local network config, its latency, used
> operating system, networking kernel options and Erlang flags. Similarly,
> your AWS config is not know to us either.
>
> Best Regards,
> Dmitry
>
>
> > On Aug 29, 2016, at 6:55 PM, Arshad Ansari <arshadansari27@REDACTED>
> wrote:
> >
> > Hello there,
> >
> > Today, as a part of evalution of erlang diameter client, relay & server,
> I was benchmarking for requests per second. This same test was performed by
> me on my local network and I was able to get 30K requests per second
> between client and server using a single connection. However, on Aws I got
> to only about 11-12K requests per second. I had used c3.x4large instance
> with 16 cores and 30 GB ram for both client and server, which is 4 times
> the core I have and twice the ram I had when I was testing locally. These
> are compute enhanced instances to make sure I get to use core to the
> maximum, but I wasn't even using 60% of cores on server and 80% of cores on
> client.  I even used the same placement group to get 10GBps network
> bandwidth. I can't think of any reason why there would be just a downgrade.
> Has anyone experienced something of this sort on Aws while benchmarking
> erlang?
> >
> > Thanks in advance for the help.
> >
> > Regards,
> > Arshad
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160829/7e98254e/attachment.htm>