[erlang-questions] Load balancing/multiplexing rpc calls amongst Erlang Nodes

[Paul Peregud]

> find the erlang random generator skewing results making around 50/60
requests hitting one Node while others are waiting

This is unusual. I've seen PRNG from random module skewing the results, but
never to such extent. Please check if it is properly seeded (random:seed/0
seeds with predefined constant).

If seeding is done properly, then you may want to consider switching to
crypto:rand_uniform/2. It's a bit slower, but it produces random numbers of
quality better then enough for purpose of load balancing.

On Tue, Oct 23, 2012 at 7:22 AM, Joshua Muzaaya <joshmuza@REDACTED> wrote:

> Yes,i tries using the random method, but because requests are so
> frequently many, you find the erlang random generator skewing results
> making around 50/60 requests hitting one Node while others are waiting.
> Another thing is that, i am not using gen_servers at the Web Server layer.
> I am using yaws web server and for each connection, yaws spawns a process,
> this process communicates with Mnesia Nodes to query for data. But the
> connections are so many and i wanted to scale the application horizontally,
> adding more web servers and more mnesia Nodes. I came to think of a load
> balancing middle ware, abstracting my processes from knowing where the call
> has hit ( i.e on what mnesia node the call has hit). This middle ware
> ensures that requests are load balanced across my Mnesia DBs.
>
> That is the background of the problem. Its a real-time Web Notification
> system, plugged into a major intranet Management System. However, clients
> are many, and yaws is sustaining 30,000 concurrent connections at low
> peaks. I am a software engineer in one of the telecommunications companies
> in Africa. I keep running into a few memory problems on single node yaws
> server, so i need ti add more web servers to assist. Also, mnesia sometime
> will get *** Too many DB Tables ** when requests are too many and too
> frequent. I changed everything to use dirty operations and when i by-passed
> the transaction manager, things improved a bit.
>
> I need fellow erlangers to think of a load balancing algorithm in such a
> situation. Do you think a process dictionary like GPROC would be so useful
> ? i was kinda thinking about it last night, but i wonder how i would apply
> it in this case.
>
> Having one gen_server to decide where the request may go, might alos slow
> down the application as all requests will have to go through that
> gen_server.
>
>   <http://www.linkedin.com/pub/muzaaya-joshua/39/2ba/202>
> Designed with WiseStamp -
> <http://r1.wisestamp.com/r/landing?u=cf16262215eb8784&v=3.11.21&t=1350969121119&promo=10&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_10>Get
> yours<http://r1.wisestamp.com/r/landing?u=cf16262215eb8784&v=3.11.21&t=1350969121119&promo=10&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_10>
>
>
>
> On Tue, Oct 23, 2012 at 2:53 AM, Yogish Baliga <baliga@REDACTED> wrote:
>
>> One option is to run proxy gen_server on each Mnesia box and register
>> these gen_server pids with pg2. Now you can do load balancing on pg2
>> processes based on message queue length as described here
>>
>> http://dev.lethain.com/load-balancing-across-erlang-process-groups/
>>
>> When I used this method in my last project in Erlang, it gave better
>> result than normal round robin. Under very low load, all requests were
>> redirected to single Mnesia instance.
>>
>> http://dev.lethain.com/load-balancing-across-erlang-process-groups/
>>
>> -- baliga
>>
>>
>> On Mon, Oct 22, 2012 at 2:22 PM, Paul Peregud <paulperegud@REDACTED>wrote:
>>
>>> May you specify why load balancing should be based on time and can not
>>> be random? Have you implemented random load balancing? Has it proved
>>> to be insufficient?
>>>
>>> On Mon, Oct 22, 2012 at 9:51 AM, muzaaya_joshua <joshmuza@REDACTED>
>>> wrote:
>>> > Building from this question (
>>> http://stackoverflow.com/q/5339329/431620 ),
>>> > imagine an application with N Erlang Web Servers, and N/2 Mnesia
>>> Database
>>> > Nodes. The set up is such that the Web Servers, each, runs on its own
>>> > hardware server (say HP DL385), and each Mnesia Instance, runs on its
>>> own
>>> > hardware Server as well.
>>> >
>>> > Web Servers make rpc:call/4 calls to the back end (the Mnesia DB
>>> Servers).
>>> > The Data is all replicated across all the Mnesia instances. Now, you
>>> want to
>>> > have the calls being made to the Database servers, MULTIPLEXED, more
>>> > precisely ( by TIME), on each Web Server, so that some kind of LOAD
>>> > BALANCING is attained.
>>> >
>>> > If Web Server A makes a connection to Mnesia Instance 3, it cannot
>>> make the
>>> > next connection to the same Instance. All Database Nodes need to be
>>> kept
>>> > busy and not having any one of them idle while the others are working.
>>> The
>>> > Load balancing Algorithm should not be random, but should be aimed at
>>> > balancing the load on the Database Servers.
>>> >
>>> > Qn 1: Come up with your load balancing strategy, in such a situation.
>>> Also,
>>> > please show with some sample illustrative code, how you would
>>> implement this
>>> > strategy.
>>> >
>>> > Qn 2: If a Mnesia Instance goes down, how would your load balance
>>> Algorithm
>>> > adapt to the changes in the cluster ?
>>> >
>>> > Qn 3: Is there any Erlang library aimed at load balancing of Erlang
>>> Servers
>>> > working within the same system, and calling each other via rpc:call/4 ?
>>> >
>>> >
>>> >
>>> > --
>>> > View this message in context:
>>> http://erlang.2086793.n4.nabble.com/Load-balancing-multiplexing-rpc-calls-amongst-Erlang-Nodes-tp4655158.html
>>> > Sent from the Erlang Questions mailing list archive at Nabble.com.
>>> > _______________________________________________
>>> > erlang-questions mailing list
>>> > erlang-questions@REDACTED
>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Paul Peregud
>>> +48602112091
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
>>
>
>
> --
> *Muzaaya Joshua
> Systems Engineer
> +256774115170*
> *"Through it all, i have learned to trust in Jesus. To depend upon His
> Word"
> *
>
>
>
>


-- 
Best regards,
Paul Peregud
+48602112091
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121023/6c4a8121/attachment.htm>