[erlang-questions] Load balancing/multiplexing rpc calls amongst Erlang Nodes

Tue Oct 23 13:44:48 CEST 2012

I believe that it may be hard. Some possible problems here:

1) you need a way to keep system clock of webcores synchronized

2) you need a way to avoid any "waves of load" that may appear because of
slight desynchronization of system clocks, or random events on webcores.

3) you need a way to manage information about number of db cores (number of
time slots)

I prefer to rely on Law of large numbers if possible.

On Tue, Oct 23, 2012 at 11:52 AM, Joshua Muzaaya <joshmuza@REDACTED> wrote:

> Thank you so much. let me try that. But is it not possible to have a
> non-random method ? one that is intelligent and fair on all the nodes
>
>   <http://www.linkedin.com/pub/muzaaya-joshua/39/2ba/202>
> Designed with WiseStamp -
> <http://r1.wisestamp.com/r/landing?u=cf16262215eb8784&v=3.11.21&t=1350985854695&promo=10&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_10>Get
> yours<http://r1.wisestamp.com/r/landing?u=cf16262215eb8784&v=3.11.21&t=1350985854695&promo=10&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_10>
>
>
>
> On Tue, Oct 23, 2012 at 10:45 AM, Paul Peregud <paulperegud@REDACTED>wrote:
>
>> > find the erlang random generator skewing results making around 50/60
>> requests hitting one Node while others are waiting
>>
>> This is unusual. I've seen PRNG from random module skewing the results,
>> but never to such extent. Please check if it is properly seeded
>> (random:seed/0 seeds with predefined constant).
>>
>> If seeding is done properly, then you may want to consider switching to
>> crypto:rand_uniform/2. It's a bit slower, but it produces random numbers of
>> quality better then enough for purpose of load balancing.
>>
>>
>> On Tue, Oct 23, 2012 at 7:22 AM, Joshua Muzaaya <joshmuza@REDACTED>wrote:
>>
>>> Yes,i tries using the random method, but because requests are so
>>> frequently many, you find the erlang random generator skewing results
>>> making around 50/60 requests hitting one Node while others are waiting.
>>> Another thing is that, i am not using gen_servers at the Web Server layer.
>>> I am using yaws web server and for each connection, yaws spawns a process,
>>> this process communicates with Mnesia Nodes to query for data. But the
>>> connections are so many and i wanted to scale the application horizontally,
>>> adding more web servers and more mnesia Nodes. I came to think of a load
>>> balancing middle ware, abstracting my processes from knowing where the call
>>> has hit ( i.e on what mnesia node the call has hit). This middle ware
>>> ensures that requests are load balanced across my Mnesia DBs.
>>>
>>> That is the background of the problem. Its a real-time Web Notification
>>> system, plugged into a major intranet Management System. However, clients
>>> are many, and yaws is sustaining 30,000 concurrent connections at low
>>> peaks. I am a software engineer in one of the telecommunications companies
>>> in Africa. I keep running into a few memory problems on single node yaws
>>> server, so i need ti add more web servers to assist. Also, mnesia sometime
>>> will get *** Too many DB Tables ** when requests are too many and too
>>> frequent. I changed everything to use dirty operations and when i by-passed
>>> the transaction manager, things improved a bit.
>>>
>>> I need fellow erlangers to think of a load balancing algorithm in such a
>>> situation. Do you think a process dictionary like GPROC would be so useful
>>> ? i was kinda thinking about it last night, but i wonder how i would apply
>>> it in this case.
>>>
>>> Having one gen_server to decide where the request may go, might alos
>>> slow down the application as all requests will have to go through that
>>> gen_server.
>>>
>>>   <http://www.linkedin.com/pub/muzaaya-joshua/39/2ba/202>
>>> Designed with WiseStamp -
>>> <http://r1.wisestamp.com/r/landing?u=cf16262215eb8784&v=3.11.21&t=1350969121119&promo=10&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_10>Get
>>> yours<http://r1.wisestamp.com/r/landing?u=cf16262215eb8784&v=3.11.21&t=1350969121119&promo=10&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_10>
>>>
>>>
>>>
>>> On Tue, Oct 23, 2012 at 2:53 AM, Yogish Baliga <baliga@REDACTED> wrote:
>>>
>>>> One option is to run proxy gen_server on each Mnesia box and register
>>>> these gen_server pids with pg2. Now you can do load balancing on pg2
>>>> processes based on message queue length as described here
>>>>
>>>> http://dev.lethain.com/load-balancing-across-erlang-process-groups/
>>>>
>>>> When I used this method in my last project in Erlang, it gave better
>>>> result than normal round robin. Under very low load, all requests were
>>>> redirected to single Mnesia instance.
>>>>
>>>> http://dev.lethain.com/load-balancing-across-erlang-process-groups/
>>>>
>>>> -- baliga
>>>>
>>>>
>>>> On Mon, Oct 22, 2012 at 2:22 PM, Paul Peregud <paulperegud@REDACTED>wrote:
>>>>
>>>>> May you specify why load balancing should be based on time and can not
>>>>> be random? Have you implemented random load balancing? Has it proved
>>>>> to be insufficient?
>>>>>
>>>>> On Mon, Oct 22, 2012 at 9:51 AM, muzaaya_joshua <joshmuza@REDACTED>
>>>>> wrote:
>>>>> > Building from this question (
>>>>> http://stackoverflow.com/q/5339329/431620 ),
>>>>> > imagine an application with N Erlang Web Servers, and N/2 Mnesia
>>>>> Database
>>>>> > Nodes. The set up is such that the Web Servers, each, runs on its own
>>>>> > hardware server (say HP DL385), and each Mnesia Instance, runs on
>>>>> its own
>>>>> > hardware Server as well.
>>>>> >
>>>>> > Web Servers make rpc:call/4 calls to the back end (the Mnesia DB
>>>>> Servers).
>>>>> > The Data is all replicated across all the Mnesia instances. Now, you
>>>>> want to
>>>>> > have the calls being made to the Database servers, MULTIPLEXED, more
>>>>> > precisely ( by TIME), on each Web Server, so that some kind of LOAD
>>>>> > BALANCING is attained.
>>>>> >
>>>>> > If Web Server A makes a connection to Mnesia Instance 3, it cannot
>>>>> make the
>>>>> > next connection to the same Instance. All Database Nodes need to be
>>>>> kept
>>>>> > busy and not having any one of them idle while the others are
>>>>> working. The
>>>>> > Load balancing Algorithm should not be random, but should be aimed at
>>>>> > balancing the load on the Database Servers.
>>>>> >
>>>>> > Qn 1: Come up with your load balancing strategy, in such a
>>>>> situation. Also,
>>>>> > please show with some sample illustrative code, how you would
>>>>> implement this
>>>>> > strategy.
>>>>> >
>>>>> > Qn 2: If a Mnesia Instance goes down, how would your load balance
>>>>> Algorithm
>>>>> > adapt to the changes in the cluster ?
>>>>> >
>>>>> > Qn 3: Is there any Erlang library aimed at load balancing of Erlang
>>>>> Servers
>>>>> > working within the same system, and calling each other via
>>>>> rpc:call/4 ?
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > View this message in context:
>>>>> http://erlang.2086793.n4.nabble.com/Load-balancing-multiplexing-rpc-calls-amongst-Erlang-Nodes-tp4655158.html
>>>>> > Sent from the Erlang Questions mailing list archive at Nabble.com.
>>>>> > _______________________________________________
>>>>> > erlang-questions mailing list
>>>>> > erlang-questions@REDACTED
>>>>> > http://erlang.org/mailman/listinfo/erlang-questions
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Paul Peregud
>>>>> +48602112091
>>>>> _______________________________________________
>>>>> erlang-questions mailing list
>>>>> erlang-questions@REDACTED
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Muzaaya Joshua
>>> Systems Engineer
>>> +256774115170*
>>> *"Through it all, i have learned to trust in Jesus. To depend upon His
>>> Word"
>>> *
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>> Paul Peregud
>> +48602112091
>>
>
>
>
> --
> *Muzaaya Joshua
> Systems Engineer
> +256774115170*
> *"Through it all, i have learned to trust in Jesus. To depend upon His
> Word"
> *
>
>
>
>

-- 
Best regards,
Paul Peregud
+48602112091
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121023/7c9af487/attachment.htm>