[erlang-questions] Discussion and proposal regarding rpc scalability

[Benoit Chesneau]

Hrm wouldn'mtwould'nt the method in gen_rpc
https://github.com/priestjim/gen_rpc

partially fit the bill?

- benoit

On Fri, 12 Feb 2016 at 02:15, José Valim <jose.valim@REDACTED>
wrote:

> Thanks everyone for the feedback so far.
>
> Douglas, I thought about such solution. The problem is that the caller
> does not know how many rex instances there are per node. Therefore you
> would need to first message a single process that does the consistent
> hashing on the node, which is an additional step (extra copying) and may
> still be a bottleneck. How would you solve this particular problem?
>
> Chandru, thanks for validating the concerns raised. I agree your proposal
> has many improvements. IIRC the release project from where the paper came
> from explored similar ideas as well. I am afraid however such solution
> can't be a replacement. A lot of the usage of rex in OTP and other
> applications is to request information from a particular node and therefore
> the load balancing and node grouping wouldn't be desired.
>
> Sean, the idea is to leverage all of the rpc functionality without
> reimplementing it. Like the group leader handling, the call/async_call API
> and so forth. So while we could tell folks to use a gen_server (or to use
> spawn), we would be neglecting what rpc provides. In other words, I would
> use it for the same reason I use rpc, I'd just have it in my own app's tree.
>
>
>
>
> *José Valim*
> www.plataformatec.com.br
> Skype: jv.ptec
> Founder and Director of R&D
>
> On Thu, Feb 11, 2016 at 10:58 PM, Sean Cribbs <seancribbs@REDACTED>
> wrote:
>
>> José,
>>
>> It's interesting to me that your example of making registered RPC
>> receivers isn't that much different from having registered processes with
>> specific messages (call/cast/etc) they handle. What do you see as the
>> use-case of allowing generic RPC in that scenario?
>>
>> Cheers,
>>
>> Sean
>>
>> On Thu, Feb 11, 2016 at 3:04 PM, José Valim <
>> jose.valim@REDACTED> wrote:
>>
>>> Hello everyone,
>>>
>>> I was reading the publication "Investigating the Scalability Limits of
>>> Distributed Erlang
>>> <http://www.dcs.gla.ac.uk/~amirg/publications/DE-Bench.pdf>" and one of
>>> the conclusions is:
>>>
>>> *> We observed that distributed Erlang scales linearly up to 150 nodes
>>> when no global command is made. Our results reveal that the latency of rpc
>>> calls rises as cluster size grows. This shows that spawn scales much better
>>> than rpc and using spawn instead of rpc in the sake of scalability is
>>> advised. *
>>>
>>> The reason why is highlighted in a previous section:
>>>
>>> *> To find out why rpc’s latency increases as the cluster size grows, we
>>> need to know more about rpc. (...) There is a generic server process (gen
>>> server) on each Erlang node which is named rex. This process is responsible
>>> for receiving and handling all rpc requests that come to an Erlang node.
>>> After handling the request, generated results will be returned to the
>>> source node. In addition to user applications, rpc is also used by many
>>> built-in OTP modules, and so it can be overloaded as a shared service.*
>>>
>>> In other words, the more applications we have relying on rpc, the more
>>> likely rpc will become a bottleneck and increase latency. I believe we have
>>> three options here:
>>>
>>> 1. Promote spawn over rpc, as the paper conclusion did (i.e. mention
>>> spawn in the rpc docs and so on)
>>> 2. Leave things as is
>>> 3. Allow "more scalable" usage of rpc by supporting application specific
>>> rpc instances
>>>
>>> In particular, my proposal for 3 is to allow developers to spawn their
>>> own rpc processes. In other words, we can expose:
>>>
>>> rpc:start_link(my_app_rpc) %% start your own rpc
>>>
>>> rpc:call({my_app_rpc, nodename}, foo, bar, [1, 2, 3]) %% invoke your own
>>> rpc at the given node
>>>
>>>
>>> This is a very simple solution that moves the bottleneck away from rpc's
>>> rex process since developers can place their own rpc processes in their
>>> application's tree. The code changes required to support this feature are
>>> also minimal and are almost all at the API level, i.e. support a tuple were
>>> today a node is expected or allow the name as argument, mimicking the same
>>> API provided by gen_server that rpc relies on. We won't change
>>> implementation details. Finally, I believe it will provide a more
>>> predictable usage of rpc.
>>>
>>> Feedback is appreciated!
>>>
>>> *José Valim*
>>> www.plataformatec.com.br
>>> Skype: jv.ptec
>>> Founder and Director of R&D
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160212/8212838f/attachment.htm>