[erlang-questions] Discussion and proposal regarding rpc scalability

[Douglas Rohrer]

As a Basho engineer, I'd be remiss if I didn't point out the opportunity
for the use of some sort of consistent hashing of Pid -> rex instance, with
a configurable number of rex instances per node. More generic and easily
scalable (what happens when your app overloads its one instance of rex? now
you need to go modify your code to add another N, etc.)

Calling node would hash the calling PID to its REX instance, which would
figure out the remote Pid's rex instance in a similar manner, send the
request over, and coordinate the return results. rpc:call would remain the
same in this case.

Doug

On Thu, Feb 11, 2016 at 4:04 PM José Valim <jose.valim@REDACTED>
wrote:

> Hello everyone,
>
> I was reading the publication "Investigating the Scalability Limits of
> Distributed Erlang
> <http://www.dcs.gla.ac.uk/~amirg/publications/DE-Bench.pdf>" and one of
> the conclusions is:
>
> *> We observed that distributed Erlang scales linearly up to 150 nodes
> when no global command is made. Our results reveal that the latency of rpc
> calls rises as cluster size grows. This shows that spawn scales much better
> than rpc and using spawn instead of rpc in the sake of scalability is
> advised. *
>
> The reason why is highlighted in a previous section:
>
> *> To find out why rpc’s latency increases as the cluster size grows, we
> need to know more about rpc. (...) There is a generic server process (gen
> server) on each Erlang node which is named rex. This process is responsible
> for receiving and handling all rpc requests that come to an Erlang node.
> After handling the request, generated results will be returned to the
> source node. In addition to user applications, rpc is also used by many
> built-in OTP modules, and so it can be overloaded as a shared service.*
>
> In other words, the more applications we have relying on rpc, the more
> likely rpc will become a bottleneck and increase latency. I believe we have
> three options here:
>
> 1. Promote spawn over rpc, as the paper conclusion did (i.e. mention spawn
> in the rpc docs and so on)
> 2. Leave things as is
> 3. Allow "more scalable" usage of rpc by supporting application specific
> rpc instances
>
> In particular, my proposal for 3 is to allow developers to spawn their own
> rpc processes. In other words, we can expose:
>
> rpc:start_link(my_app_rpc) %% start your own rpc
>
> rpc:call({my_app_rpc, nodename}, foo, bar, [1, 2, 3]) %% invoke your own
> rpc at the given node
>
>
> This is a very simple solution that moves the bottleneck away from rpc's
> rex process since developers can place their own rpc processes in their
> application's tree. The code changes required to support this feature are
> also minimal and are almost all at the API level, i.e. support a tuple were
> today a node is expected or allow the name as argument, mimicking the same
> API provided by gen_server that rpc relies on. We won't change
> implementation details. Finally, I believe it will provide a more
> predictable usage of rpc.
>
> Feedback is appreciated!
>
> *José Valim*
> www.plataformatec.com.br
> Skype: jv.ptec
> Founder and Director of R&D
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160211/b1c79cf8/attachment.htm>