[erlang-questions] Hidden node and topic subscription rpc's

[Michael Truog]

On 01/29/2015 09:15 AM, Matthieu Tourne wrote:
>
>
> On Wed, Jan 28, 2015 at 9:59 PM, Michael Truog <mjtruog@REDACTED <mailto:mjtruog@REDACTED>> wrote:
>
>     On 01/28/2015 07:19 PM, Matthieu Tourne wrote:
>>     Hi,
>>
>>     I started using py_interface as a layer to do rpc into an Erlang system, receive and send message, and eventually provide functionality to the that Erlang system.
>>
>>     One of the functions on the Erlang side would be :
>>
>>     handle_call({subscribe, Channel}, From, State) ->
>>       %% Saves From as a subscriber of this channel into the State
>>       %% When something will arrive on the Channel From will get a message.
>>
>>     The problem is when I'm sending an rpc:call via py_interface (a call message on the rex mailbox essentially). It seems that the pid associated to that rpc essentially disappears right when the rpc is completed.
>>
>>     As I gathered from this example [1], an alternative way to do this would be to rewrite my handle_call this way :
>>
>>     handle_call({subscribe, Channel, SubscribingPid}, _From, State)  ->
>>       %% Saves SubscribingPid instead of From.
>>
>>     And I would pass in the Pid of an active mailbox from my py_interface hidden node.
>>
>>     My question is two fold, would there be a way to keep a live Pid in the system (maybe [2] answers this problem as a byproduct), for longer than getting a result to that rpc call.
>>     Or would a proper way to do this be to have a proxy process to make the link between rpc's and a live mailbox in my hidden node.
>>     Also, is a hidden node the best way of achieving all of this ?
>     py_interface makes the running Python interpreter into a hidden node, similar to a cnode (http://www.erlang.org/doc/tutorial/cnode.html), just using Python source code (like jinterface does with Java source code).  You could probably modify the py_interface source code to run as an Erlang port (http://www.erlang.org/doc/tutorial/c_port.html). In both cases, all Python usage will have a single connection bottleneck (a single socket for a cnode and a single pair of pipes for a port).  Depending on the Python source code, you could argue that it will never matter due to the global interpreter lock.
>
>     However, if you do care about Python latency, it is more efficient to use the Python CloudI API.  To do RPC you would send a service request to another CloudI service (based on your example, an Erlang CloudI service).  An example of a Python service is at http://cloudi.org/#Python to show how a service request can be handled in Python (adding a send_sync CloudI API function call (http://cloudi.org/api.html#1_send_sync) would be the same as doing RPC).  The Python CloudI API is implemented only in Python in the cloudi module and with Python/C integration in the cloudi_c module (the quickest Python integration).
>
>
> I was actually looking at CloudI just yesterday!
> It looks very interesting, and seems to work like magic in my use case [1] :)
>
> I would have couple questions about it, what powers the api system ?
> Reading the code briefly I saw some Erlang external term format, but it wasn't clear which channel the protocol used (socket, pipes, etc ).
The Erlang external term format is used for sending a CloudI service request to the Erlang VM from an external service.  For an external service the service configuration sets the count_thread (number of threads to use for CloudI API objects concurrently) and count_process (number of OS processes for the external service instance, each using the same number of threads).  When an external service is started, count_thread * count_process sockets are created, so there is concurrent usage of the CloudI API among all external service threads.  The socket type (i.e., 'protocol' described at http://cloudi.org/api.html#2_services_add) is either inet tcp (tcp), unix domain socket tcp (local or default), or inet udp (udp (only really used for testing)).
>
> Also I was wondering if everything would need to become a cloudi_service to be consumed seamlessly by an external API, I'm working with existing code that already has a lot of gen_server patterns, if I have to layer on top this could be a burden.
Anything that handles CloudI service requests needs to be a CloudI service.  External Erlang source code can use the cloudi module to send to CloudI services without being services (it can just be a normal Erlang process that keeps the Context data created by cloudi:new/0)
>
> Finally not running on windows could be a problem, but I'm guessing I could rip out just CloudI API part of CloudI (it's a big repo) and make sure that works.
The CloudI source code is portable, so making it work on Windows shouldn't be a problem.  However, the main reason I have avoided trying to make Windows work is just due to no one using Windows for anything fault-tolerant (due to Window's historical instability) or any serious server usage.  I understand people have tried to (like the London Stock Exchange: http://www.computerworld.com/article/2467082/data-center/london-stock-exchange-to-abandon-failed-windows-platform.html and the Microsoft sponsored work with MPI clusters on Windows).  If Windows was somehow important for testing, CloudI could be used on Windows.  The simplest approach would likely be with Cygwin, but I have not attempted that yet.
>
> [1] https://github.com/CloudI/CloudI/tree/develop/doc#python

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150129/ca06ede8/attachment.htm>