[erlang-questions] Difference between enif_schedule_dirty_nif and driver_async?

[Michael Truog]

On 06/23/2014 08:29 AM, Max Lapshin wrote:
>
>
>
> On Mon, Jun 23, 2014 at 6:09 PM, Robert Raschke <rtrlists@REDACTED <mailto:rtrlists@REDACTED>> wrote:
>
>     Hi Max,
>
>     since you are already using separate nodes running NIFs, there's probably not much worth in converting to a C node or port. So this is just for posterity.
>
>
> Wrong. It will be a very big work to convert to C node.
>
> Right now my data flow is following:
>
> 1) 5-8 mbit of data are transferred to separate node
> 2) they pass through complicated code on erlang that prepares them for decoding
> 3) then original frame goes to decoder
> 4) decoder outputs from 100 to 1000 megabits of data
> 5) then they again go to tricky logic
> 6) after this huge amount of raw data gets into encoders
> 7) and again through erlang it gets back, being around 2-4 mbit/s
>
>
> So erlang has to copy less than 10 mbit here and there. All serious payload is zerocopies inside one node.
>
> If I change to driver, then I will have to write much more boilerplate demarshalling code.
>
> If I change to C node or port then I will have to rewrite big amount of working erlang code in C.
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
If you want, you could use https://github.com/okeuday/GEPD/ to be able to switch between an Erlang port and an Erlang port driver (where the port driver can use the async thread pool).  The code is done with the preprocessor to avoid overhead.  It would be possible to add NIF generation that uses the dirty scheduler for async calls.

Since you are dealing with higher throughput though, it seems like you should want more than 1 connection to the Erlang VM and CloudI would help you there (http://cloudi.org), since otherwise the single connection (cnode or port) will always be a bottleneck (making multiple connections for higher throughput isn't really a way of solving the problem, just a way of ignoring it).

With a port driver and port level locking you can have multiple ports used concurrently if your C/C++ code is perfect.  So, that is an approach if you are able to have perfect source code for integration.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140623/54f5dab9/attachment.htm>