[erlang-questions] Difference between enif_schedule_dirty_nif and driver_async?

[Steve Vinoski]

On Sat, Jun 21, 2014 at 8:40 AM, Max Lapshin <max.lapshin@REDACTED> wrote:

> I'm doing very CPU bound task: video transcoding.
>
>
> What is better to use: dirty NIF or driver async?
>
> Is it correct to ask "what is better" in this case?
>

IMO dirty NIFs are much easier to write, but there are other factors to
consider.

If you're going to have a number of concurrent Erlang processes running
dirty native code, then you need to watch out for occupying all the dirty
CPU scheduler threads such that your dirty jobs queue up faster than they
can be processed. By default you get one dirty CPU scheduler thread per
regular scheduler thread, and the number of dirty CPU scheduler threads may
not exceed the number of regular scheduler threads (i.e, if you reduce the
number of regular scheduler threads online at runtime, the number of dirty
CPU scheduler threads online is likewise reduced). So if you plan to have a
lot of video transcoding jobs running concurrently, where "a lot" is a
number much larger than the number of scheduler threads, it might be better
to use the async thread pool since you can easily make it as large as you
want. Note that a dirty NIF can cooperatively yield a dirty scheduler
thread by calling enif_schedule_dirty_nif() again, as that allows the dirty
scheduler thread to reenter the emulator and dequeue a new job, but this
still doesn't solve the problem of having too many jobs competing for dirty
CPU scheduler thread resources.

A nice thing about dirty NIFs is they're based on the normal Erlang process
model, where a regular process runs native code on a dirty scheduler
thread. Drivers and the async pool do not follow this model, which is part
of the reason that sometime in the future, the async thread pool and
drivers could go away, all in favor of native processes. None of this will
happen prior to Erlang 19 at the earliest, but still, it's perhaps
something to consider.

And one other caveat: the dirty NIF API is experimental in 17.x and is
subject to change.

--steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140621/1665fca2/attachment.htm>