[erlang-questions] couchdb performace 10x: using NIF for file io

[Kenneth Lundin]

The so called async threads selected with the +A flag are only used by
the file driver that is in the original Erlang distribution from
Ericsson (erlang.org). They can also be used by other user written
drivers (I don't know if CouchDB has any driver like that). An
implementation of file operations as NIF's will not make use of the
asynch threads at all and starting Erlang with the +A <NN> flag should
not make any difference at all for the
execution of the NIFs.

NIFs are executing in the scheduler thread and all other Erlang
processes handled by the same scheduler
are blocked while the NIF is executed. Erlang processes handled by
other schedulers in an SMP setup can whoever execute as normal.

Calling NIFs with potentially long execution times can easily destroy
all multi processing capabilities and
soft real time characteristics for an Erlang node.

I wonder how responsive the system is to other events when running the
benchmark.

Of course it is possible to optimize the file operations if you know
exactly what file system you are working
towards, what file operations the application makes etc. but in the
general case it is not that easy.


/Kenneth Erlang/OTP , Ericsson
On Sun, Oct 24, 2010 at 7:35 PM, Joel Reymont <joelr1@REDACTED> wrote:
> Dave,
>
> On Oct 24, 2010, at 6:12 PM, Dave Smith wrote:
>
>> +A anything won't help you anymore. You are now doing the I/O on the
>> VM scheduler thread and blocking ANY erlang code (on that scheduler)
>> while the file op runs.
>
> Are you sure of this? Can you refer me to the spot in the VM code where I can learn more?
>
> The reason I'm surprised is that increasing the number of threads in the async thread pool had a clear effect on my benchmark. Spikes were, basically, eliminated.
>
>        Thanks in advance, Joel
>
> ---
> http://twitter.com/wagerlabs
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>