[erlang-questions] couchdb performace 10x: using NIF for file io

[Kenneth Lundin]

My and the OTP teams view in this matter is:

- Reimplementing standard functionality already written in C (but as a
driver and with asynch thread support) as NIFs is generally a bad
idea)
- Implementing potentially blocking function calls with NIFs is a bad idea.
- You should have VERY strong reasons for writing NIFs at all. It is a
big point in not writing anything in C if it can be avoided.
- The implementation of NIFs is more modern than the driver concept
and among other things the passing of data between Erlang and C-code
is more efficient for NIFs than for drivers. The driver concept does
still have its place and advantages especially for
handling external asynchronous input to Erlang processes. We plan to
improve the driver mechanisms and make solutions
from the NIFs to be available when writing drivers as well.

If it is correct that NIFs for file operations is 2 times faster than
the current file operations in raw mode we will do something about it
because that difference is not justified. But first we must
investigate if that really is the case and where the performance is
lost.

/Kenneth Erlang/OTP Ericsson

On Wed, Oct 27, 2010 at 9:46 AM, Edmond Begumisa
<ebegumisa@REDACTED> wrote:
> Hi,
>
> I hope the Couch team isn't planning on doing this by default -- something
> about it makes me nervous...
>
> When CouchDB is on it's own, it might not be alarming/noticeable, but I'm
> using CouchDB "embedded" in a wider Erlang/OTP application stack (i.e. where
> Couch is just one of many OTP apps running in the *SAME* VM -- I have a few
> hacks for avoiding socket communication.) I too worry about the potential
> for NIF-endowed couch io disturbing the balance of Erlang's scheduling.
>
> It would be good to see similar benchmarking with the VM concurrently doing
> things other than handling couch-related requests (which are implicitly
> synchronised in your case.)
>
> - Edmond -
>
> On Mon, 25 Oct 2010 07:59:11 +1100, Joel Reymont <joelr1@REDACTED> wrote:
>
>>
>> On Oct 24, 2010, at 7:16 PM, Kenneth Lundin wrote:
>>
>>> I wonder how responsive the system is to other events when running the
>>> benchmark.
>>
>> The benchmark simulates several hundred clients hitting a (mochiweb) web
>> server to read and write couchdb (json) documents. The system seems to stay
>> -highly- responsive, 10x so compared to the same system not using NIFs at
>> all.
>>
>> If low write response time is taken as a measure of system responsiveness
>> then the first graph shows that the responsiveness of the system has
>> increased dramatically. The write response here is the take taken to process
>> a web request to write a couch document.
>>
>> ---
>> http://twitter.com/wagerlabs
>>
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>
>
>
> --
> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>