[erlang-questions] couchdb performace 10x: using NIF for file io

[Edmond Begumisa]

As a devout Couch user, what I'd like to see is a pragmatic approach as  
the couch team investigates this -- possibly targeting an optional  
configuration with a clear description of the trade-offs to Couch users.  
For example, the Couch team could advise using this new feature strictly  
in a private emulator devoted to couch since the benchmarks show that NIF  
io doesn't seem to have a detrimental effect on the rest of couch.

Personally, for my current uses, Couch's existing io performance is  
sufficient. The use of hovercraft, the Erlang view server and some  
workarounds to avoid mochiweb have been effective in giving me performance  
boosts where I need them.

That said, it cannot be denied that for a database server, a 2-10 fold  
increment in io performance is a good enough reason to do things that you  
should not really be doing or that are potentially disruptive. For some  
couch users, the pros might outweigh the cons. What I'd like to see is a  
clear description of what those cons are so I can decide for myself.

- Edmond -


On Thu, 28 Oct 2010 06:28:03 +1100, Joel Reymont <joelr1@REDACTED> wrote:

> Kenneth,
>
> The difference on the Mac is 10x for writes. It's at least 2x on Linux.
>
> How do you propose to investigate?
>
> Sent from my iPhone
>
> On 27/10/2010, at 20:13, Kenneth Lundin <kenneth.lundin@REDACTED> wrote:
>
>> My and the OTP teams view in this matter is:
>>
>> - Reimplementing standard functionality already written in C (but as a
>> driver and with asynch thread support) as NIFs is generally a bad
>> idea)
>> - Implementing potentially blocking function calls with NIFs is a bad  
>> idea.
>> - You should have VERY strong reasons for writing NIFs at all. It is a
>> big point in not writing anything in C if it can be avoided.
>> - The implementation of NIFs is more modern than the driver concept
>> and among other things the passing of data between Erlang and C-code
>> is more efficient for NIFs than for drivers. The driver concept does
>> still have its place and advantages especially for
>> handling external asynchronous input to Erlang processes. We plan to
>> improve the driver mechanisms and make solutions
>> from the NIFs to be available when writing drivers as well.
>>
>> If it is correct that NIFs for file operations is 2 times faster than
>> the current file operations in raw mode we will do something about it
>> because that difference is not justified. But first we must
>> investigate if that really is the case and where the performance is
>> lost.
>>
>> /Kenneth Erlang/OTP Ericsson
>>
>> On Wed, Oct 27, 2010 at 9:46 AM, Edmond Begumisa
>> <ebegumisa@REDACTED> wrote:
>>> Hi,
>>>
>>> I hope the Couch team isn't planning on doing this by default --  
>>> something
>>> about it makes me nervous...
>>>
>>> When CouchDB is on it's own, it might not be alarming/noticeable, but  
>>> I'm
>>> using CouchDB "embedded" in a wider Erlang/OTP application stack (i.e.  
>>> where
>>> Couch is just one of many OTP apps running in the *SAME* VM -- I have  
>>> a few
>>> hacks for avoiding socket communication.) I too worry about the  
>>> potential
>>> for NIF-endowed couch io disturbing the balance of Erlang's scheduling.
>>>
>>> It would be good to see similar benchmarking with the VM concurrently  
>>> doing
>>> things other than handling couch-related requests (which are implicitly
>>> synchronised in your case.)
>>>
>>> - Edmond -
>>>
>>> On Mon, 25 Oct 2010 07:59:11 +1100, Joel Reymont <joelr1@REDACTED>  
>>> wrote:
>>>
>>>>
>>>> On Oct 24, 2010, at 7:16 PM, Kenneth Lundin wrote:
>>>>
>>>>> I wonder how responsive the system is to other events when running  
>>>>> the
>>>>> benchmark.
>>>>
>>>> The benchmark simulates several hundred clients hitting a (mochiweb)  
>>>> web
>>>> server to read and write couchdb (json) documents. The system seems  
>>>> to stay
>>>> -highly- responsive, 10x so compared to the same system not using  
>>>> NIFs at
>>>> all.
>>>>
>>>> If low write response time is taken as a measure of system  
>>>> responsiveness
>>>> then the first graph shows that the responsiveness of the system has
>>>> increased dramatically. The write response here is the take taken to  
>>>> process
>>>> a web request to write a couch document.
>>>>
>>>> ---
>>>> http://twitter.com/wagerlabs
>>>>
>>>>
>>>> ________________________________________________________________
>>>> erlang-questions (at) erlang.org mailing list.
>>>> See http://www.erlang.org/faq.html
>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>>>
>>>
>>>
>>> --
>>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>>>


-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/