[erlang-questions] couchdb performace 10x: using NIF for file io

[Joel Reymont]

Kenneth,

The difference on the Mac is 10x for writes. It's at least 2x on Linux. 

How do you propose to investigate?

Sent from my iPhone

On 27/10/2010, at 20:13, Kenneth Lundin <kenneth.lundin@REDACTED> wrote:

> My and the OTP teams view in this matter is:
> 
> - Reimplementing standard functionality already written in C (but as a
> driver and with asynch thread support) as NIFs is generally a bad
> idea)
> - Implementing potentially blocking function calls with NIFs is a bad idea.
> - You should have VERY strong reasons for writing NIFs at all. It is a
> big point in not writing anything in C if it can be avoided.
> - The implementation of NIFs is more modern than the driver concept
> and among other things the passing of data between Erlang and C-code
> is more efficient for NIFs than for drivers. The driver concept does
> still have its place and advantages especially for
> handling external asynchronous input to Erlang processes. We plan to
> improve the driver mechanisms and make solutions
> from the NIFs to be available when writing drivers as well.
> 
> If it is correct that NIFs for file operations is 2 times faster than
> the current file operations in raw mode we will do something about it
> because that difference is not justified. But first we must
> investigate if that really is the case and where the performance is
> lost.
> 
> /Kenneth Erlang/OTP Ericsson
> 
> On Wed, Oct 27, 2010 at 9:46 AM, Edmond Begumisa
> <ebegumisa@REDACTED> wrote:
>> Hi,
>> 
>> I hope the Couch team isn't planning on doing this by default -- something
>> about it makes me nervous...
>> 
>> When CouchDB is on it's own, it might not be alarming/noticeable, but I'm
>> using CouchDB "embedded" in a wider Erlang/OTP application stack (i.e. where
>> Couch is just one of many OTP apps running in the *SAME* VM -- I have a few
>> hacks for avoiding socket communication.) I too worry about the potential
>> for NIF-endowed couch io disturbing the balance of Erlang's scheduling.
>> 
>> It would be good to see similar benchmarking with the VM concurrently doing
>> things other than handling couch-related requests (which are implicitly
>> synchronised in your case.)
>> 
>> - Edmond -
>> 
>> On Mon, 25 Oct 2010 07:59:11 +1100, Joel Reymont <joelr1@REDACTED> wrote:
>> 
>>> 
>>> On Oct 24, 2010, at 7:16 PM, Kenneth Lundin wrote:
>>> 
>>>> I wonder how responsive the system is to other events when running the
>>>> benchmark.
>>> 
>>> The benchmark simulates several hundred clients hitting a (mochiweb) web
>>> server to read and write couchdb (json) documents. The system seems to stay
>>> -highly- responsive, 10x so compared to the same system not using NIFs at
>>> all.
>>> 
>>> If low write response time is taken as a measure of system responsiveness
>>> then the first graph shows that the responsiveness of the system has
>>> increased dramatically. The write response here is the take taken to process
>>> a web request to write a couch document.
>>> 
>>> ---
>>> http://twitter.com/wagerlabs
>>> 
>>> 
>>> ________________________________________________________________
>>> erlang-questions (at) erlang.org mailing list.
>>> See http://www.erlang.org/faq.html
>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>> 
>> 
>> 
>> --
>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>>