[erlang-questions] couchdb performace 10x: using NIF for file io

[Paul Davis]

On Sun, Oct 24, 2010 at 4:59 PM, Joel Reymont <joelr1@REDACTED> wrote:
>
> On Oct 24, 2010, at 7:16 PM, Kenneth Lundin wrote:
>
>> I wonder how responsive the system is to other events when running the
>> benchmark.
>
> The benchmark simulates several hundred clients hitting a (mochiweb) web server to read and write couchdb (json) documents. The system seems to stay -highly- responsive, 10x so compared to the same system not using NIFs at all.
>
> If low write response time is taken as a measure of system responsiveness then the first graph shows that the responsiveness of the system has increased dramatically. The write response here is the take taken to process a web request to write a couch document.
>
> ---
> http://twitter.com/wagerlabs
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>

Joel,

I think the issue here is the way that we sequester file io to a
single Erlang process. In your benchmark you'd basically be
sacrificing an Erlang scheduler thread to speed up the synchronous
writes while the other schedulers are freely available to handle
read/write requests.

I'm not sure if Mikeal has written a test for Relaximation that runs a
similar test that hits multiple databases. I think the way I'd try and
show the issue would be to have a large number of clients attempting
to write to their own database.

Also, a trivial way to prove to your self that the NIF's are indeed
called in the same scheduler threads is to create a NIF that has a
function that just does "while(1){}". If you call that function in
more processes than you have schedulers the VM will halt.

HTH,
Paul Davis