[erlang-questions] couchdb performace 10x: using NIF for file io

[Paul Davis]

On Mon, Oct 25, 2010 at 10:55 AM, Joel Reymont <joelr1@REDACTED> wrote:
>
> On Oct 25, 2010, at 3:25 PM, Paul Davis wrote:
>
>> In your benchmark you'd basically be
>> sacrificing an Erlang scheduler thread to speed up the synchronous
>> writes while the other schedulers are freely available to handle
>> read/write requests.
>
> I sacrifice the schedule to make -both- read and write requests since I replaced all calls to the Erlang file module with calls to my NIFs. Well, except for trivial bits like rename and delete.
>
> I think it should be fine to sacrifice a scheduler for small periods of time on a multicore machine.
>
> I'll ask Mikeal for multi-database tests.
>
> ---
> http://twitter.com/wagerlabs
>
>

Joel,

What's the final size of your database though? IIRC, the point of
those tests wasn't really to test how fast the disk can read/write
data, but to look at hour readers and writers interact, ie, do lots of
readers make writes disproportionately slower? The point being that if
you have enough RAM you could be caching extensive parts of the
database in memory which would have the general effect of making most
reads be roughly non-blocking.

The proper way to test this would be to try and figure out a way to
saturate disk io so that a large number of read/write calls are
blocking and then try and do something that doesn't touch disk. I
can't think of a very good way to set that up other than maybe to
create a large number of large databases, compact each of them
simultaneously, and then try and run the reader/writer tests or some
such.

HTH,
Paul Davis