[erlang-questions] why is mmap so darn difficult

[Max Lapshin]

mmap is faster than file when you are reading from vfs cache. It is
faster on, lets say 5%, or even 15%.

But speed of microbenchmarks is nothing when you run real service. You
need to make your software controllable.

For example, erlyvideo makes gen_server:call to a special queue before
each disk read. It is slower, but it makes server much more stable.
It makes many other different checks like process_info(Pid,
message_queue_len) to make things more stable and server behaviour
smooth.

If you chase for speed of microbenchmarks, you can build a server that
runs excellently on your laptop, but begins to block on production.

This is my lesson from my experience.

On Wed, Jun 12, 2013 at 9:44 PM, Tim Watson <watson.timothy@REDACTED> wrote:
> Hi Max,
>
> On 12 Jun 2013, at 10:43, Max Lapshin wrote:
>
>> Of course, you will not be able to handle such load with mmap, because
>> you will suffer from blocking whole VM on mmap page faults.
>>
>
> Yes of course you're right about that, with mmap running in a NIF. I attempted to use both emmap and the mmap driver you wrote for erlyvideo and found the resulting behaviour highly variable.
>
>> Add more async threads and profile application. Don't waste your time on mmap.
>
> I'm starting to wonder whether attempting to provide very high speed concurrent reads + writes in Erlang is going to work at all. I might spend some time looking at lmdb instead, since it claims some extremely high numbers. I was hoping to avoid writing a driver though, but ho hum...
>
> Cheers,
> Tim