[erlang-questions] why is mmap so darn difficult

[Michael Loftis]

On Wed, Jun 12, 2013 at 2:41 AM, Tim Watson <watson.timothy@REDACTED> wrote:
> Max,
>
> On 12 Jun 2013, at 09:42, Max Lapshin wrote:
>
> I thought that video streaming, when you need to read about half of
> gigabyte per second from disk, repack it and write to network, is the
> good place for mmap.
> It was a mistake. If you get slow mmap disk read, than whole erlang
> sticks and you get PHP-like performance when whole thread is blocked
> for one client.
>
>
> Isn't that because the pread call you used was in a NIF, and therefore could
> block the scheulder(s) no? My thought's about the approach were initially
> somewhat different. The write operations were just file:write, whereas there
> would be no pread, but rather data would be delivered to the socket using
> sendfile/5, which would utilise the async thread pool if it's enabled.
>
> Perhaps in embedded mmap would be better when you want to save memory
> and have 10 requests per day, but on high load mmap doesn't help.
>
>
> I'd like to handle writing hundreds of thousands of input events per second
> and reading at various different speeds, but supporting (reads of) up to
> tens of thousands per second.
>

mmap would still need to run in a thread to make things consistently
faster.  Whenever a miss happens without being in a thread you're
going down the slow path of a page fault.  If this is in a main erlang
VM thread, that thread is just stalled out and doing no useful work
while this happens, and then requests stack up behind it.  The thread
eventually becomes available again and hopefully services a lot of
those backlogged requests before being hit with another page fault
to/from disk.  If the page faults happen often enough the whole thing
comes apart really quickly.  This is why file:* would, should, be
faster.

The problem isn't in the "everything is going well" case, it's when
the slow path hits, it doesn't block just that one request, it blocks
all possible work on that entire thread with mmap, partly because you
can't even guess if something is available or not, you just have to
hope/pray it is.  With erlang async i/o threads, the i/o thread is the
one that gets blocked on a read or a write.  Zero copy read/write
could help some, but your overwhelming latency is going to be in those
slow hits.  You still benefit from the OSes disk cache even without
mmap.


--

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler