[erlang-questions] append to a memory-mapped file

[Joel Meyer]

On Thu, Dec 16, 2010 at 3:36 AM, Joel Reymont <joelr1@REDACTED> wrote:

> Morten,
>
> On Dec 16, 2010, at 11:31 AM, Morten Krogh wrote:
>
> > What about keeping variables with the file size, the size of your data,
> and then when you want to append beyond the end of the file, you ftruncate
> and mmap again. The ftruncate should be with much more than 8 bytes of
> course.
>
> You are suggesting pre-allocating a file and writing to a specific position
> rather than appending to the end, correct?
>

I'm using this strategy for a simple time-series data DB written in C (with
port driver for access from Erlang). It works well enough. I've also
considered having each additional pre-allocated slab be in its own mmap'ed
region* so I don't have to munmap and mmap the entire region every time I
grow. So far pre-allocating works well enough that the occasional
lock/ftruncate/munmap/mmap/unlock hiccup doesn't really affect performance.

Joel

*The mmap region will most likely not line-up exactly with your
pre-allocated (via ftruncate) file chunks since you may want to write M
bytes at index K when only C are available - in this case you'd ftruncate a
large chunk and mmap at index K so you don't have to stitch together a read
from two different mmap'ed regions. Reads may no longer be constant time
since you may have to look at meta-data for N mmap regions to determine
which you should read from.


>        Thanks, Joel
>
> ---
> http://twitter.com/wagerlabs
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>