DETS table auto_save behaviour

[Ulf Wiger]

Mnesia has a WAL (Write-Ahead Log), in which it writes data safely. It then
writes to dets (if that's the chosen table type).

At startup, dets files are repaired if they don't appear to have been
properly closed. Then the transaction log is applied, making sure that the
database is consistent.

Repairs of dets files have been known to take time in the past, but I think
OTP has optimized it, Klarna optimized the mnesia end of it, and both
computers and disks are insanely faster now.

I'd say that the most glaring issue with disc_only_copies in mnesia is not
even the 2 GB limit, but the fact that if you get there, dets will simply
discard the update, and mnesia won't even notice. That is, your application
must ensure that you never exceed the dets limit.

Most people use disc_copies for persistence, since they have better
performance and better reliability than disc_only_copies. The downside is
that the table will also fit in RAM. A different approach would be to use a
backend plugin. There are three alternatives to choose from, as far as I
know: leveldb, leveled, and rocksdb. There may be issues building leveldb
on newer OTP versions. Leveled is (almost) entirely erlang-based, so it
wins hands-down on build time. Rocksdb should be the fastest, although the
difference isn't dramatic.

BR,
Ulf W



On Thu, May 27, 2021 at 8:52 AM Frank Muller <frank.muller.erl@REDACTED>
wrote:

> How about Mnesia and persistence to disk?
>
>
>> It's always tricky with open files during some abrupt crashes. OS-level
>> file system caching means that not all written data may have been
>> physically written to disk.
>>
>> To detect this, dets has a flag indicating whether the file was properly
>> closed. As I understand it, the 'auto-save' does the same thing as when the
>> file is closed, except the file stays open.
>>
>> BR,
>> Ulf W
>>
>> Den ons 26 maj 2021 23:10Mikael Pettersson <mikpelinux@REDACTED> skrev:
>>
>>> On Tue, May 25, 2021 at 8:43 AM Nicolas Martyanoff <khaelin@REDACTED>
>>> wrote:
>>> > I was hoping to use DETS as a local persistent buffer in case data
>>> > cannot be written to a remote database, but it seems impossible to
>>> > guarantee that every entry is being sync-ed to disk.
>>>
>>> I'm not too familiar with the internals of DETS, but basically data
>>> goes straight to/from disk while meta-data about allocated and free
>>> areas of the file are cached in memory. I don't know if writes are
>>> sync or not. In our experience, DETS files are somewhat fragile, plus
>>> they have a hard 2GB size limitation which made them extremely awkward
>>> for our use case (large mnesia tables). That's part of the reason we
>>> migrated most of our mnesia tables to eleveldb.
>>>
>>> If I had to have a standalone (not mnesia) local persistent store I'd
>>> probably go with eleveldb (or one of its spinoffs) if I needed lookups
>>> by key, or a disk_log if I just needed a FIFO buffer. disk_log allows
>>> you to choose how sync or async your writes are. _I_ wouldn't use
>>> DETS.
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210527/6fa1f1c1/attachment.htm>