[erlang-questions] Mnesia deleting log file

[Jesper Louis Andersen]

I'm curious: are the data written as part of a mnesia transaction?


On Thu, Nov 23, 2017 at 12:22 PM Arun <arunp@REDACTED> wrote:

> Even after setting the "dump_log_update_in_place" configuration to false,
> the data is not getting retained
> after a power outage. Any more configurations that can be tried.
> Regards,
> Arun P
>
>
>
> On Thursday 23 November 2017 02:46 AM, Michael Schmidt wrote:
>
> I agree—setting dump_log_in_place to false makes extremely unlikely for
> mnesia to fail in this way.  Both JFFS2 and UBIFS ensure that rename is
> atomic, even when power is pulled.
>
>
>
> For desktop applications, the culprit is likely the VM being killed, not
> the whole system crashing.  Thus the rename was either executed or not;
> either way we have a complete DAT file.
>
>
>
> There is always a chance of things like this happening:
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits, but
> this shouldn’t happen on most hardware.
>
>
>
> Note that the default for dump_log_time_threshold is 3 minutes.  Speed vs
> Safety is always a trade-off, but many people will be surprised to lose
> data in a crash 2 minutes after saving.
>
>
>
>
>
> *From:* Jesper Louis Andersen [mailto:jesper.louis.andersen@REDACTED
> <jesper.louis.andersen@REDACTED>]
> *Sent:* Wednesday, November 22, 2017 7:54 AM
> *To:* Michael Schmidt <Michael.K.Schmidt@REDACTED>
> <Michael.K.Schmidt@REDACTED>
> *Cc:* Arun <arunp@REDACTED> <arunp@REDACTED>; erlang-questions@REDACTED
> *Subject:* Re: [erlang-questions] Mnesia deleting log file
>
>
>
> This thread makes we wonder about the following section in the Mnesia
> documentation:
>
> "If a power failure occurs during the dump, this can cause the randomly
> accessed DAT files to become corrupt. If the parameter is set to false,
> Mnesia copies the DAT files and target the dump to the new temporary
> files. If the dump is successful, the temporary files are renamed to their
> normal DAT suffixes. The possibility for unrecoverable inconsistencies in
> the data files becomes much smaller with this strategy. However, the actual
> dumping of the transaction log becomes considerably slower. The system
> designer must decide whether speed or safety is the higher priority."
>
> In UNIX(Posix) a rename(2) call is atomic. The worst case, which is an OS
> crash might leave you with a temporary file and the target file, hardlinked
> as if ln() was called, but the move itself is atomic, even under OS
> crashes. Of course, this assumes you correctly wrote your temporary file,
> fsync()'ed the file and then performed the rename() as you should.
>
> With that information in mind, what is the scenario which can make mnesia
> fail if the dump_log_update_in_place is set to false? There may be a
> window, but the operating system cannot be the provider of this window at
> all, unless it is incorrectly implemented or configured[1]. Hence my
> question.
>
> [1] The primary concern are write caches on disks which can have a write
> hole unless they have proper (working!) battery backup of their cache or
> are able to write their memory to a small solid state part of the chip. Or
> if you are using ZFS where copy-on-write semantics completely eliminates
> the write hole.
>
>
>
>
>
> On Wed, Nov 22, 2017 at 2:29 PM Michael Schmidt <
> Michael.K.Schmidt@REDACTED> wrote:
>
> Hi Arun,
>
>
>
> If you happen to reboot when Mnesia is writing out its logfile, it can
> become corrupt (and lose data).  We fight this issue on embedded devices a
> lot.
>
>
>
> Look “Configuration Parameters” section here:
>
> http://erlang.org/doc/man/mnesia.html#id69584
>
>
>
> You will want to set dump_log_update_in_place to false.  This will make
> it write out the table to a new file, and rename it the very end.
>
>
>
> One other option to look at is dump_log_time_threshold to change how
> often items are flushed to disk.  For configuration data, you can likely
> set this to a few seconds.
>
>
>
> These options can be set via the sys.config file as well
>
>
>
> Mike
>
>
>
> *From:* erlang-questions-bounces@REDACTED [mailto:
> erlang-questions-bounces@REDACTED] *On Behalf Of *Arun
> *Sent:* Tuesday, November 21, 2017 10:16 PM
> *To:* erlang-questions@REDACTED
> *Subject:* [erlang-questions] Mnesia deleting log file
>
>
>
> Dear all,
>
> I've a program written in erlang which uses Mnesia application as the
> database application. I've a table created by name "configuration_table"
> which stores
>
> certain configurations that need to be persistent. Occasionally, whenever
> I restart my program the following error is thrown by mnesia and I end up
> losing
>
> all the persistent configurations.
>
> I've searched about this problem in the erlang documentation and all it
> tells is "Node not running". What could be the probable cause for this
> problem
>
> and how do I fix it?
>
> Mnesia('test@REDACTED <ommp82_plus@REDACTED>'): Data may be missing,
> Corrupt logfile deleted:
> "/home/utl/mnesia_database/configuration_table.DCL", {node_not_running,
> 'test@REDACTED <ommp82_plus@REDACTED>'}
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> ______________________________________________________________________
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> ______________________________________________________________________
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20171123/5eb6a794/attachment.htm>