[erlang-questions] mnesia silent failure and table corruption

[Ahmed Omar]

Hi Janos,
In my view, the main purpose for mnesia is being in-memory distributed
database, not a complete RDBMS replacement.

There are some projects to user different backends than DETS for mnesia
(for persistence data).
One project was to a framework for mnesia pluggable backends (based on
earlier work of Ulf Wiger on mnesia_ext_filesystem). The first plugin was
based on LevelDB.
It was presented in EUC 2012
http://www.erlang-factory.com/conference/ErlangUserConference2012/speakers/RolandKarlsson

I'm not sure about the project status right now.

Best Regards,
Ahmed



On Wed, Sep 4, 2013 at 4:19 PM, Janos Hary <janos.n.hary@REDACTED> wrote:

> Ahmed,****
>
> ** **
>
> Thanks for the detailed explanation. ****
>
> ** **
>
> As per http://www.erlang.org/doc/apps/mnesia/Mnesia_chap4.html :****
>
> “Durability. This transaction property ensures that changes made to the
> DBMS by a transaction are permanent. Once a transaction has been committed,
> all changes made to the database are durable - i.e. they are written safely
> to disc and will not be corrupted or disappear.”****
>
> ** **
>
> I would definitely expect {aborted, Reason} in case of a storage operation
> failure. ****
>
> ** **
>
> Since then I found another occurrence of this problem. If I’m precautious
> and put my data in fragmented table, but have a secondary index, that index
> will still kept in a single dets file. In my case I tried to index SHA
> hashes and the index grows pretty fast.****
>
> ** **
>
> Thanks again for your answer, even if it was a bad news for me. Mnesia
> looks to be a great database, but I think it was a mistake from my side to
> try using it as a complete RDBMS replacement.****
>
> ** **
>
> Best Regards,****
>
> Tamas****
>
> ** **
>
> *From:* Ahmed Omar [mailto:spawn.think@REDACTED]
> *Sent:* Wednesday, September 04, 2013 2:25 PM
> *To:* Janos Hary
> *Cc:* erlang-questions; Håkan Mattsson
>
> *Subject:* Re: [erlang-questions] mnesia silent failure and table
> corruption****
>
> ** **
>
> Hi Janos, ****
>
> ** **
>
> Indeed, in that case transactions could be considered succeeded while
> actual storage operation failed...****
>
> ** **
>
> The thing is, If mnesia needs to write data to a store (in this case
> dets), it doesn't check for results for this operation. Actually, it does
> even catch exceptions. (mnesia_tm:do_update/4)****
>
> ** **
>
> As dets have a maximum of 2GB for file size, I believe the table gets
> corrupted after that. (somebody correct me please)****
>
> So any operation on that dets table will return the bad_object error.****
>
> ** **
>
> So the commit is considered succeeded, mnesia logs it, transaction returns
> {atomic, ok}****
>
> ** **
>
> sync_transaction won't help, because it just waits for the commit to be
> synced/logged, but also doesn't check results of the store operation.****
>
> ** **
>
> so you could end up in situation where dirty write operation returns an
> error, and in a transaction returns {atomic, ok}****
>
> ** **
>
> > dets:insert(table1, SomeKey, SomeVal).****
>
> {error,{{bad_object,read_buckets},****
>
>         "/tmp/table1.DAT"}}****
>
> > mnesia:dirty_write(SomeRecord).****
>
> {error,{{bad_object,read_buckets},****
>
>         "/tmp/table1.DAT"}}****
>
> > mnesia:transaction(fun()-> mnesia:write(SomeRecord) end).  ****
>
> {atomic,ok}****
>
> ** **
>
> You could try to detect that in your transaction by adding an operation
> that will fail (like read, first, last, ..etc) but that won't protect you
> against corrupting the file.****
>
> ** **
>
> ** **
>
> The question is, should mnesia detect this kind of errors and if yes, what
> to do about it?****
>
> Also, can dets protect against the corruption somehow? (reject the
> operation that could lead to size> 2GB?)****
>
> ** **
>
> ** **
>
> Best Regards, ****
>
> Ahmed****
>
> ** **
>
> ** **
>
> On Tue, Aug 27, 2013 at 5:06 PM, Janos Hary <janos.n.hary@REDACTED>
> wrote:****
>
> Thanks, but no luck. It behaves the same with sync_transaction.
>
> Janos****
>
>
> -----Original Message-----
> From: hawk.mattsson@REDACTED [mailto:hawk.mattsson@REDACTED] On Behalf
> Of
> Håkan Mattsson
> Sent: Tuesday, August 27, 2013 4:02 PM
> To: Janos Hary
> Cc: erlang-questions
> Subject: Re: [erlang-questions] mnesia silent failure and table corruption
>
> On Tue, Aug 27, 2013 at 3:05 PM, Janos Hary <janos.n.hary@REDACTED>
> wrote:
> > All,
> >
> > I started to write records into a table from a simple test function.
> > When the table size reaches 2GB write transactions still report
> > success, but they are silently ignored. The table became unusable,
> > read operations return error, and mnesia cannot recover the table. How
> > shall I avoid such situation?
>
> Try using mnesia:sync_transaction() instead of mnesia:transaction().
> Then Mnesia will use syncronized calls to disk_log (and syncronized commits
> when several nodes are involved).
>
> /Håkan
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions****
>
> ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130904/86c08ad6/attachment.htm>