[erlang-questions] mnesia silent failure and table corruption

[Janos Hary]

Ahmed,

 

Thanks for the detailed explanation. 

 

As per http://www.erlang.org/doc/apps/mnesia/Mnesia_chap4.html :

“Durability. This transaction property ensures that changes made to the DBMS
by a transaction are permanent. Once a transaction has been committed, all
changes made to the database are durable - i.e. they are written safely to
disc and will not be corrupted or disappear.”

 

I would definitely expect {aborted, Reason} in case of a storage operation
failure. 

 

Since then I found another occurrence of this problem. If I’m precautious
and put my data in fragmented table, but have a secondary index, that index
will still kept in a single dets file. In my case I tried to index SHA
hashes and the index grows pretty fast.

 

Thanks again for your answer, even if it was a bad news for me. Mnesia looks
to be a great database, but I think it was a mistake from my side to try
using it as a complete RDBMS replacement.

 

Best Regards,

Tamas

 

From: Ahmed Omar [mailto:spawn.think@REDACTED] 
Sent: Wednesday, September 04, 2013 2:25 PM
To: Janos Hary
Cc: erlang-questions; Håkan Mattsson
Subject: Re: [erlang-questions] mnesia silent failure and table corruption

 

Hi Janos, 

 

Indeed, in that case transactions could be considered succeeded while actual
storage operation failed...

 

The thing is, If mnesia needs to write data to a store (in this case dets),
it doesn't check for results for this operation. Actually, it does even
catch exceptions. (mnesia_tm:do_update/4)

 

As dets have a maximum of 2GB for file size, I believe the table gets
corrupted after that. (somebody correct me please)

So any operation on that dets table will return the bad_object error.

 

So the commit is considered succeeded, mnesia logs it, transaction returns
{atomic, ok}

 

sync_transaction won't help, because it just waits for the commit to be
synced/logged, but also doesn't check results of the store operation.

 

so you could end up in situation where dirty write operation returns an
error, and in a transaction returns {atomic, ok}

 

> dets:insert(table1, SomeKey, SomeVal).

{error,{{bad_object,read_buckets},

        "/tmp/table1.DAT"}}

> mnesia:dirty_write(SomeRecord).

{error,{{bad_object,read_buckets},

        "/tmp/table1.DAT"}}

> mnesia:transaction(fun()-> mnesia:write(SomeRecord) end).  

{atomic,ok}

 

You could try to detect that in your transaction by adding an operation that
will fail (like read, first, last, ..etc) but that won't protect you against
corrupting the file.

 

 

The question is, should mnesia detect this kind of errors and if yes, what
to do about it?

Also, can dets protect against the corruption somehow? (reject the operation
that could lead to size> 2GB?)

 

 

Best Regards, 

Ahmed

 

 

On Tue, Aug 27, 2013 at 5:06 PM, Janos Hary <janos.n.hary@REDACTED> wrote:

Thanks, but no luck. It behaves the same with sync_transaction.

Janos


-----Original Message-----
From: hawk.mattsson@REDACTED [mailto:hawk.mattsson@REDACTED] On Behalf Of
Håkan Mattsson
Sent: Tuesday, August 27, 2013 4:02 PM
To: Janos Hary
Cc: erlang-questions
Subject: Re: [erlang-questions] mnesia silent failure and table corruption

On Tue, Aug 27, 2013 at 3:05 PM, Janos Hary <janos.n.hary@REDACTED> wrote:
> All,
>
> I started to write records into a table from a simple test function.
> When the table size reaches 2GB write transactions still report
> success, but they are silently ignored. The table became unusable,
> read operations return error, and mnesia cannot recover the table. How
> shall I avoid such situation?

Try using mnesia:sync_transaction() instead of mnesia:transaction().
Then Mnesia will use syncronized calls to disk_log (and syncronized commits
when several nodes are involved).

/Håkan

_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED
http://erlang.org/mailman/listinfo/erlang-questions

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130904/2120a8a1/attachment.htm>