<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Courier New";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Ahmed,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Thanks for the detailed explanation. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>As per <a href="http://www.erlang.org/doc/apps/mnesia/Mnesia_chap4.html">http://www.erlang.org/doc/apps/mnesia/Mnesia_chap4.html</a> :<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>“Durability. This transaction property ensures that changes made to the DBMS by a transaction are permanent. Once a transaction has been committed, all changes made to the database are durable - i.e. they are written safely to disc and will not be corrupted or disappear.”<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>I would definitely expect {aborted, Reason} in case of a storage operation failure. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Since then I found another occurrence of this problem. If I’m precautious and put my data in fragmented table, but have a secondary index, that index will still kept in a single dets file. In my case I tried to index SHA hashes and the index grows pretty fast.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Thanks again for your answer, even if it was a bad news for me. Mnesia looks to be a great database, but I think it was a mistake from my side to try using it as a complete RDBMS replacement.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Best Regards,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Tamas<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Ahmed Omar [mailto:spawn.think@gmail.com] <br><b>Sent:</b> Wednesday, September 04, 2013 2:25 PM<br><b>To:</b> Janos Hary<br><b>Cc:</b> erlang-questions; Håkan Mattsson<br><b>Subject:</b> Re: [erlang-questions] mnesia silent failure and table corruption<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hi Janos, <o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Indeed, in that case transactions could be considered succeeded while actual storage operation failed...<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>The thing is, If mnesia needs to write data to a store (in this case dets), it doesn't check for results for this operation. Actually, it does even catch exceptions. (mnesia_tm:do_update/4)<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>As dets have a maximum of 2GB for file size, I believe the table gets corrupted after that. (somebody correct me please)<o:p></o:p></p></div><div><p class=MsoNormal>So any operation on that dets table will return the bad_object error.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>So the commit is considered succeeded, mnesia logs it, transaction returns {atomic, ok}<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>sync_transaction won't help, because it just waits for the commit to be synced/logged, but also doesn't check results of the store operation.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>so you could end up in situation where dirty write operation returns an error, and in a transaction returns {atomic, ok}<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>> dets:insert(table1, SomeKey, SomeVal).<o:p></o:p></p></div><div><p class=MsoNormal>{error,{{bad_object,read_buckets},<o:p></o:p></p></div><div><p class=MsoNormal> "/tmp/table1.DAT"}}<o:p></o:p></p></div></div><div><div><p class=MsoNormal>> mnesia:dirty_write(SomeRecord).<o:p></o:p></p></div><div><p class=MsoNormal>{error,{{bad_object,read_buckets},<o:p></o:p></p></div><div><p class=MsoNormal> "/tmp/table1.DAT"}}<o:p></o:p></p></div><div><p class=MsoNormal>> mnesia:transaction(fun()-> mnesia:write(SomeRecord) end). <o:p></o:p></p></div><div><p class=MsoNormal>{atomic,ok}<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>You could try to detect that in your transaction by adding an operation that will fail (like read, first, last, ..etc) but that won't protect you against corrupting the file.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>The question is, should mnesia detect this kind of errors and if yes, what to do about it?<o:p></o:p></p></div><div><p class=MsoNormal>Also, can dets protect against the corruption somehow? (reject the operation that could lead to size> 2GB?)<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Best Regards, <o:p></o:p></p></div><div><p class=MsoNormal>Ahmed<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p><div><p class=MsoNormal>On Tue, Aug 27, 2013 at 5:06 PM, Janos Hary <<a href="mailto:janos.n.hary@gmail.com" target="_blank">janos.n.hary@gmail.com</a>> wrote:<o:p></o:p></p><p class=MsoNormal>Thanks, but no luck. It behaves the same with sync_transaction.<br><span style='color:#888888'><br><span class=hoenzb>Janos</span></span><o:p></o:p></p><div><div><p class=MsoNormal><br>-----Original Message-----<br>From: <a href="mailto:hawk.mattsson@gmail.com">hawk.mattsson@gmail.com</a> [mailto:<a href="mailto:hawk.mattsson@gmail.com">hawk.mattsson@gmail.com</a>] On Behalf Of<br>Håkan Mattsson<br>Sent: Tuesday, August 27, 2013 4:02 PM<br>To: Janos Hary<br>Cc: erlang-questions<br>Subject: Re: [erlang-questions] mnesia silent failure and table corruption<br><br>On Tue, Aug 27, 2013 at 3:05 PM, Janos Hary <<a href="mailto:janos.n.hary@gmail.com">janos.n.hary@gmail.com</a>> wrote:<br>> All,<br>><br>> I started to write records into a table from a simple test function.<br>> When the table size reaches 2GB write transactions still report<br>> success, but they are silently ignored. The table became unusable,<br>> read operations return error, and mnesia cannot recover the table. How<br>> shall I avoid such situation?<br><br>Try using mnesia:sync_transaction() instead of mnesia:transaction().<br>Then Mnesia will use syncronized calls to disk_log (and syncronized commits<br>when several nodes are involved).<br><br>/Håkan<br><br>_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><o:p></o:p></p></div></div></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>