<div>Many thanks for your sharing.</div>
<div><br> </div>
<div><span class="gmail_quote">2008/7/28, Valentin Micic <<a href="mailto:v@pharos-avantgard.com">v@pharos-avantgard.com</a>>:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div lang="EN-US" vlink="blue" link="blue">
<div>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">We used to have up to 200 million records (+200 million for indices, therefore 400 million) in a production system, spread over 896 fragments (768 for data and 128 for indices (*)) and that worked reasonably well (all of that on a single node) ¨C the tested limit on that particular system was more than 2*250 million records. </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Eventually, due to an increase in performance requirements (750 million rows/records and significantly higher transaction rates) we decided to drop MNESIA in subsequent releases of the system (and no, we did *<b><span style="FONT-WEIGHT: bold">not</span></b>* replace it with any of commercially available RDBMSes). Mind you, the main reason was really considerable skepticism expressed in this forum regarding suitability of MNESIA for large data volumes (**).</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">At any rate, I cannot tell you how big the largest fragment was (not in terms of bytes anyway), but we did not store more than 300,000 records per fragment. Our performance requirement was to service 100 transactions per second with sub 100 ms response time. These criteria were compromised whenever we approached this load per fragment -- response time would increase to 150-180 ms, or even more when we had lots of deletions).</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">V.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">(*) Please note that standard MNESIA indexing is not recommended for fragmented table. Rather, as Ulf suggested once, use another (first-class) mensia table to do indexing yourself.</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">(**) Truth be told, I do not think that we would have even consider MNESIA if we were exposed to these kind of views before we actually developed the system. Fortunately, we did not, thus, we had two very successful system releases before we abandon the MNESIA, and by that time, we were quite capable of developing a custom storage mechanism that would satisfy new requirements. What I'm trying to say is: even if it is not the best long-term choice, MNESIA can be used to buy you some valuable time while you're still figuring out what (and how) you're trying to do (it).</span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<p><font face="Arial" color="navy" size="2"><span style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"> </span></font></p>
<div>
<div style="TEXT-ALIGN: center" align="center"><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">
<hr align="center" width="100%" size="2">
</span></font></div>
<p><b><font face="Tahoma" size="2"><span style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">From:</span></font></b><font face="Tahoma" size="2"><span style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> <a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:erlang-questions-bounces@erlang.org" target="_blank">erlang-questions-bounces@erlang.org</a> [mailto:<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:erlang-questions-bounces@erlang.org" target="_blank">erlang-questions-bounces@erlang.org</a>] <b><span style="FONT-WEIGHT: bold">On Behalf Of </span></b>devdoer bird<br>
<b><span style="FONT-WEIGHT: bold">Sent:</span></b> 28 July 2008 12:28 PM<br><b><span style="FONT-WEIGHT: bold">To:</span></b> Valentin Micic<br><b><span style="FONT-WEIGHT: bold">Cc:</span></b> <a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>
<div><span class="e" id="q_11b69b838b3034e9_1"><br><b><span style="FONT-WEIGHT: bold">Subject:</span></b> Re: [erlang-questions] How many fragments would be proper for1milions records in mensia?</span></div></span></font>
<p></p></p></div>
<div><span class="e" id="q_11b69b838b3034e9_3">
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p>
<p style="MARGIN-BOTTOM: 12pt"><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p>
<div>
<p><span><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">2008/7/28, Valentin Micic <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:valentin@pixie.co.za" target="_blank">valentin@pixie.co.za</a>>:</span></font></span> </p>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">I'd say that also depends on what storage method you're using. Thus, in a case you're planning to use DETS (disc_only_copy), you may want to consider fragmentation even for 1 million... In my experience, the performance deteriorates(*) when table gets populated beyond, say, 250,000 entries (if entries are reasonably complex terms).</span></font></p>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Oh,that will be a problem.Can I say mnesia is not proper for very large data-set?</span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">If I keep every fragment 250,000 records, then 100 million records will be distrubted in </span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">400 fragments,is it a problem for mnesia?</span></font></p></div>
<div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">What's the size of your largest mnesia table in your application ?<br> </span></font></p></div>
<blockquote style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: medium none; PADDING-LEFT: 6pt; PADDING-BOTTOM: 0cm; MARGIN-LEFT: 4.8pt; BORDER-LEFT: #cccccc 1pt solid; MARGIN-RIGHT: 0cm; PADDING-TOP: 0cm; BORDER-BOTTOM: medium none">
<p style="MARGIN-BOTTOM: 12pt"><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">In addition, if you increase a number of dets fragments, make sure that you also increase a number of I/O threads (+A) to some reasonable value(**).<br>
<br>V.<br><br>(*) Inserts and/or updates are affected, read seems to keep good performance levels.<br><br>(**) If you are using 32-bit environment, try not to get carried away with a number of I/O threads, as the cost in memory may be substantial.<br>
<br>----- Original Message ----- From: "Ulf Wiger" <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:ulf@wiger.net" target="_blank">ulf@wiger.net</a>><br>To: "devdoer bird" <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:devdoer2@gmail.com" target="_blank">devdoer2@gmail.com</a>>; <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><br>
Sent: Sunday, July 27, 2008 10:12 AM<br>Subject: Re: [erlang-questions] How many fragments would be proper for 1milions records in mensia?<br><br></span></font></p>
<div><span>
<p><span><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Depending on object size, you may well have just a single</span></font></span><br><span>table/fragment. 1 million objects is not that much.</span><br>
<br><span>BR,</span><br><span>Ulf W</span><br><br><span>2008/7/27, devdoer bird <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:devdoer2@gmail.com" target="_blank">devdoer2@gmail.com</a>>:</span></p>
<p style="MARGIN-BOTTOM: 12pt"><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">Hi:<br><br>Considering the access performance , easy mangement ,how many fragments<br>would be proper for 1 milions records in mensia? Are there any rules about<br>
the number of fragments and the number of total records?<br><br>Regards</span></font></p></span></div>
<p style="MARGIN-BOTTOM: 12pt"><span></span><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt">_______________________________________________</span></font><br><span>erlang-questions mailing list</span><br>
<span><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a></span><br><span><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a></span></p>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></blockquote></div>
<p><font face="Times New Roman" size="3"><span style="FONT-SIZE: 12pt"> </span></font></p></span></div></div></div></blockquote></div><br>