<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>We used to have up to 200 million records (+200
million for indices, therefore 400 million) in a production system, spread over
896 fragments (768 for data and 128 for indices (*)) and that worked reasonably
well (all of that on a single node) – the tested limit on that particular
system was more than 2*250 million records. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Eventually, due to an increase in performance
requirements (750 million rows/records and significantly higher transaction
rates) we decided to drop MNESIA in subsequent releases of the system (and no,
we did *<b><span style='font-weight:bold'>not</span></b>* replace it with any
of commercially available RDBMSes). Mind you, the main reason was really
considerable skepticism expressed in this forum regarding suitability of MNESIA
for large data volumes (**).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>At any rate, I cannot tell you how big the
largest fragment was (not in terms of bytes anyway), but we did not store more
than 300,000 records per fragment. Our performance requirement was to service
100 transactions per second with sub 100 ms response time. These criteria were
compromised whenever we approached this load per fragment -- response time
would increase to 150-180 ms, or even more when we had lots of deletions).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>V.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>(*) Please note that standard MNESIA indexing
is not recommended for fragmented table. Rather, as Ulf suggested once, use another
(first-class) mensia table to do indexing yourself.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>(**) Truth be told, I do not think that we
would have even consider MNESIA if we were exposed to these kind of views
before we actually developed the system. Fortunately, we did not, thus, we had
two very successful system releases before we abandon the MNESIA, and by that
time, we were quite capable of developing a custom storage mechanism that would
satisfy new requirements. What I’m trying to say is: even if it is not
the best long-term choice, MNESIA can be used to buy you some valuable time
while you’re still figuring out what (and how) you’re trying to do
(it).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>
erlang-questions-bounces@erlang.org
[mailto:erlang-questions-bounces@erlang.org] <b><span style='font-weight:bold'>On
Behalf Of </span></b>devdoer bird<br>
<b><span style='font-weight:bold'>Sent:</span></b> 28 July 2008 12:28 PM<br>
<b><span style='font-weight:bold'>To:</span></b> Valentin Micic<br>
<b><span style='font-weight:bold'>Cc:</span></b> erlang-questions@erlang.org<br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [erlang-questions]
How many fragments would be proper for1milions records in mensia?</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'><o:p> </o:p></span></font></p>
<div>
<p class=MsoNormal><span class=gmailquote><font size=3 face="Times New Roman"><span
style='font-size:12.0pt'>2008/7/28, Valentin Micic <<a
href="mailto:valentin@pixie.co.za">valentin@pixie.co.za</a>>:</span></font></span>
<o:p></o:p></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>I'd say that also depends on what storage method you're using. Thus, in
a case you're planning to use DETS (disc_only_copy), you may want to consider
fragmentation even for 1 million... In my experience, the performance
deteriorates(*) when table gets populated beyond, say, 250,000 entries (if
entries are reasonably complex terms).<o:p></o:p></span></font></p>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>Oh,that will be a problem.Can I say mnesia is not proper for very large
data-set?<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>If I keep every fragment 250,000 records, then 100 million records will
be distrubted in <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>400 fragments,is it a problem for mnesia?<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>What's the size of your largest mnesia table in your application ?<br>
<o:p></o:p></span></font></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>In addition, if you
increase a number of dets fragments, make sure that you also increase a number
of I/O threads (+A) to some reasonable value(**).<br>
<br>
V.<br>
<br>
(*) Inserts and/or updates are affected, read seems to keep good performance levels.<br>
<br>
(**) If you are using 32-bit environment, try not to get carried away with a
number of I/O threads, as the cost in memory may be substantial.<br>
<br>
----- Original Message ----- From: "Ulf Wiger" <<a
href="mailto:ulf@wiger.net" target="_blank">ulf@wiger.net</a>><br>
To: "devdoer bird" <<a href="mailto:devdoer2@gmail.com"
target="_blank">devdoer2@gmail.com</a>>; <<a
href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><br>
Sent: Sunday, July 27, 2008 10:12 AM<br>
Subject: Re: [erlang-questions] How many fragments would be proper for 1milions
records in mensia?<br>
<br>
<o:p></o:p></span></font></p>
<div><span id="q_11b68c827f9acfca_1">
<p class=MsoNormal><span class=e><font size=3 face="Times New Roman"><span
style='font-size:12.0pt'>Depending on object size, you may well have just a
single</span></font></span><br>
<span class=e>table/fragment. 1 million objects is not that much.</span><br>
<br>
<span class=e>BR,</span><br>
<span class=e>Ulf W</span><br>
<br>
<span class=e>2008/7/27, devdoer bird <<a href="mailto:devdoer2@gmail.com"
target="_blank">devdoer2@gmail.com</a>>:<o:p></o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>Hi:<br>
<br>
Considering the access performance , easy mangement ,how many fragments<br>
would be proper for 1 milions records in mensia? Are there any rules about<br>
the number of fragments and the number of total records?<br>
<br>
Regards<o:p></o:p></span></font></p>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><span class=q></span><font
size=3 face="Times New Roman"><span style='font-size:12.0pt'>_______________________________________________</span></font></span><br>
<span class=q>erlang-questions mailing list</span><br>
<span class=q><a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a></span><br>
<span class=q><a href="http://www.erlang.org/mailman/listinfo/erlang-questions"
target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a></span><o:p></o:p></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</blockquote>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>