[erlang-questions] How many fragments would be proper for1milions records in mensia?

[Valentin Micic]

We used to have up to 200 million records (+200 million for indices,
therefore 400 million) in a production system, spread over 896 fragments
(768 for data and 128 for indices (*)) and that worked reasonably well (all
of that on a single node) - the tested limit on that particular system was
more than 2*250 million records. 

 

Eventually, due to an increase in performance requirements (750 million
rows/records and significantly higher transaction rates) we decided to drop
MNESIA in subsequent releases of the system (and no, we did *not* replace it
with any of commercially available RDBMSes). Mind you, the main reason was
really considerable skepticism expressed in this forum regarding suitability
of MNESIA for large data volumes (**).

 

At any rate, I cannot tell you how big the largest fragment was (not in
terms of bytes anyway), but we did not store more than 300,000 records per
fragment. Our performance requirement was to service 100 transactions per
second with sub 100 ms response time. These criteria were compromised
whenever we approached this load per fragment -- response time would
increase to 150-180 ms, or even more when we had lots of deletions).

 

 

V.

 

(*) Please note that standard MNESIA indexing is not recommended for
fragmented table. Rather, as Ulf suggested once, use another (first-class)
mensia table to do indexing yourself.

 

(**) Truth be told, I do not think that we would have even consider MNESIA
if we were exposed to these kind of views before we actually developed the
system. Fortunately, we did not, thus, we had two very successful system
releases before we abandon the MNESIA, and by that time, we were quite
capable of developing a custom storage mechanism that would satisfy new
requirements. What I'm trying to say is: even if it is not the best
long-term choice, MNESIA can be used to buy you some valuable time while
you're still figuring out what (and how) you're trying to do (it).

 

 

  _____  

From: erlang-questions-bounces@REDACTED
[mailto:erlang-questions-bounces@REDACTED] On Behalf Of devdoer bird
Sent: 28 July 2008 12:28 PM
To: Valentin Micic
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] How many fragments would be proper
for1milions records in mensia?

 

 

2008/7/28, Valentin Micic <valentin@REDACTED>: 

I'd say that also depends on what storage method you're using. Thus, in a
case you're planning to use DETS (disc_only_copy), you may want to consider
fragmentation even for 1 million... In my experience, the performance
deteriorates(*) when table gets populated beyond, say, 250,000 entries (if
entries are reasonably complex terms).

 

Oh,that will be a problem.Can I say mnesia is not proper for very large
data-set?

If I keep every fragment 250,000 records, then 100 million records will be
distrubted in 

400 fragments,is  it a problem for mnesia?

What's the size of your largest mnesia table in your application ?
 

In addition, if you increase a number of dets fragments, make sure that you
also increase a number of I/O threads (+A) to some reasonable value(**).

V.

(*) Inserts and/or updates are affected, read seems to keep good performance
levels.

(**) If you are using 32-bit environment, try not to get carried away with a
number of I/O threads, as the cost in memory may be substantial.

----- Original Message ----- From: "Ulf Wiger" <ulf@REDACTED>
To: "devdoer bird" <devdoer2@REDACTED>; <erlang-questions@REDACTED>
Sent: Sunday, July 27, 2008 10:12 AM
Subject: Re: [erlang-questions] How many fragments would be proper for
1milions records in mensia?



Depending on object size, you may well have just a single
table/fragment. 1 million objects is not that much.

BR,
Ulf W

2008/7/27, devdoer bird <devdoer2@REDACTED>:

Hi:

Considering the access  performance , easy mangement ,how many fragments
would be proper for 1 milions records in mensia? Are there any rules about
the number of fragments and the number of total records?

Regards

_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED
http://www.erlang.org/mailman/listinfo/erlang-questions

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20080728/be42cce5/attachment.htm>