[erlang-questions] How many fragments would be proper for1milions records in mensia?

[devdoer bird]

Many thanks for your sharing.


2008/7/28, Valentin Micic <v@REDACTED>:
>
>  We used to have up to 200 million records (+200 million for indices,
> therefore 400 million) in a production system, spread over 896 fragments
> (768 for data and 128 for indices (*)) and that worked reasonably well (all
> of that on a single node) ¨C the tested limit on that particular system was
> more than 2*250 million records.
>
>
>
> Eventually, due to an increase in performance requirements (750 million
> rows/records and significantly higher transaction rates) we decided to drop
> MNESIA in subsequent releases of the system (and no, we did **not**
> replace it with any of commercially available RDBMSes). Mind you, the main
> reason was really considerable skepticism expressed in this forum regarding
> suitability of MNESIA for large data volumes (**).
>
>
>
> At any rate, I cannot tell you how big the largest fragment was (not in
> terms of bytes anyway), but we did not store more than 300,000 records per
> fragment. Our performance requirement was to service 100 transactions per
> second with sub 100 ms response time. These criteria were compromised
> whenever we approached this load per fragment -- response time would
> increase to 150-180 ms, or even more when we had lots of deletions).
>
>
>
>
>
> V.
>
>
>
> (*) Please note that standard MNESIA indexing is not recommended for
> fragmented table. Rather, as Ulf suggested once, use another (first-class)
> mensia table to do indexing yourself.
>
>
>
> (**) Truth be told, I do not think that we would have even consider MNESIA
> if we were exposed to these kind of views before we actually developed the
> system. Fortunately, we did not, thus, we had two very successful system
> releases before we abandon the MNESIA, and by that time, we were quite
> capable of developing a custom storage mechanism that would satisfy new
> requirements. What I'm trying to say is: even if it is not the best
> long-term choice, MNESIA can be used to buy you some valuable time while
> you're still figuring out what (and how) you're trying to do (it).
>
>
>
>
>  ------------------------------
>
> *From:* erlang-questions-bounces@REDACTED [mailto:
> erlang-questions-bounces@REDACTED] *On Behalf Of *devdoer bird
> *Sent:* 28 July 2008 12:28 PM
> *To:* Valentin Micic
> *Cc:* erlang-questions@REDACTED
> *Subject:* Re: [erlang-questions] How many fragments would be proper
> for1milions records in mensia?
>
>
>
>
>
> 2008/7/28, Valentin Micic <valentin@REDACTED>:
>
> I'd say that also depends on what storage method you're using. Thus, in a
> case you're planning to use DETS (disc_only_copy), you may want to consider
> fragmentation even for 1 million... In my experience, the performance
> deteriorates(*) when table gets populated beyond, say, 250,000 entries (if
> entries are reasonably complex terms).
>
>
>
> Oh,that will be a problem.Can I say mnesia is not proper for very large
> data-set?
>
> If I keep every fragment 250,000 records, then 100 million records will be
> distrubted in
>
> 400 fragments,is  it a problem for mnesia?
>
> What's the size of your largest mnesia table in your application ?
>
>
> In addition, if you increase a number of dets fragments, make sure that you
> also increase a number of I/O threads (+A) to some reasonable value(**).
>
> V.
>
> (*) Inserts and/or updates are affected, read seems to keep good
> performance levels.
>
> (**) If you are using 32-bit environment, try not to get carried away with
> a number of I/O threads, as the cost in memory may be substantial.
>
> ----- Original Message ----- From: "Ulf Wiger" <ulf@REDACTED>
> To: "devdoer bird" <devdoer2@REDACTED>; <erlang-questions@REDACTED>
> Sent: Sunday, July 27, 2008 10:12 AM
> Subject: Re: [erlang-questions] How many fragments would be proper for
> 1milions records in mensia?
>
>  Depending on object size, you may well have just a single
> table/fragment. 1 million objects is not that much.
>
> BR,
> Ulf W
>
> 2008/7/27, devdoer bird <devdoer2@REDACTED>:
>
> Hi:
>
> Considering the access  performance , easy mangement ,how many fragments
> would be proper for 1 milions records in mensia? Are there any rules about
> the number of fragments and the number of total records?
>
> Regards
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20080728/ed25324e/attachment.htm>