[erlang-questions] How many fragments would be proper for1milions records in mensia?

[Paul Mineiro]

As another opinion, arguing against a large number of fragments:

We have some large dbs as well, and we started with a large number of
fragments (initially: 1260) but have actually dropped the number of
fragments over time (currently: 12).  We're using EC2 as we found that
introducing a new node or removing an old one went faster with less
fragments.  Also we have several dbs like this and it was more challenging
to inspect mnesia's state when the total number of tables in the system
was circa 10000.  Finally, with dets there is some memory overhead per
fragment that we never completely understood (the "buddy system"), but
we found less fragments meant less memory usage.

One disadvantage of fewer fragments is that, since it is relatively cheap
to move fragments versus repartition, starting with a larger number of
fragments allows for relatively straightforward load balancing when
introducing new nodes, until you get to the point where you have a
fragment for each node.  However, lately we've been running the bigger EC2
instance types and using tcerl for the backend and so our node count has
been reduced from 24 to 6, thus we could afford to reduce the number of
fragments.

-- p

On Mon, 28 Jul 2008, Valentin Micic wrote:

> We used to have up to 200 million records (+200 million for indices,
> therefore 400 million) in a production system, spread over 896 fragments
> (768 for data and 128 for indices (*)) and that worked reasonably well (all
> of that on a single node) - the tested limit on that particular system was
> more than 2*250 million records.
>
>
>
> Eventually, due to an increase in performance requirements (750 million
> rows/records and significantly higher transaction rates) we decided to drop
> MNESIA in subsequent releases of the system (and no, we did *not* replace it
> with any of commercially available RDBMSes). Mind you, the main reason was
> really considerable skepticism expressed in this forum regarding suitability
> of MNESIA for large data volumes (**).
>
>
>
> At any rate, I cannot tell you how big the largest fragment was (not in
> terms of bytes anyway), but we did not store more than 300,000 records per
> fragment. Our performance requirement was to service 100 transactions per
> second with sub 100 ms response time. These criteria were compromised
> whenever we approached this load per fragment -- response time would
> increase to 150-180 ms, or even more when we had lots of deletions).
>
>
>
>
>
> V.
>
>
>
> (*) Please note that standard MNESIA indexing is not recommended for
> fragmented table. Rather, as Ulf suggested once, use another (first-class)
> mensia table to do indexing yourself.
>
>
>
> (**) Truth be told, I do not think that we would have even consider MNESIA
> if we were exposed to these kind of views before we actually developed the
> system. Fortunately, we did not, thus, we had two very successful system
> releases before we abandon the MNESIA, and by that time, we were quite
> capable of developing a custom storage mechanism that would satisfy new
> requirements. What I'm trying to say is: even if it is not the best
> long-term choice, MNESIA can be used to buy you some valuable time while
> you're still figuring out what (and how) you're trying to do (it).
>
>
>
>
>
>   _____
>
> From: erlang-questions-bounces@REDACTED
> [mailto:erlang-questions-bounces@REDACTED] On Behalf Of devdoer bird
> Sent: 28 July 2008 12:28 PM
> To: Valentin Micic
> Cc: erlang-questions@REDACTED
> Subject: Re: [erlang-questions] How many fragments would be proper
> for1milions records in mensia?
>
>
>
>
>
> 2008/7/28, Valentin Micic <valentin@REDACTED>:
>
> I'd say that also depends on what storage method you're using. Thus, in a
> case you're planning to use DETS (disc_only_copy), you may want to consider
> fragmentation even for 1 million... In my experience, the performance
> deteriorates(*) when table gets populated beyond, say, 250,000 entries (if
> entries are reasonably complex terms).
>
>
>
> Oh,that will be a problem.Can I say mnesia is not proper for very large
> data-set?
>
> If I keep every fragment 250,000 records, then 100 million records will be
> distrubted in
>
> 400 fragments,is  it a problem for mnesia?
>
> What's the size of your largest mnesia table in your application ?
>
>
> In addition, if you increase a number of dets fragments, make sure that you
> also increase a number of I/O threads (+A) to some reasonable value(**).
>
> V.
>
> (*) Inserts and/or updates are affected, read seems to keep good performance
> levels.
>
> (**) If you are using 32-bit environment, try not to get carried away with a
> number of I/O threads, as the cost in memory may be substantial.
>
> ----- Original Message ----- From: "Ulf Wiger" <ulf@REDACTED>
> To: "devdoer bird" <devdoer2@REDACTED>; <erlang-questions@REDACTED>
> Sent: Sunday, July 27, 2008 10:12 AM
> Subject: Re: [erlang-questions] How many fragments would be proper for
> 1milions records in mensia?
>
>
>
> Depending on object size, you may well have just a single
> table/fragment. 1 million objects is not that much.
>
> BR,
> Ulf W
>
> 2008/7/27, devdoer bird <devdoer2@REDACTED>:
>
> Hi:
>
> Considering the access  performance , easy mangement ,how many fragments
> would be proper for 1 milions records in mensia? Are there any rules about
> the number of fragments and the number of total records?
>
> Regards
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
>
>
>
>
>

In an artificial world, only extremists live naturally.

        -- Paul Graham