[erlang-questions] How many fragments would be proper for1milions records in mensia?

[devdoer bird]

How can you upgrade the record schema when your fragment is  large? The
cocurrent r/w will be blocked for a long time without stopping
service. 2008/7/29,
Paul Mineiro <paul-trapexit@REDACTED>:
>
> As another opinion, arguing against a large number of fragments:
>
> We have some large dbs as well, and we started with a large number of
> fragments (initially: 1260) but have actually dropped the number of
> fragments over time (currently: 12).  We're using EC2 as we found that
> introducing a new node or removing an old one went faster with less
> fragments.  Also we have several dbs like this and it was more challenging
> to inspect mnesia's state when the total number of tables in the system
> was circa 10000.  Finally, with dets there is some memory overhead per
> fragment that we never completely understood (the "buddy system"), but
> we found less fragments meant less memory usage.
>
> One disadvantage of fewer fragments is that, since it is relatively cheap
> to move fragments versus repartition, starting with a larger number of
> fragments allows for relatively straightforward load balancing when
> introducing new nodes, until you get to the point where you have a
> fragment for each node.  However, lately we've been running the bigger EC2
> instance types and using tcerl for the backend and so our node count has
> been reduced from 24 to 6, thus we could afford to reduce the number of
> fragments.
>
> -- p
>
> On Mon, 28 Jul 2008, Valentin Micic wrote:
>
> > We used to have up to 200 million records (+200 million for indices,
> > therefore 400 million) in a production system, spread over 896 fragments
> > (768 for data and 128 for indices (*)) and that worked reasonably well
> (all
> > of that on a single node) - the tested limit on that particular system
> was
> > more than 2*250 million records.
> >
> >
> >
> > Eventually, due to an increase in performance requirements (750 million
> > rows/records and significantly higher transaction rates) we decided to
> drop
> > MNESIA in subsequent releases of the system (and no, we did *not* replace
> it
> > with any of commercially available RDBMSes). Mind you, the main reason
> was
> > really considerable skepticism expressed in this forum regarding
> suitability
> > of MNESIA for large data volumes (**).
> >
> >
> >
> > At any rate, I cannot tell you how big the largest fragment was (not in
> > terms of bytes anyway), but we did not store more than 300,000 records
> per
> > fragment. Our performance requirement was to service 100 transactions per
> > second with sub 100 ms response time. These criteria were compromised
> > whenever we approached this load per fragment -- response time would
> > increase to 150-180 ms, or even more when we had lots of deletions).
> >
> >
> >
> >
> >
> > V.
> >
> >
> >
> > (*) Please note that standard MNESIA indexing is not recommended for
> > fragmented table. Rather, as Ulf suggested once, use another
> (first-class)
> > mensia table to do indexing yourself.
> >
> >
> >
> > (**) Truth be told, I do not think that we would have even consider
> MNESIA
> > if we were exposed to these kind of views before we actually developed
> the
> > system. Fortunately, we did not, thus, we had two very successful system
> > releases before we abandon the MNESIA, and by that time, we were quite
> > capable of developing a custom storage mechanism that would satisfy new
> > requirements. What I'm trying to say is: even if it is not the best
> > long-term choice, MNESIA can be used to buy you some valuable time while
> > you're still figuring out what (and how) you're trying to do (it).
> >
> >
> >
> >
> >
> >   _____
> >
> > From: erlang-questions-bounces@REDACTED
> > [mailto:erlang-questions-bounces@REDACTED] On Behalf Of devdoer bird
> > Sent: 28 July 2008 12:28 PM
> > To: Valentin Micic
> > Cc: erlang-questions@REDACTED
> > Subject: Re: [erlang-questions] How many fragments would be proper
> > for1milions records in mensia?
> >
> >
> >
> >
> >
> > 2008/7/28, Valentin Micic <valentin@REDACTED>:
> >
> > I'd say that also depends on what storage method you're using. Thus, in a
> > case you're planning to use DETS (disc_only_copy), you may want to
> consider
> > fragmentation even for 1 million... In my experience, the performance
> > deteriorates(*) when table gets populated beyond, say, 250,000 entries
> (if
> > entries are reasonably complex terms).
> >
> >
> >
> > Oh,that will be a problem.Can I say mnesia is not proper for very large
> > data-set?
> >
> > If I keep every fragment 250,000 records, then 100 million records will
> be
> > distrubted in
> >
> > 400 fragments,is  it a problem for mnesia?
> >
> > What's the size of your largest mnesia table in your application ?
> >
> >
> > In addition, if you increase a number of dets fragments, make sure that
> you
> > also increase a number of I/O threads (+A) to some reasonable value(**).
> >
> > V.
> >
> > (*) Inserts and/or updates are affected, read seems to keep good
> performance
> > levels.
> >
> > (**) If you are using 32-bit environment, try not to get carried away
> with a
> > number of I/O threads, as the cost in memory may be substantial.
> >
> > ----- Original Message ----- From: "Ulf Wiger" <ulf@REDACTED>
> > To: "devdoer bird" <devdoer2@REDACTED>; <erlang-questions@REDACTED>
> > Sent: Sunday, July 27, 2008 10:12 AM
> > Subject: Re: [erlang-questions] How many fragments would be proper for
> > 1milions records in mensia?
> >
> >
> >
> > Depending on object size, you may well have just a single
> > table/fragment. 1 million objects is not that much.
> >
> > BR,
> > Ulf W
> >
> > 2008/7/27, devdoer bird <devdoer2@REDACTED>:
> >
> > Hi:
> >
> > Considering the access  performance , easy mangement ,how many fragments
> > would be proper for 1 milions records in mensia? Are there any rules
> about
> > the number of fragments and the number of total records?
> >
> > Regards
> >
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://www.erlang.org/mailman/listinfo/erlang-questions
> >
> >
> >
> >
> >
> >
>
> In an artificial world, only extremists live naturally.
>
>        -- Paul Graham
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20080729/beefdfde/attachment.htm>