[erlang-questions] understanding the scaleability limits of erlang and mnesia

Brian Acton acton@REDACTED
Wed Jan 27 02:40:13 CET 2010


Disk and Network activity are good.

In fact, after I made Paul M's change to no_table_loaders (raise to 20), I
see the network interface pretty much peg out at 12 MB/s. Doing the math,
this hits 96 Mbs which is pretty good for a 100Mbs interface.

I guess at this point, I've got to go to a gigabit interface to improve
startup time / throughput ? As it stands, my app takes about 7 minutes to
startup. 5 of those minutes are lost in mnesia startup.

--b


On Tue, Jan 26, 2010 at 2:33 PM, Paul Fisher <pfisher@REDACTED> wrote:

> You misunderstood me... uniqueness is good.  Non-unique keys is bad.
>
> And for disk_copies tables it is effectively inserting into ets tables,
> yes.  Extremely, non-unique keys really slows down loads of tables at
> startup.  You should be able to test this by monitoring disk and network
> i/o while it is starting the tables.  If both are low, but while it
> takes forever to start you table(s), then the keys are probably the
> problem.
>
> On Tue, 2010-01-26 at 16:23 -0600, Brian Acton wrote:
> > I understand that inserts would take longer in the former example than
> > in the latter. However, I don't understand why this would so greatly
> > impact restart / recovery time ? Is this because mnesia is effectively
> > reinserting the records, rather than blatting the file from another
> > node over the wire ? Presumably, the latter would be much much
> > faster...
> >
> > Any references (besides what is on erlang.org) that you use that
> > details this info ?
> >
> > Also, rewriting my app to lessen the uniqueness of the indexes is
> > virtually impossible. I'm dealing with user ids which are inherently
> > unique.
> >
> > --b
> >
> >
> > On Tue, Jan 26, 2010 at 12:50 PM, Paul Fisher <pfisher@REDACTED>
> > wrote:
> >         On Tue, 2010-01-26 at 14:19 -0600, Brian Acton wrote:
> >         > Can you explain more about how index uniqueness affects
> >         recover /
> >         > startup times ?
> >
> >
> >         Fundamentally, this comes down to the way that ets tables
> >         work... Try to
> >         insert 100000 records with unique index values into a table,
> >         then try to
> >         do 100000 records with either the atoms 'true' or 'false'.
> >          You will see
> >         the problem in the runtime results.
> >
> >         --
> >         paul
> >
> >
>
>
>


More information about the erlang-questions mailing list