[erlang-questions] understanding the scaleability limits of erlang and mnesia

[Igor Ribeiro Sucupira]

You might want to try this patch, which is in the branch of proposed updates:
http://www.erlang.org/cgi-bin/ezmlm-cgi/3/684

If network bandwidth is your botteneck (and not disk I/O nor CPU), it
might help. It would also be a good test for the patch.

Thanks.
Igor.

On Tue, Jan 26, 2010 at 11:40 PM, Brian Acton <acton@REDACTED> wrote:
> Disk and Network activity are good.
>
> In fact, after I made Paul M's change to no_table_loaders (raise to 20), I
> see the network interface pretty much peg out at 12 MB/s. Doing the math,
> this hits 96 Mbs which is pretty good for a 100Mbs interface.
>
> I guess at this point, I've got to go to a gigabit interface to improve
> startup time / throughput ? As it stands, my app takes about 7 minutes to
> startup. 5 of those minutes are lost in mnesia startup.
>
> --b
>
>
> On Tue, Jan 26, 2010 at 2:33 PM, Paul Fisher <pfisher@REDACTED> wrote:
>
>> You misunderstood me... uniqueness is good.  Non-unique keys is bad.
>>
>> And for disk_copies tables it is effectively inserting into ets tables,
>> yes.  Extremely, non-unique keys really slows down loads of tables at
>> startup.  You should be able to test this by monitoring disk and network
>> i/o while it is starting the tables.  If both are low, but while it
>> takes forever to start you table(s), then the keys are probably the
>> problem.
>>
>> On Tue, 2010-01-26 at 16:23 -0600, Brian Acton wrote:
>> > I understand that inserts would take longer in the former example than
>> > in the latter. However, I don't understand why this would so greatly
>> > impact restart / recovery time ? Is this because mnesia is effectively
>> > reinserting the records, rather than blatting the file from another
>> > node over the wire ? Presumably, the latter would be much much
>> > faster...
>> >
>> > Any references (besides what is on erlang.org) that you use that
>> > details this info ?
>> >
>> > Also, rewriting my app to lessen the uniqueness of the indexes is
>> > virtually impossible. I'm dealing with user ids which are inherently
>> > unique.
>> >
>> > --b
>> >
>> >
>> > On Tue, Jan 26, 2010 at 12:50 PM, Paul Fisher <pfisher@REDACTED>
>> > wrote:
>> >         On Tue, 2010-01-26 at 14:19 -0600, Brian Acton wrote:
>> >         > Can you explain more about how index uniqueness affects
>> >         recover /
>> >         > startup times ?
>> >
>> >
>> >         Fundamentally, this comes down to the way that ets tables
>> >         work... Try to
>> >         insert 100000 records with unique index values into a table,
>> >         then try to
>> >         do 100000 records with either the atoms 'true' or 'false'.
>> >          You will see
>> >         the problem in the runtime results.
>> >
>> >         --
>> >         paul
>> >


-- 
"The secret of joy in work is contained in one word - excellence. To
know how to do something well is to enjoy it." - Pearl S. Buck.