[erlang-questions] The 2 GB limit

[Igor Ribeiro Sucupira]

On Fri, Nov 27, 2009 at 9:51 PM, Ulf Wiger
<ulf.wiger@REDACTED> wrote:
>
> I don't think there are major problems with performance
> when using many fragments. You should perhaps do some
> measurements first?

Sorry I was not clear enough. The performance penalty I observed was
more related to the number of fragments PER SERVER.
If I insert a total of 10 GB of data on 500 initially empty fragments,
distributed over 2 servers, and then repeat the experiment with 1000
initially empty fragments (without changing the number of servers nor
the total amount of data), the insertion rate will be smaller (I don't
have the data here, but I think it was something around 20% smaller on
my experiments).

Other people in my company have conducted similar experiments (for
completely independent projects) and have arrived at the same general
conclusion: try to keep the number of fragments per server as small as
possible.

> It can be a problem that you need to have enough
> fragments that you are absolutely sure never to
> reach 2 GB in any fragment, since mnesia has no
> way of dealing with dets refusing to insert more
> records.

I'm aware of that. I need to create the pool with enough fragments
that I will not need to add more in the future (adding fragments in
that situation will certainly be a very heavy operation), but, at the
same time, I can't just "start with 100 thousand fragments" (:-)), as
I thought in the beginning.
That is an important trade-off you have to consider when deciding to
use Mnesia's fragmented tables in disc_only_copies.

> BTW, if you use disc_copies (not disc_only_copies),
> dets is not used for the persistent storage, and
> there is no 2 GB limit per table (only the restriction
> that the table must fit in RAM).

I've read about that. But having to keep all my data in RAM would
increase so much the cost of the project that it would be much wiser
to give up Mnesia.

Thank you.
Igor.

> BR,
> Ulf W
>
>
> Igor Ribeiro Sucupira wrote:
>>
>> On Tue, Nov 24, 2009 at 1:06 PM, Ulf Wiger
>> <ulf.wiger@REDACTED> wrote:
>>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Igor Ribeiro Sucupira wrote:
>>>>
>>>> Hi.
>>>>
>>>> Is there any plan (or work in progress) for removing the 2 GB size
>>>> limit of dets files?
>>>>
>>>> And, while this is not accomplished, do you think it could be easier
>>>> to try to raise that limit to 4 GB? Assuming the issue is related to
>>>> 32-bit addressing, I'm guessing (just guessing) that dealing with 4 GB
>>>> files should not be difficult. Am I wrong? Why?
>>>
>>> Rather than re-writing dets, I would go with something like Tokyo
>>> Tyrant, build on a suitable erlang wrapper (like
>>
>> Hi.
>>
>> I'm running Mnesia with fragmented tables, over a pool of servers.
>> Considering the amount of records to be inserted and the size limits
>> of dets, I will need a lot of fragments for each table and there seems
>> to be a big performance penalty on using many Mnesia fragments (even
>> when they don't have so much data).
>>
>> The more plausible replacement, then, would be something like tcerl
>> with mnesiaex, but I've read somewhere that mnesiaex is still
>> experimental.
>> I'm dealing with data that affects millions of users, in my current
>> project, so I have to be very careful. That was the reason for wanting
>> to raise the limits of dets tables.
>>
>> Thank you.
>> Igor.
>>
>>> http://github.com/mallipeddi/tora), and perhaps put something like
>>> my sext library on top
>>> (http://svn.ulf.wiger.net/sext/trunk/sext/examples/tt_proto.erl).
>>>
>>> To match dets, I'd like to see some administrative code around the
>>> creation and opening of Tyrant tables, including the ability to
>>> "repair" the data from the transaction log. Seems doable, though.
>>>
>>> http://1978th.net/tokyotyrant/
>>>
>>> BR,
>>> Ulf W
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.9 (GNU/Linux)
>>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>>
>>> iEYEARECAAYFAksL9nUACgkQtqqFieqzed1v7wCfUf9bgDU+JfD8Gey51lvR7fxG
>>> 3iIAn0J5+8NpHH7DkLTgRlRuUKWomkXO
>>> =C66x
>>> -----END PGP SIGNATURE-----
>>
>>
>
>
> --
> Ulf Wiger
> CTO, Erlang Training & Consulting Ltd
> http://www.erlang-consulting.com
>



-- 
"The secret of joy in work is contained in one word - excellence. To
know how to do something well is to enjoy it." - Pearl S. Buck.