[erlang-questions] The 2 GB limit

[Ulf Wiger]

Mnesiaex is not supported by OTP. As far as I know,
it is stable.

There are some things to consider...

- If you put significantly more than 2GB into
   each table, mnesia's method of synchronizing
   replicas (copying the entire table over the
   network) may start to hurt.

- Tokyo Cabinet has slightly different recovery
   characteristics than dets. This may make a
   difference in some crash recovery situations.
   That is, TC gets its speed partly from making
   use of memory-mapped IO, so more data is likely
   to get lost in some failure situations. TC is
   supposedly also more sensitive to corrupted files
   than dets, since it doesn't 'repair' the file
   like dets does.

I don't think there are major problems with performance
when using many fragments. You should perhaps do some
measurements first?

It can be a problem that you need to have enough
fragments that you are absolutely sure never to
reach 2 GB in any fragment, since mnesia has no
way of dealing with dets refusing to insert more
records.

BTW, if you use disc_copies (not disc_only_copies),
dets is not used for the persistent storage, and
there is no 2 GB limit per table (only the restriction
that the table must fit in RAM).

BR,
Ulf W


Igor Ribeiro Sucupira wrote:
> On Tue, Nov 24, 2009 at 1:06 PM, Ulf Wiger
> <ulf.wiger@REDACTED> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Igor Ribeiro Sucupira wrote:
>>> Hi.
>>>
>>> Is there any plan (or work in progress) for removing the 2 GB size
>>> limit of dets files?
>>>
>>> And, while this is not accomplished, do you think it could be easier
>>> to try to raise that limit to 4 GB? Assuming the issue is related to
>>> 32-bit addressing, I'm guessing (just guessing) that dealing with 4 GB
>>> files should not be difficult. Am I wrong? Why?
>> Rather than re-writing dets, I would go with something like Tokyo
>> Tyrant, build on a suitable erlang wrapper (like
> 
> Hi.
> 
> I'm running Mnesia with fragmented tables, over a pool of servers.
> Considering the amount of records to be inserted and the size limits
> of dets, I will need a lot of fragments for each table and there seems
> to be a big performance penalty on using many Mnesia fragments (even
> when they don't have so much data).
> 
> The more plausible replacement, then, would be something like tcerl
> with mnesiaex, but I've read somewhere that mnesiaex is still
> experimental.
> I'm dealing with data that affects millions of users, in my current
> project, so I have to be very careful. That was the reason for wanting
> to raise the limits of dets tables.
> 
> Thank you.
> Igor.
> 
>> http://github.com/mallipeddi/tora), and perhaps put something like
>> my sext library on top
>> (http://svn.ulf.wiger.net/sext/trunk/sext/examples/tt_proto.erl).
>>
>> To match dets, I'd like to see some administrative code around the
>> creation and opening of Tyrant tables, including the ability to
>> "repair" the data from the transaction log. Seems doable, though.
>>
>> http://1978th.net/tokyotyrant/
>>
>> BR,
>> Ulf W
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>
>> iEYEARECAAYFAksL9nUACgkQtqqFieqzed1v7wCfUf9bgDU+JfD8Gey51lvR7fxG
>> 3iIAn0J5+8NpHH7DkLTgRlRuUKWomkXO
>> =C66x
>> -----END PGP SIGNATURE-----
> 
> 


-- 
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com