[erlang-questions] The 2 GB limit

[Igor Ribeiro Sucupira]

On Tue, Nov 24, 2009 at 1:06 PM, Ulf Wiger
<ulf.wiger@REDACTED> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Igor Ribeiro Sucupira wrote:
>> Hi.
>>
>> Is there any plan (or work in progress) for removing the 2 GB size
>> limit of dets files?
>>
>> And, while this is not accomplished, do you think it could be easier
>> to try to raise that limit to 4 GB? Assuming the issue is related to
>> 32-bit addressing, I'm guessing (just guessing) that dealing with 4 GB
>> files should not be difficult. Am I wrong? Why?
>
> Rather than re-writing dets, I would go with something like Tokyo
> Tyrant, build on a suitable erlang wrapper (like

Hi.

I'm running Mnesia with fragmented tables, over a pool of servers.
Considering the amount of records to be inserted and the size limits
of dets, I will need a lot of fragments for each table and there seems
to be a big performance penalty on using many Mnesia fragments (even
when they don't have so much data).

The more plausible replacement, then, would be something like tcerl
with mnesiaex, but I've read somewhere that mnesiaex is still
experimental.
I'm dealing with data that affects millions of users, in my current
project, so I have to be very careful. That was the reason for wanting
to raise the limits of dets tables.

Thank you.
Igor.

> http://github.com/mallipeddi/tora), and perhaps put something like
> my sext library on top
> (http://svn.ulf.wiger.net/sext/trunk/sext/examples/tt_proto.erl).
>
> To match dets, I'd like to see some administrative code around the
> creation and opening of Tyrant tables, including the ability to
> "repair" the data from the transaction log. Seems doable, though.
>
> http://1978th.net/tokyotyrant/
>
> BR,
> Ulf W
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAksL9nUACgkQtqqFieqzed1v7wCfUf9bgDU+JfD8Gey51lvR7fxG
> 3iIAn0J5+8NpHH7DkLTgRlRuUKWomkXO
> =C66x
> -----END PGP SIGNATURE-----


-- 
"The secret of joy in work is contained in one word - excellence. To
know how to do something well is to enjoy it." - Pearl S. Buck.