[erlang-questions] High volume CDR analysis

[Chandru]

On 21/01/2008, Ukyo Virgden <listproc@REDACTED> wrote:
>
> Any suggestions? I suppose there is a storage limit of 2gig for
> mnesia per node right?
>

There is a limit of 2GB per dets file table. Mnesia's disc_copies
tables and disc_only_copies tables are stored on disk in dets file.
That doesn't stop you from having multiple tables each with about 2GB
of data.

If you use 32 bit erlang, you can address 4GB of memory.

If you use 64 bit erlang, you can address more memory than you will
need to but will hit other design limitations in mnesia. A mix of
fragmented mnesia tables and 64 bit erlang will go a long way.

For 300 million records, at 1KB a record, you will roughly need 300GB
of storage. Since a machine with that much RAM is out of the question
for most people/organisations, you will have to process them
incrementally. That is where Joe's mail comes in about partitioning
your problem.

cheers
Chandru