[erlang-questions] Efficient Insertions in Mnesia tables

[Dan Gudmundsson]

On Fri, Nov 5, 2010 at 11:15 AM, Olivier BOUDEVILLE
<olivier.boudeville@REDACTED> wrote:
> Hi,
>
> We are trying to write (with mnesia:dirty_write) in a disc_only_copies
> Mnesia table (type: set, not fragmented, not replicated) records (ex: 60
> 000 of them) and we observe that the insertion time is increasing as the
> table is increasingly crowded. This is not really a surprise but something
> we need to avoid. What we would like is to have constant (and preferably
> low) insertion times, like we had when writing directly to a file.
>
> We tried to get as close as possible with the following settings and use:
>
>                        % We want tables to be dumped less frequently from
> memory to disc,
>                        % in order to buffer writings (default value is
> 4):
>                        ok = application:set_env( mnesia, dc_dump_limit, 1
> ),

You are using disc_only_copies, there is no memory to dump here,
this does nothing for you.

>
>                        % Increases a lot (default value is 100) the
> maximum number of
>                        % writes to the transaction log before a new dump
> is performed:
>                        ok = application:set_env( mnesia,
> dump_log_write_threshold, 50000 ),
>
> Over time we see the CPU load decrease steadily, the computer seems to
> spend most of its time fighting for locks.
>

Locks ? You are using dirty i.e. no locks are taken, or are you
talking about mutex's in the emulator).

> We happen to be in a pretty favorable situation (only writes, no
> concurrent access to a given table). We chose disc_only_copies as there
> might be a large number of such tables and if they filled over time they
> could exhaust the RAM.
>
> Is there anything we missed that would allow us (roughly) constant
> insertion times with Mnesia?
>

Mnesia should mostly behave as dets does when using it directly, does it?
i.e. switch the mnesia calls to direct dets calls and see if it does.

You are not doing something stupid, like have an index on the table and all
test entries have the same value on the index field?
(which would cause a O(n) insertion time, and have happend more than
once when someone is
measuring performance).

/Dan

> Thanks in advance for any hint,
> Best regards,
>
> Olivier.
> ---------------------------
> Olivier Boudeville
>
> EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart, France
> Département SINETICS, groupe ASICS (I2A), bureau B-226
> Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47
> 65 27 13
>
>
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.
>
> Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
> ____________________________________________________
>
> This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.
>
> E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
>