[erlang-bugs] All keys in the same slot (Mnesia/dets)

[Igor Ribeiro Sucupira]

Hello.

Thank you very much (I mean it) for teaching me I can use min_no_slots
together with repair. I always thought I could only use min_no_slots
"when the table is created", as the docs say. Most mortals won't think
that repair is such a case.  :-)
However, I also had to specify other non-default properties of the
table ({type, bag}, for example). Otherwise, it would fail with
type_mismatch.

As for the other issues (I'm taking the liberty of copying Mr.
Gudmundsson, who knows a lot about the internals of Mnesia)...

These dets files with all keys in the same slot are not corrupt. They
are just a consequence of the awful combination of table fragmentation
with disc_only_copies (mnesia_frag_hash with dets, to be more
precise).

If a table is split in N fragments, at most 1/N of the slots in each
dets fragment will be used.
I have verified that this is the case for all disc_only_copies tables
I have deployed. The only exception is the table that does not use
mnesia_frag_hash (another module was specified as hash_module at table
creation).

What happened to my "uc" table (which is a bag) during the upgrade
from R13B04 to R14B02 seems to have been unintended behavior from
Mnesia, which ended up benefiting me.

Each of the 1024 fragments in the uc table has about 300,000 objects,
but only about 1,500 keys. Before the upgrade, I think (not sure) each
fragment had around 1,500 slots, with all keys in the same slot (as
expected).
During the upgrade, for some reason, the fragments in most servers
(and in their replicas) seem to have been recreated with around
500,000 slots each.

This fact caused a performance improvement, since 1/1024 of the 500k
slots in each fragment are being used now.

I believe it was unintended, because (I think) Mnesia used the number
of objects (instead of the the number of keys) as estimated_no_objects
when it recreated the dets tables. I don't suppose it has bags in mind
for this step.

If I now pick one of the fragments that were left with around 1,500
slots and perform mnesia:change_table_copy_type to disc_copies and
then back to disc_only_copies, the number of slots is increased to
around 500k (Mnesia's source code gave me this tip). But I still don't
understand how it happened during the restarts/repairs for the
upgrade.

Calling del_table_copy and add_table_copy does not increase the number
of slots in the dets table (I think it's just copied from the replica
"as is", the same way it happens during the restarts with all nodes in
R14B02).

Thanks!
Igor.

On Thu, Jun 9, 2011 at 8:59 AM, Hans Bolinder
<hans.bolinder@REDACTED> wrote:
> Hi,
>
> [Igor Ribeiro Sucupira:]
>> The weirdest difference being that all records in uc_frag1004 are in
>> the same slot!
>>
>> 9> length([S || S <- lists:seq(0, element(2, dets:info(F1004,
>>    no_slots)) - 1), length(dets:slot(F1004, S)) > 0]).
>> 1
>
> If you still have the file available: How many objects are there in
> the slot? How many objects does a traversal of the file return? Try
>
>  length(dets:match_object(F1004,'_')).
>
> If the numbers differ then the file is corrupt.
>
>> Is there a way to manually raise the number of slots in a dets table?
>
> You can try closing the file, and then force a repair:
>
>  dets:open_file(T, [{min_no_slots,N},{repair,force}]).
>
> where N is some suitable integer.
>
> Best regards,
>
> Hans Bolinder, Erlang/OTP team, Ericsson
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs@REDACTED
> http://erlang.org/mailman/listinfo/erlang-bugs