[erlang-questions] inconsistent results with fragmented table

Chandru <>
Fri Jul 5 21:12:40 CEST 2013


Sounds like the redistribution of data among fragments hasn't happened for
whatever reason. One way to fix this is as follows:

- Take a list of all the keys (assuming there aren't millions of them)
- Figure out if it is in the correct fragment using functions in
mnesia_frag_hash.erl
- If not, read the record, delete it from the appropriate fragment, and
re-insert

If taking a list of all the keys isn't practical, I suggest:

- take a backup
- write a script which will traverses the backup and extracts all the keys
which are in the wrong fragments
- write another script to move said records from wrong fragment to the
correct one

cheers
Chandru

On 5 July 2013 20:03, Noah Schwartz <> wrote:

> This was originally an un-fragmented table that I converted to a
> fragmented table. I converted by calling change_table_frag with active and
> then adding my frags.
>
>
> On Fri, Jul 5, 2013 at 2:58 PM, Chandru <
> > wrote:
>
>> How were these records written to the table in the first place? I've seen
>> this happen when you write to the table without using mnesia_frag as the
>> activity callback module.
>>
>> cheers
>> Chandru
>>
>>  On 5 July 2013 16:07, Noah Schwartz <> wrote:
>>
>>>  I have a table of type set that worked fine as a non-fragmented table.
>>> We started seeing performance issues with the table and with the
>>> anticipation of more clients in the system, we were worried the size would
>>> exceed the 2 GB max.
>>>
>>> The table contains chat messages and we purge records older than 7 days.
>>> There seem to be a number of keys that don't show up in a read activity
>>> but, do show up when doing a table dump, a select, or using qlc to find
>>> records. As such, when this purge policy runs records are found that when I
>>> try to delete don't actually get deleted. It almost seems like read/delete
>>> are working off of one set while qlc is working off of another set.
>>>
>>> Code/outputs below. As you can see for a given key read returns nothing,
>>> qlc returns an object, delete says it worked ok. Running the code again
>>> gives the same results. Am I using the fragmentation API incorrectly
>>> somehow?
>>>
>>> Thanks in advance
>>>
>>> *Code:*
>>> -module(sel).
>>> -include_lib("stdlib/include/qlc.hrl").
>>> -record(archive_message313, {owner_with_day_utc, owner, with,
>>> day_gregorian_seconds, messages = []}).
>>> -export([run/0]).
>>>
>>> run () ->
>>>   Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "dj.barker.xxx.com"},
>>>   With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "dj.barker.xxx.com"},
>>>   DayUtc = {2013, 6, 28},
>>>   K = {Owner, With, DayUtc},
>>>
>>>   DelFun = fun () -> mnesia:delete({archive_message313, K}) end,
>>>   ReadFun = fun () -> mnesia:read({archive_message313, K}) end,
>>>
>>>   QlcFun = fun () ->
>>>     Q = qlc:q([M || M <- mnesia:table(archive_message313),
>>> M#archive_message313.owner_with_day_utc == K]),
>>>     qlc:eval(Q)
>>>   end,
>>>
>>>   Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag),
>>>   Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag),
>>>   Del = mnesia:activity(transaction, DelFun, [], mnesia_frag),
>>>
>>>   {read_result, Read, qlc_result, Qlc, del_result, Del}.
>>>
>>> *Output:*
>>> {read_result,[],qlc_result,
>>>
>>>  [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e",
>>>                                     "dj.barker.xxx.com"},
>>>
>>>  {"d735280e-d263-4e2b-beff-ed8ccaca5535",
>>>                                     "dj.barker.xxx.com"},
>>>                                    {2013,6,28}},
>>>
>>> {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e",
>>>                                    "dj.barker.xxx.com"},
>>>
>>> {"d735280e-d263-4e2b-beff-ed8ccaca5535",
>>>                                    "dj.barker.xxx.com"},
>>>                                   63539596800,
>>>                                   [{{{2013,6,28},{13,36,34}},
>>>                                     {xmlelement,"forwarded",
>>>
>>> [{"xmlns","urn:xmpp:forward:0"}],
>>>                                                 [{xmlelement,"delay",
>>>
>>>  [{"xmlns","urn:xmpp:delay"},
>>>
>>> {"stamp","2013-06-28T13:36:34.000000Z"}],
>>>                                                              []},
>>>                                                  {xmlelement,"message",
>>>                                                              [{"to",
>>>                                                                "
>>> "},
>>>                                                                {"from",
>>>                                                                "
>>> "},
>>>
>>> {"type","chat"}],
>>>
>>>  [{xmlelement,"body",[],[...]}]}]},
>>>                                     1372426594.889164},
>>>                                    {{{2013,6,28},{13,36,35}},
>>>                                     {xmlelement,"forwarded",
>>>
>>> [{"xmlns","urn:xmpp:forward:0"}],
>>>                                                 [{xmlelement,"delay",
>>>
>>>  [{"xmlns","urn:xmpp:delay"},
>>>
>>> {"stamp","2013-06-28T13:36:35.000000Z"}],
>>>                                                              []},
>>>                                                  {xmlelement,"message",
>>>                                                              [{"to",
>>>                                                                "
>>> "},
>>>                                                               {"from",
>>>                                                                "
>>> "},
>>>
>>> {"type",[...]}],
>>>
>>>  [{xmlelement,"body",[],...}]}]},
>>>                                     1372426595.124266},
>>>                                    {{{2013,6,28},{13,36,35}},
>>>                                     {xmlelement,"forwarded",
>>>
>>> [{"xmlns","urn:xmpp:forward:0"}],
>>>                                                 [{xmlelement,"delay",
>>>
>>>  [{"xmlns","urn:xmpp:delay"},
>>>
>>> {"stamp","2013-06-28T13:36:35.000000Z"}],
>>>                                                              []},
>>>                                                  {xmlelement,"message",
>>>                                                              [{"to",
>>>                                                                "
>>> "},
>>>
>>> {"from",[...]},
>>>
>>> {[...],...}],
>>>
>>>  [{xmlelement,[...],...}]}]},
>>>                                     1372426595.337535}]}],
>>>              del_result,ok}
>>> --
>>> Noah
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> 
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>
>
>
> --
> Noah
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130705/c622fb2b/attachment.html>


More information about the erlang-questions mailing list