<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Gotcha. That's kind of concerning given that its possible we will need to add fragments in the future...</div><div><br></div><div>I've also noticed that there's a primary key violation. I have the same exact record in two fragments which should be impossible. I've never done an operation on the fragments directly after converting the table.</div><div><br>Sent from my iPhone</div><div><br>On Jul 6, 2013, at 9:56 AM, Chandru <<a href="mailto:chandrashekhar.mullaparthi@gmail.com">chandrashekhar.mullaparthi@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div>On 6 July 2013 14:17, Noah Schwartz <span dir="ltr"><<a href="mailto:noah.schwartz1@gmail.com" target="_blank">noah.schwartz1@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div dir="auto"><div>I can give this a shot. So you think that at some point while adding fragments items got put into the wrong fragment causing the deletes/reads to fail? Qlc works since its essentially scanning the whole table?</div>

</div></blockquote><div><br>Yes.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>How can I avoid this problem when adding fragments in the future?<br>

</div></div></blockquote><div><br>Not entirely sure. I've never relied on this feature myself because I didn't trust it to work properly! I just create a fragmented table, and repopulate the DB.<br><br>On Jul 5, 2013, at 3:12 PM, Chandru <<a href="mailto:chandrashekhar.mullaparthi@gmail.com" target="_blank">chandrashekhar.mullaparthi@gmail.com</a>> wrote:<br>

<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div><div class="h5"><blockquote type="cite"><div>Sounds like the redistribution of data among fragments hasn't happened for whatever reason. One way to fix this is as follows:<br>

<br>- Take a list of all the keys (assuming there aren't millions of them)<br>- Figure out if it is in the correct fragment using functions in mnesia_frag_hash.erl<br>

- If not, read the record, delete it from the appropriate fragment, and re-insert<br><br>If taking a list of all the keys isn't practical, I suggest:<br><br>- take a backup<br>- write a script which will traverses the backup and extracts all the keys which are in the wrong fragments<br>



- write another script to move said records from wrong fragment to the correct one<br><br>cheers<br>Chandru<br><br>On 5 July 2013 20:03, Noah Schwartz <span dir="ltr"><<a href="mailto:noah.schwartz1@gmail.com" target="_blank">noah.schwartz1@gmail.com</a>></span> wrote:<br>



<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This was originally an un-fragmented table that I converted to a fragmented table. I converted by calling change_table_frag with active and then adding my frags. </div>



<div class="gmail_extra"><div><div><br><br><div class="gmail_quote">

On Fri, Jul 5, 2013 at 2:58 PM, Chandru <span dir="ltr"><<a href="mailto:chandrashekhar.mullaparthi@gmail.com" target="_blank">chandrashekhar.mullaparthi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">





How were these records written to the table in the first place? I've seen this happen when you write to the table without using mnesia_frag as the activity callback module.<br><br>cheers<br>Chandru<br><br><div class="gmail_quote">





<div><div>

On 5 July 2013 16:07, Noah Schwartz <span dir="ltr"><<a href="mailto:noah.schwartz1@gmail.com" target="_blank">noah.schwartz1@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">





<div><div>

<div dir="ltr">I have a table of type set that worked fine as a non-fragmented table. We started seeing performance issues with the table and with the anticipation of more clients in the system, we were worried the size would exceed the 2 GB max. <div>









<br></div><div>The table contains chat messages and we purge records older than 7 days. There seem to be a number of keys that don't show up in a read activity but, do show up when doing a table dump, a select, or using qlc to find records. As such, when this purge policy runs records are found that when I try to delete don't actually get deleted. It almost seems like read/delete are working off of one set while qlc is working off of another set.</div>









<div><br></div><div>Code/outputs below. As you can see for a given key read returns nothing, qlc returns an object, delete says it worked ok. Running the code again gives the same results. Am I using the fragmentation API incorrectly somehow?</div>









<div><br></div><div>Thanks in advance</div><div><br></div><div><b>Code:</b><br><div>-module(sel).</div><div>-include_lib("stdlib/include/qlc.hrl").</div><div>-record(archive_message313, {owner_with_day_utc, owner, with, day_gregorian_seconds, messages = []}).</div>









<div>-export([run/0]).</div><div><br></div><div>run () -></div><div>  Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div>







<div>  With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"}, </div>

<div>  DayUtc = {2013, 6, 28}, </div><div>  K = {Owner, With, DayUtc},</div><div><br></div><div>  DelFun = fun () -> mnesia:delete({archive_message313, K}) end,</div><div>  ReadFun = fun () -> mnesia:read({archive_message313, K}) end,</div>









<div><br></div><div>  QlcFun = fun () -></div><div>    Q = qlc:q([M || M <- mnesia:table(archive_message313), M#archive_message313.owner_with_day_utc == K]), </div><div>    qlc:eval(Q)</div><div>  end,</div><div>  </div>









<div>  Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag),</div><div>  Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag),</div><div>  Del = mnesia:activity(transaction, DelFun, [], mnesia_frag),</div>









<div><br></div><div>  {read_result, Read, qlc_result, Qlc, del_result, Del}.</div><div><br></div><div><b>Output:</b></div><div><div>{read_result,[],qlc_result,</div><div>             [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e",</div>









<div>                                    "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div><div>                                   {"d735280e-d263-4e2b-beff-ed8ccaca5535",</div>







<div>                                    "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div>

<div>                                   {2013,6,28}},</div><div>                                  {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e",</div><div>                                   "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div>









<div>                                  {"d735280e-d263-4e2b-beff-ed8ccaca5535",</div><div>                                   "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div>







<div>                                  63539596800,</div>

<div>                                  [{{{2013,6,28},{13,36,34}},</div><div>                                    {xmlelement,"forwarded",</div><div>                                                [{"xmlns","urn:xmpp:forward:0"}],</div>









<div>                                                [{xmlelement,"delay",</div><div>                                                             [{"xmlns","urn:xmpp:delay"},</div><div>                                                              {"stamp","2013-06-28T13:36:34.000000Z"}],</div>









<div>                                                             []},</div><div>                                                 {xmlelement,"message",</div><div>                                                             [{"to",</div>









<div>                                                               "<a href="mailto:9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com" target="_blank">9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com</a>"},</div>







<div>

                                                              {"from",</div><div>                                                               "<a href="mailto:d735280e-d263-4e2b-beff-ed8ccaca5535@dj.barker.xxx.com" target="_blank">d735280e-d263-4e2b-beff-ed8ccaca5535@dj.barker.xxx.com</a>"},</div>









<div>                                                              {"type","chat"}],</div><div>                                                             [{xmlelement,"body",[],[...]}]}]},</div>









<div>                                    1372426594.889164},</div><div>                                   {{{2013,6,28},{13,36,35}},</div><div>                                    {xmlelement,"forwarded",</div><div>









                                                [{"xmlns","urn:xmpp:forward:0"}],</div><div>                                                [{xmlelement,"delay",</div><div>                                                             [{"xmlns","urn:xmpp:delay"},</div>









<div>                                                              {"stamp","2013-06-28T13:36:35.000000Z"}],</div><div>                                                             []},</div><div>                                                 {xmlelement,"message",</div>









<div>                                                             [{"to",</div><div>                                                               "<a href="mailto:9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com" target="_blank">9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com</a>"},</div>









<div>                                                              {"from",</div><div>                                                               "<a href="mailto:d735280e-d263-4e2b-beff-ed8ccaca5535@dj.barker.xxx.com" target="_blank">d735280e-d263-4e2b-beff-ed8ccaca5535@dj.barker.xxx.com</a>"},</div>









<div>                                                              {"type",[...]}],</div><div>                                                             [{xmlelement,"body",[],...}]}]},</div><div>                                    1372426595.124266},</div>









<div>                                   {{{2013,6,28},{13,36,35}},</div><div>                                    {xmlelement,"forwarded",</div><div>                                                [{"xmlns","urn:xmpp:forward:0"}],</div>









<div>                                                [{xmlelement,"delay",</div><div>                                                             [{"xmlns","urn:xmpp:delay"},</div><div>                                                              {"stamp","2013-06-28T13:36:35.000000Z"}],</div>









<div>                                                             []},</div><div>                                                 {xmlelement,"message",</div><div>                                                             [{"to",</div>









<div>                                                               "<a href="mailto:9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com" target="_blank">9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com</a>"},</div>







<div>

                                                              {"from",[...]},</div><div>                                                              {[...],...}],</div><div>                                                             [{xmlelement,[...],...}]}]},</div>









<div>                                    1372426595.337535}]}],</div><div>             del_result,ok}</div></div><span><font color="#888888">-- <br>Noah
</font></span></div></div>
<br></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>Noah
</font></span></div>
</blockquote></div><br>
</div></blockquote></div></div></div><br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br>
</div></blockquote></body></html>