On 6 July 2013 14:17, Noah Schwartz <span dir="ltr"><<a href="mailto:noah.schwartz1@gmail.com" target="_blank">noah.schwartz1@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto"><div>I can give this a shot. So you think that at some point while adding fragments items got put into the wrong fragment causing the deletes/reads to fail? Qlc works since its essentially scanning the whole table?</div>
</div></blockquote><div><br>Yes.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>How can I avoid this problem when adding fragments in the future?<br>
</div></div></blockquote><div><br>Not entirely sure. I've never relied on this feature myself because I didn't trust it to work properly! I just create a fragmented table, and repopulate the DB.<br><br>On Jul 5, 2013, at 3:12 PM, Chandru <<a href="mailto:chandrashekhar.mullaparthi@gmail.com" target="_blank">chandrashekhar.mullaparthi@gmail.com</a>> wrote:<br>
<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div><div class="h5"><blockquote type="cite"><div>Sounds like the redistribution of data among fragments hasn't happened for whatever reason. One way to fix this is as follows:<br>
<br>- Take a list of all the keys (assuming there aren't millions of them)<br>- Figure out if it is in the correct fragment using functions in mnesia_frag_hash.erl<br>
- If not, read the record, delete it from the appropriate fragment, and re-insert<br><br>If taking a list of all the keys isn't practical, I suggest:<br><br>- take a backup<br>- write a script which will traverses the backup and extracts all the keys which are in the wrong fragments<br>
- write another script to move said records from wrong fragment to the correct one<br><br>cheers<br>Chandru<br><br>On 5 July 2013 20:03, Noah Schwartz <span dir="ltr"><<a href="mailto:noah.schwartz1@gmail.com" target="_blank">noah.schwartz1@gmail.com</a>></span> wrote:<br>
<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This was originally an un-fragmented table that I converted to a fragmented table. I converted by calling change_table_frag with active and then adding my frags. </div>
<div class="gmail_extra"><div><div><br><br><div class="gmail_quote">
On Fri, Jul 5, 2013 at 2:58 PM, Chandru <span dir="ltr"><<a href="mailto:chandrashekhar.mullaparthi@gmail.com" target="_blank">chandrashekhar.mullaparthi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
How were these records written to the table in the first place? I've seen this happen when you write to the table without using mnesia_frag as the activity callback module.<br><br>cheers<br>Chandru<br><br><div class="gmail_quote">
<div><div>
On 5 July 2013 16:07, Noah Schwartz <span dir="ltr"><<a href="mailto:noah.schwartz1@gmail.com" target="_blank">noah.schwartz1@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>
<div dir="ltr">I have a table of type set that worked fine as a non-fragmented table. We started seeing performance issues with the table and with the anticipation of more clients in the system, we were worried the size would exceed the 2 GB max. <div>
<br></div><div>The table contains chat messages and we purge records older than 7 days. There seem to be a number of keys that don't show up in a read activity but, do show up when doing a table dump, a select, or using qlc to find records. As such, when this purge policy runs records are found that when I try to delete don't actually get deleted. It almost seems like read/delete are working off of one set while qlc is working off of another set.</div>
<div><br></div><div>Code/outputs below. As you can see for a given key read returns nothing, qlc returns an object, delete says it worked ok. Running the code again gives the same results. Am I using the fragmentation API incorrectly somehow?</div>
<div><br></div><div>Thanks in advance</div><div><br></div><div><b>Code:</b><br><div>-module(sel).</div><div>-include_lib("stdlib/include/qlc.hrl").</div><div>-record(archive_message313, {owner_with_day_utc, owner, with, day_gregorian_seconds, messages = []}).</div>
<div>-export([run/0]).</div><div><br></div><div>run () -></div><div> Owner = {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e", "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div>
<div> With = {"d735280e-d263-4e2b-beff-ed8ccaca5535", "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"}, </div>
<div> DayUtc = {2013, 6, 28}, </div><div> K = {Owner, With, DayUtc},</div><div><br></div><div> DelFun = fun () -> mnesia:delete({archive_message313, K}) end,</div><div> ReadFun = fun () -> mnesia:read({archive_message313, K}) end,</div>
<div><br></div><div> QlcFun = fun () -></div><div> Q = qlc:q([M || M <- mnesia:table(archive_message313), M#archive_message313.owner_with_day_utc == K]), </div><div> qlc:eval(Q)</div><div> end,</div><div> </div>
<div> Read = mnesia:activity(transaction, ReadFun, [], mnesia_frag),</div><div> Qlc = mnesia:activity(transaction, QlcFun, [], mnesia_frag),</div><div> Del = mnesia:activity(transaction, DelFun, [], mnesia_frag),</div>
<div><br></div><div> {read_result, Read, qlc_result, Qlc, del_result, Del}.</div><div><br></div><div><b>Output:</b></div><div><div>{read_result,[],qlc_result,</div><div> [{archive_message313,{{"9afa8671-7e88-436c-9ad4-4cb13ad45e1e",</div>
<div> "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div><div> {"d735280e-d263-4e2b-beff-ed8ccaca5535",</div>
<div> "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div>
<div> {2013,6,28}},</div><div> {"9afa8671-7e88-436c-9ad4-4cb13ad45e1e",</div><div> "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div>
<div> {"d735280e-d263-4e2b-beff-ed8ccaca5535",</div><div> "<a href="http://dj.barker.xxx.com" target="_blank">dj.barker.xxx.com</a>"},</div>
<div> 63539596800,</div>
<div> [{{{2013,6,28},{13,36,34}},</div><div> {xmlelement,"forwarded",</div><div> [{"xmlns","urn:xmpp:forward:0"}],</div>
<div> [{xmlelement,"delay",</div><div> [{"xmlns","urn:xmpp:delay"},</div><div> {"stamp","2013-06-28T13:36:34.000000Z"}],</div>
<div> []},</div><div> {xmlelement,"message",</div><div> [{"to",</div>
<div> "<a href="mailto:9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com" target="_blank">9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com</a>"},</div>
<div>
{"from",</div><div> "<a href="mailto:d735280e-d263-4e2b-beff-ed8ccaca5535@dj.barker.xxx.com" target="_blank">d735280e-d263-4e2b-beff-ed8ccaca5535@dj.barker.xxx.com</a>"},</div>
<div> {"type","chat"}],</div><div> [{xmlelement,"body",[],[...]}]}]},</div>
<div> 1372426594.889164},</div><div> {{{2013,6,28},{13,36,35}},</div><div> {xmlelement,"forwarded",</div><div>
[{"xmlns","urn:xmpp:forward:0"}],</div><div> [{xmlelement,"delay",</div><div> [{"xmlns","urn:xmpp:delay"},</div>
<div> {"stamp","2013-06-28T13:36:35.000000Z"}],</div><div> []},</div><div> {xmlelement,"message",</div>
<div> [{"to",</div><div> "<a href="mailto:9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com" target="_blank">9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com</a>"},</div>
<div> {"from",</div><div> "<a href="mailto:d735280e-d263-4e2b-beff-ed8ccaca5535@dj.barker.xxx.com" target="_blank">d735280e-d263-4e2b-beff-ed8ccaca5535@dj.barker.xxx.com</a>"},</div>
<div> {"type",[...]}],</div><div> [{xmlelement,"body",[],...}]}]},</div><div> 1372426595.124266},</div>
<div> {{{2013,6,28},{13,36,35}},</div><div> {xmlelement,"forwarded",</div><div> [{"xmlns","urn:xmpp:forward:0"}],</div>
<div> [{xmlelement,"delay",</div><div> [{"xmlns","urn:xmpp:delay"},</div><div> {"stamp","2013-06-28T13:36:35.000000Z"}],</div>
<div> []},</div><div> {xmlelement,"message",</div><div> [{"to",</div>
<div> "<a href="mailto:9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com" target="_blank">9afa8671-7e88-436c-9ad4-4cb13ad45e1e@dj.barker.xxx.com</a>"},</div>
<div>
{"from",[...]},</div><div> {[...],...}],</div><div> [{xmlelement,[...],...}]}]},</div>
<div> 1372426595.337535}]}],</div><div> del_result,ok}</div></div><span><font color="#888888">-- <br>Noah
</font></span></div></div>
<br></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>Noah
</font></span></div>
</blockquote></div><br>
</div></blockquote></div></div></div><br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br>