<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Thu, Dec 17, 2015 at 7:44 PM Antonio SJ Musumeci <trapexit@spawn.link> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">What exactly is that you need behaviorally? You could also have a process which just continuously iterates over the table placing the records into a rotating `disk_log`. If you include a timestamp or have something to know precisely which version of the record you can replay the log and recover the state you want. If you need straight up snapshots then maybe a liberal select would give you a dump. I don't recall however at what level the select/match locks and if the table is large it'd be expensive memory wise.<br><br>It's hard to beat a COW setup if you need snapshots.</div></blockquote><div><br></div><div>Right but COW over 1M key in memory could be hard. I need to think more.</div><div><br></div><div>As an experiment i wrote this small code:</div><div><a href="https://github.com/barrel-db/memdb">https://github.com/barrel-db/memdb</a></div><div><br></div><div>which implement MVCC over an ETS table (ordered set). For convenience each keys are for now only binaries though that can change. It allows poor prefix lookup by checking the next key starting with the prefix. Iterators offers a consistant view from a point of time. Multiple readers can consume an iterator while write happen behind. The compaction (removing old revisions) is not implemented.</div><div><br></div><div>I don't think it will go further though. I got another idea I wanted to test to build a memory database, maximising the use of the processes. Just wanted to share the code.</div><div><br></div><div>- benoit</div><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 17, 2015 at 11:38 AM, Felix Gallo <span dir="ltr"><<a href="mailto:felixgallo@gmail.com" target="_blank">felixgallo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">You can take advantage of erlang's concurrency to get arbitrarily-close-to-redis semantics.<div><br></div><div>For example, redis's bgsave could be achieved by writing as usual to your ets table, but also sending a duplicate message to a gen_server whose job it is to keep up to date a second, slave, ets table. That gen_server would be the one to provide dumps (via to_dets or whatever other facility). Then if it has to pause while it dumps, its message queue grows during the duration but eventually flushes out and brings itself back up to date. Meanwhile the primary ets replica continues to be usable.</div><div><br></div><div>It's not a silver bullet because, like redis, you would still have to worry about the pathological conditions, like dumps taking so long that the slave gen_server's queue gets out of control, or out of memory conditions, etc., etc. But if you feel like implementing paxos or waiting about 3 months, you could also generalize the gen_server so that a group of them formed a distributed cluster.</div><div><br></div><div>F.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Dec 17, 2015 at 8:16 AM, Benoit Chesneau <span dir="ltr"><<a href="mailto:bchesneau@gmail.com" target="_blank">bchesneau@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><br><br><div class="gmail_quote"><div><div><div dir="ltr">On Thu, Dec 17, 2015 at 5:13 PM Benoit Chesneau <<a href="mailto:bchesneau@gmail.com" target="_blank">bchesneau@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Thu, Dec 17, 2015 at 3:24 PM Fred Hebert <<a href="mailto:mononcqc@ferd.ca" target="_blank">mononcqc@ferd.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 12/17, Benoit Chesneau wrote:<br>
>But what happen when I use `ets:tab2file/2` while keys are continuously<br>
>added at the end? When does it stop?<br>
><br>
<br>
I'm not sure what answer you expect to the question "how can I keep an<br>
infinitely growing table from taking an infinite amount of time to dump<br>
to disk" that doesn't require locking it to prevent the growth from<br>
showing up.<br></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>well by keeping a version of the data at some point :) But that's not how it works unfortunately.</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Do note that safe_fixtable/2 does *not* prevent new inserted elements<br>
from showing up in your table -- it only prevents objects from being<br>
taken out or being iterated over twice. While it's easier to create a<br>
pathological case with an ordered_set table (keeping adding +1 as keys<br>
near the end), it is not beyond the realm of possibility to do so with<br>
other table types (probably with lots of insertions and playing with<br>
process priorities, or predictable hash sequences).<br>
<br>
I don't believe there's any way to lock a public table (other than<br>
implicit blocking in match and select functions). If I were to give a<br>
wild guess, I'd say to look at ets:info(Tab,size), and have your<br>
table-dumping process stop when it reaches the predetermined size or<br>
meets an earlier exit. This would let you bound the time it takes you to<br>
dump the table, at the cost of possibly neglecting to add information<br>
(which you would do anyway -- you would just favor older info before<br>
newer info). This would however imply reimplementing your own tab2file<br>
functionality.<br>
<br></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>Good idea, i need to think a little more about it.. I wish it could be possible to fork an ets table at some point and only use this snapshot in memory like REDIS does literally by forking the process when dumping it. That would be useful...</div><div><br></div><div>Thanks for the answer!</div></div></div><div dir="ltr"><div class="gmail_quote"><br></div></div></blockquote><div><br></div></div></div><div>side note, but i am thinking that selecting keys per batch also limit the possible effects of the concurrent writes since it can work faster that way. though writing to the file is slow.</div><span><font color="#888888"><div><br></div><div>- benoit </div></font></span></div></div>
<br></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div>
</blockquote></div></div>