[erlang-questions] non-atomic nature of mnesia:dirty_update_counter

Mon Mar 7 17:01:20 CET 2011

Hi,

Any chance you will be giving this talk at the Bay Area Erlang Factory conference in May? 

Thanks,

Dan.
On Monday, March 7, 2011 at 5:29 AM, Ulf Wiger wrote: 
> 
> A key issue here is that mnesia has basically been in maintenance mode for about a decade now. The main reason for this is that Ericsson itself is using mnesia mainly as a configuration database, and is pretty comfortable with its current capabilities.
> 
> At the Erlang User Group meeting at QCon London this Thursday, I will give a short talk about what we at Erlang Solutions are doing with mnesia right now, together with some of our customers, to address the kind of problems Bob is referring to.
> 
> http://www.erlang-solutions.com/etc/usergroup/london
> 
> BR,
> Ulf
> 
> On 6 Mar 2011, at 20:33, Bob Ippolito wrote:
> 
> > Well that's the problem. There is nothing that does a better job of
> > exactly what Mnesia does. However, with a few years of operational
> > experience with Mnesia, you will get network partitions and/or you'll
> > run into the 2GB limit (or run out of RAM) and if any of those things
> > happen you'll be in pain. Node restarts also get painfully slow when
> > the indexes are large.
> > 
> > If there is another data model or database that can fit your
> > application, you might be better off without Mnesia.
> > 
> > On Sun, Mar 6, 2011 at 11:17 AM, <bile@REDACTED> wrote:
> > > Such as? There are lots of random DBs available but any with the ease
> > > of use and speed mnesia seems to offer?
> > > 
> > > On Sun, 6 Mar 2011 10:01:24 -0800
> > > Bob Ippolito <bob@REDACTED> wrote:
> > > 
> > > > Well, I'm sure you do know what you're doing in general, but not with
> > > > Mnesia. Mnesia is quirky. You will probably eventually regret using
> > > > Mnesia. We try and avoid it as much as possible, because of its
> > > > inability to do the right thing with network partitions and the dets
> > > > limitations for disk tables. If you can afford to, I'd look at using
> > > > something else instead.
> > > > 
> > > > On Sun, Mar 6, 2011 at 9:54 AM, Ken Ellis <kaellis@REDACTED> wrote:
> > > > > Yeah but to be blunt, that's two responses with "if you know what
> > > > > you're doing". I know what I'm doing, I just have no idea what
> > > > > Mnesia is doing. The documentation is inconsistent with itself and
> > > > > with Gudmundsson's remarks. Perhaps switch it off for any table
> > > > > spread across multiple nodes if no guarantees can be made in that
> > > > > case?
> > > > > 
> > > > > 
> > > > > On Sun, Mar 6, 2011 at 12:07 PM, Bob Ippolito <bob@REDACTED>
> > > > > wrote:
> > > > > > Well, they're actually quite useful (for performance) if you know
> > > > > > what you're doing and the application can tolerate it. Often times
> > > > > > the return value of a counter is not important. I suppose if they
> > > > > > were ripped out, maybe fewer people would use Mnesia, which might
> > > > > > be a net good :)
> > > > > > 
> > > > > > On Sun, Mar 6, 2011 at 9:00 AM, Ken Ellis <kaellis@REDACTED>
> > > > > > wrote:
> > > > > > > Amen, the whole dirty_* set of functions should be ripped out of
> > > > > > > the release.
> > > > > > > 
> > > > > > > 
> > > > > > > On Sun, Mar 6, 2011 at 11:28 AM, Dan Gudmundsson
> > > > > > > <dgud@REDACTED> wrote:
> > > > > > > > Nothing is promised with dirty_write don't use it.
> > > > > > > > 
> > > > > > > > Dirty read may be ok if you know what you are doing
> > > > > > > > but avoid dirty_write in multiple node system.
> > > > > > > > 
> > > > > > > > /Dan
> > > > > > > > 
> > > > > > > > On Sun, Mar 6, 2011 at 5:12 PM, Ken Ellis <kaellis@REDACTED>
> > > > > > > > wrote:
> > > > > > > > > Hi Ulf,
> > > > > > > > > 
> > > > > > > > > Thanks for the clarification. I also went back to that User
> > > > > > > > > Guide and it does an admirable job of defining atomic as
> > > > > > > > > meaning the ACID all-or-nothing variety and not what for
> > > > > > > > > example you'd call atomic when incrementing counters within the
> > > > > > > > > linux kernel. My bad, forgot I was dealing with a database.
> > > > > > > > >  So I think in that sense the dirty_update_counter is atomic
> > > > > > > > > across all nodes. I guess the question is when the users guide
> > > > > > > > > says atomicity is lost with dirty operations, what exactly is
> > > > > > > > > non-atomic about dirty_write or any dirty operation given that
> > > > > > > > > "Mnesia also ensures that all replicas of a table are updated
> > > > > > > > > if a dirty write operation is [successfully?] performed on a
> > > > > > > > > [any?] table". Isn't that the definition of all-or-nothing
> > > > > > > > > atomicity? Unless that statement is inaccurate, that would seem
> > > > > > > > > to mean all dirty_* operations are atomic, and that dirty is a
> > > > > > > > > reference to what an RDBMAS would call dirty reads, with an
> > > > > > > > > additional qualification that it can break isolation of
> > > > > > > > > transactions. Which by the way means there is no isolation
> > > > > > > > > guarantee for any transaction, since dirty operations
> > > > > > > > > compromise their isolation. (IMaybe filthy_* would be a better
> > > > > > > > > prefix :). But I suppose for now I'll take the statement about
> > > > > > > > > lack of atomicity as true, and that dirty writes can be only
> > > > > > > > > partially executed and can leave tables on different nodes in
> > > > > > > > > an inconsistent state.
> > > > > > > > > 
> > > > > > > > > Ken
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > On Sun, Mar 6, 2011 at 2:19 AM, Ulf Wiger
> > > > > > > > > <ulf.wiger@REDACTED> wrote:
> > > > > > > > > > 
> > > > > > > > > > On 6 Mar 2011, at 06:54, Ken Ellis wrote:
> > > > > > > > > > 
> > > > > > > > > > After running test_counter:run(5000,1) on each node
> > > > > > > > > > simultaneously, and comparing the files, i get 2893 duplicates
> > > > > > > > > > out of 5000. Although the counter has been incremented by
> > > > > > > > > > 10000. I can't repro running on a single node with a large
> > > > > > > > > > number of processes, so it seems atomic in that case.
> > > > > > > > > > 
> > > > > > > > > > It is exactly the single-node case that is atomic. The
> > > > > > > > > > documentation should spell this out.
> > > > > > > > > > The User Guide does say that you lose mnesia's atomicity and
> > > > > > > > > > isolation properties if you use dirty operations, which does
> > > > > > > > > > seem to contradict what is said about dirty_update_counter(),
> > > > > > > > > > but what the Reference Manual (*and* User Guide) should say is
> > > > > > > > > > that dirty_update_counter() is atomic only in a very limited
> > > > > > > > > > context. The User Guide does say something else that is,
> > > > > > > > > > strictly speaking, wrong: "It is not possible to have
> > > > > > > > > > transaction protected updates of counter records."
> > > > > > > > > > It is of course possible - they are only records after all.
> > > > > > > > > > You can do an update counter inside a transaction through a
> > > > > > > > > > combination of read() and write().
> > > > > > > > > > BR,
> > > > > > > > > > Ulf W
> > > > > > > > > > Ulf Wiger, CTO, Erlang Solutions, Ltd.
> > > > > > > > > > http://erlang-solutions.com
> > > > > > > > > 
> > > > > > > > > ________________________________________________________________
> > > > > > > > > erlang-questions (at) erlang.org mailing list.
> > > > > > > > > See http://www.erlang.org/faq.html
> > > > > > > > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > > > > > > 
> > > > > > > ________________________________________________________________
> > > > > > > erlang-questions (at) erlang.org mailing list.
> > > > > > > See http://www.erlang.org/faq.html
> > > > > > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > > > 
> > > > ________________________________________________________________
> > > > erlang-questions (at) erlang.org mailing list.
> > > > See http://www.erlang.org/faq.html
> > > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > > 
> > > 
> > > ________________________________________________________________
> > > erlang-questions (at) erlang.org mailing list.
> > > See http://www.erlang.org/faq.html
> > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > 
> > ________________________________________________________________
> > erlang-questions (at) erlang.org mailing list.
> > See http://www.erlang.org/faq.html
> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> 
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
> 
> 
> 
> 
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> 