[erlang-questions] non-atomic nature of mnesia:dirty_update_counter

Mon Mar 7 14:12:52 CET 2011

Will the talk or it's notes be available online?

On Mon, 7 Mar 2011 12:29:54 +0000
Ulf Wiger <ulf.wiger@REDACTED> wrote:

> 
> A key issue here is that mnesia has basically been in maintenance
> mode for about a decade now. The main reason for this is that
> Ericsson itself is using mnesia mainly as a configuration database,
> and is pretty comfortable with its current capabilities.
> 
> At the Erlang User Group meeting at QCon London this Thursday, I will
> give a short talk about what we at Erlang Solutions are doing with
> mnesia right now, together with some of our customers, to address the
> kind of problems Bob is referring to.
> 
> http://www.erlang-solutions.com/etc/usergroup/london
> 
> BR,
> Ulf
> 
> On 6 Mar 2011, at 20:33, Bob Ippolito wrote:
> 
> > Well that's the problem. There is nothing that does a better job of
> > exactly what Mnesia does. However, with a few years of operational
> > experience with Mnesia, you will get network partitions and/or
> > you'll run into the 2GB limit (or run out of RAM) and if any of
> > those things happen you'll be in pain. Node restarts also get
> > painfully slow when the indexes are large.
> > 
> > If there is another data model or database that can fit your
> > application, you might be better off without Mnesia.
> > 
> > On Sun, Mar 6, 2011 at 11:17 AM,  <bile@REDACTED> wrote:
> >> Such as? There are lots of random DBs available but any with the
> >> ease of use and speed mnesia seems to offer?
> >> 
> >> On Sun, 6 Mar 2011 10:01:24 -0800
> >> Bob Ippolito <bob@REDACTED> wrote:
> >> 
> >>> Well, I'm sure you do know what you're doing in general, but not
> >>> with Mnesia. Mnesia is quirky. You will probably eventually
> >>> regret using Mnesia. We try and avoid it as much as possible,
> >>> because of its inability to do the right thing with network
> >>> partitions and the dets limitations for disk tables. If you can
> >>> afford to, I'd look at using something else instead.
> >>> 
> >>> On Sun, Mar 6, 2011 at 9:54 AM, Ken Ellis <kaellis@REDACTED>
> >>> wrote:
> >>>> Yeah but to be blunt, that's two responses with "if you know what
> >>>> you're doing".  I know what I'm doing, I just have no idea what
> >>>> Mnesia is doing.  The documentation is inconsistent with itself
> >>>> and with Gudmundsson's remarks.  Perhaps switch it off for any
> >>>> table spread across multiple nodes if no guarantees can be made
> >>>> in that case?
> >>>> 
> >>>> 
> >>>> On Sun, Mar 6, 2011 at 12:07 PM, Bob Ippolito <bob@REDACTED>
> >>>> wrote:
> >>>>> Well, they're actually quite useful (for performance) if you
> >>>>> know what you're doing and the application can tolerate it.
> >>>>> Often times the return value of a counter is not important. I
> >>>>> suppose if they were ripped out, maybe fewer people would use
> >>>>> Mnesia, which might be a net good :)
> >>>>> 
> >>>>> On Sun, Mar 6, 2011 at 9:00 AM, Ken Ellis <kaellis@REDACTED>
> >>>>> wrote:
> >>>>>> Amen, the whole dirty_* set of functions should be ripped out
> >>>>>> of the release.
> >>>>>> 
> >>>>>> 
> >>>>>> On Sun, Mar 6, 2011 at 11:28 AM, Dan Gudmundsson
> >>>>>> <dgud@REDACTED> wrote:
> >>>>>>> Nothing is promised with dirty_write don't use it.
> >>>>>>> 
> >>>>>>> Dirty read may be ok if you know what you are doing
> >>>>>>> but avoid dirty_write in multiple node system.
> >>>>>>> 
> >>>>>>> /Dan
> >>>>>>> 
> >>>>>>> On Sun, Mar 6, 2011 at 5:12 PM, Ken Ellis <kaellis@REDACTED>
> >>>>>>> wrote:
> >>>>>>>> Hi Ulf,
> >>>>>>>> 
> >>>>>>>> Thanks for the clarification.  I also went back to that User
> >>>>>>>> Guide and it does an admirable job of defining atomic as
> >>>>>>>> meaning the ACID all-or-nothing variety and not what for
> >>>>>>>> example you'd call atomic when incrementing counters within
> >>>>>>>> the linux kernel.  My bad, forgot I was dealing with a
> >>>>>>>> database. So I think in that sense the dirty_update_counter
> >>>>>>>> is atomic across all nodes.  I guess the question is when
> >>>>>>>> the users guide says atomicity is lost with dirty
> >>>>>>>> operations, what exactly is non-atomic about dirty_write or
> >>>>>>>> any dirty operation given that "Mnesia also ensures that all
> >>>>>>>> replicas of a table are updated if a dirty write operation
> >>>>>>>> is [successfully?] performed on a [any?] table".  Isn't that
> >>>>>>>> the definition of all-or-nothing atomicity? Unless that
> >>>>>>>> statement is inaccurate, that would seem to mean all dirty_*
> >>>>>>>> operations are atomic, and that dirty is a reference to what
> >>>>>>>> an RDBMAS would call dirty reads, with an additional
> >>>>>>>> qualification that it can break isolation of transactions.
> >>>>>>>> Which by the way means there is no isolation guarantee for
> >>>>>>>> any transaction, since dirty operations compromise their
> >>>>>>>> isolation.  (IMaybe filthy_* would be a better prefix :).
> >>>>>>>> But I suppose for now I'll take the statement about lack of
> >>>>>>>> atomicity as true, and that dirty writes can be only
> >>>>>>>> partially executed and can leave tables on different nodes
> >>>>>>>> in an inconsistent state.
> >>>>>>>> 
> >>>>>>>> Ken
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> On Sun, Mar 6, 2011 at 2:19 AM, Ulf Wiger
> >>>>>>>> <ulf.wiger@REDACTED> wrote:
> >>>>>>>>> 
> >>>>>>>>> On 6 Mar 2011, at 06:54, Ken Ellis wrote:
> >>>>>>>>> 
> >>>>>>>>> After running test_counter:run(5000,1) on each node
> >>>>>>>>> simultaneously, and comparing the files, i get 2893
> >>>>>>>>> duplicates out of 5000.  Although the counter has been
> >>>>>>>>> incremented by 10000.  I can't repro running on a single
> >>>>>>>>> node with a large number of processes, so it seems atomic
> >>>>>>>>> in that case.
> >>>>>>>>> 
> >>>>>>>>> It is exactly the single-node case that is atomic. The
> >>>>>>>>> documentation should spell this out.
> >>>>>>>>> The User Guide does say that you lose mnesia's atomicity and
> >>>>>>>>> isolation properties if you use dirty operations, which does
> >>>>>>>>> seem to contradict what is said about
> >>>>>>>>> dirty_update_counter(), but what the Reference Manual
> >>>>>>>>> (*and* User Guide) should say is that
> >>>>>>>>> dirty_update_counter() is atomic only in a very limited
> >>>>>>>>> context. The User Guide does say something else that is,
> >>>>>>>>> strictly speaking, wrong: "It is not possible to have
> >>>>>>>>> transaction protected updates of counter records." It is of
> >>>>>>>>> course possible - they are only records after all. You can
> >>>>>>>>> do an update counter inside a transaction through a
> >>>>>>>>> combination of read() and write(). BR,
> >>>>>>>>> Ulf W
> >>>>>>>>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> >>>>>>>>> http://erlang-solutions.com
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>>> ________________________________________________________________
> >>>>>>>> erlang-questions (at) erlang.org mailing list.
> >>>>>>>> See http://www.erlang.org/faq.html
> >>>>>>>> To unsubscribe;
> >>>>>>>> mailto:erlang-questions-unsubscribe@REDACTED
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> ________________________________________________________________
> >>>>>> erlang-questions (at) erlang.org mailing list.
> >>>>>> See http://www.erlang.org/faq.html
> >>>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>> 
> >>> 
> >>> ________________________________________________________________
> >>> erlang-questions (at) erlang.org mailing list.
> >>> See http://www.erlang.org/faq.html
> >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >>> 
> >> 
> >> 
> >> ________________________________________________________________
> >> erlang-questions (at) erlang.org mailing list.
> >> See http://www.erlang.org/faq.html
> >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >> 
> >> 
> > 
> > ________________________________________________________________
> > erlang-questions (at) erlang.org mailing list.
> > See http://www.erlang.org/faq.html
> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > 
> 
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
> 
> 
> 
> 
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> 