[erlang-questions] non-atomic nature of mnesia:dirty_update_counter

Mon Mar 7 15:39:29 CET 2011

FWIW, we use mnesia in about 20 different systems all doing various stuff in
the network. Yes, mnesia doesn't handle partitioned networks very well, but
we've just learned to work around this. The 2G limit is not a limit at all -
just fragment your table. Our customer database of about 60 million entries
is partitioned into 128 fragments and it handles about 350 million lookups a
day, and the CPUs are rather bored on that machine.

What it does, it does amazingly well. One can keep searching for the perfect
database, or one can use what one's got.

I can't wait to see what new stuff is being done with mnesia, then I can use
it even more :-)

cheers
Chandru

On 7 March 2011 12:29, Ulf Wiger <ulf.wiger@REDACTED> wrote:

>
> A key issue here is that mnesia has basically been in maintenance mode for
> about a decade now. The main reason for this is that Ericsson itself is
> using mnesia mainly as a configuration database, and is pretty comfortable
> with its current capabilities.
>
> At the Erlang User Group meeting at QCon London this Thursday, I will give
> a short talk about what we at Erlang Solutions are doing with mnesia right
> now, together with some of our customers, to address the kind of problems
> Bob is referring to.
>
> http://www.erlang-solutions.com/etc/usergroup/london
>
> BR,
> Ulf
>
> On 6 Mar 2011, at 20:33, Bob Ippolito wrote:
>
> > Well that's the problem. There is nothing that does a better job of
> > exactly what Mnesia does. However, with a few years of operational
> > experience with Mnesia, you will get network partitions and/or you'll
> > run into the 2GB limit (or run out of RAM) and if any of those things
> > happen you'll be in pain. Node restarts also get painfully slow when
> > the indexes are large.
> >
> > If there is another data model or database that can fit your
> > application, you might be better off without Mnesia.
> >
> > On Sun, Mar 6, 2011 at 11:17 AM,  <bile@REDACTED> wrote:
> >> Such as? There are lots of random DBs available but any with the ease
> >> of use and speed mnesia seems to offer?
> >>
> >> On Sun, 6 Mar 2011 10:01:24 -0800
> >> Bob Ippolito <bob@REDACTED> wrote:
> >>
> >>> Well, I'm sure you do know what you're doing in general, but not with
> >>> Mnesia. Mnesia is quirky. You will probably eventually regret using
> >>> Mnesia. We try and avoid it as much as possible, because of its
> >>> inability to do the right thing with network partitions and the dets
> >>> limitations for disk tables. If you can afford to, I'd look at using
> >>> something else instead.
> >>>
> >>> On Sun, Mar 6, 2011 at 9:54 AM, Ken Ellis <kaellis@REDACTED> wrote:
> >>>> Yeah but to be blunt, that's two responses with "if you know what
> >>>> you're doing".  I know what I'm doing, I just have no idea what
> >>>> Mnesia is doing.  The documentation is inconsistent with itself and
> >>>> with Gudmundsson's remarks.  Perhaps switch it off for any table
> >>>> spread across multiple nodes if no guarantees can be made in that
> >>>> case?
> >>>>
> >>>>
> >>>> On Sun, Mar 6, 2011 at 12:07 PM, Bob Ippolito <bob@REDACTED>
> >>>> wrote:
> >>>>> Well, they're actually quite useful (for performance) if you know
> >>>>> what you're doing and the application can tolerate it. Often times
> >>>>> the return value of a counter is not important. I suppose if they
> >>>>> were ripped out, maybe fewer people would use Mnesia, which might
> >>>>> be a net good :)
> >>>>>
> >>>>> On Sun, Mar 6, 2011 at 9:00 AM, Ken Ellis <kaellis@REDACTED>
> >>>>> wrote:
> >>>>>> Amen, the whole dirty_* set of functions should be ripped out of
> >>>>>> the release.
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Mar 6, 2011 at 11:28 AM, Dan Gudmundsson
> >>>>>> <dgud@REDACTED> wrote:
> >>>>>>> Nothing is promised with dirty_write don't use it.
> >>>>>>>
> >>>>>>> Dirty read may be ok if you know what you are doing
> >>>>>>> but avoid dirty_write in multiple node system.
> >>>>>>>
> >>>>>>> /Dan
> >>>>>>>
> >>>>>>> On Sun, Mar 6, 2011 at 5:12 PM, Ken Ellis <kaellis@REDACTED>
> >>>>>>> wrote:
> >>>>>>>> Hi Ulf,
> >>>>>>>>
> >>>>>>>> Thanks for the clarification.  I also went back to that User
> >>>>>>>> Guide and it does an admirable job of defining atomic as
> >>>>>>>> meaning the ACID all-or-nothing variety and not what for
> >>>>>>>> example you'd call atomic when incrementing counters within the
> >>>>>>>> linux kernel.  My bad, forgot I was dealing with a database.
> >>>>>>>>  So I think in that sense the dirty_update_counter is atomic
> >>>>>>>> across all nodes.  I guess the question is when the users guide
> >>>>>>>> says atomicity is lost with dirty operations, what exactly is
> >>>>>>>> non-atomic about dirty_write or any dirty operation given that
> >>>>>>>> "Mnesia also ensures that all replicas of a table are updated
> >>>>>>>> if a dirty write operation is [successfully?] performed on a
> >>>>>>>> [any?] table".  Isn't that the definition of all-or-nothing
> >>>>>>>> atomicity? Unless that statement is inaccurate, that would seem
> >>>>>>>> to mean all dirty_* operations are atomic, and that dirty is a
> >>>>>>>> reference to what an RDBMAS would call dirty reads, with an
> >>>>>>>> additional qualification that it can break isolation of
> >>>>>>>> transactions.  Which by the way means there is no isolation
> >>>>>>>> guarantee for any transaction, since dirty operations
> >>>>>>>> compromise their isolation.  (IMaybe filthy_* would be a better
> >>>>>>>> prefix :).  But I suppose for now I'll take the statement about
> >>>>>>>> lack of atomicity as true, and that dirty writes can be only
> >>>>>>>> partially executed and can leave tables on different nodes in
> >>>>>>>> an inconsistent state.
> >>>>>>>>
> >>>>>>>> Ken
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Sun, Mar 6, 2011 at 2:19 AM, Ulf Wiger
> >>>>>>>> <ulf.wiger@REDACTED> wrote:
> >>>>>>>>>
> >>>>>>>>> On 6 Mar 2011, at 06:54, Ken Ellis wrote:
> >>>>>>>>>
> >>>>>>>>> After running test_counter:run(5000,1) on each node
> >>>>>>>>> simultaneously, and comparing the files, i get 2893 duplicates
> >>>>>>>>> out of 5000.  Although the counter has been incremented by
> >>>>>>>>> 10000.  I can't repro running on a single node with a large
> >>>>>>>>> number of processes, so it seems atomic in that case.
> >>>>>>>>>
> >>>>>>>>> It is exactly the single-node case that is atomic. The
> >>>>>>>>> documentation should spell this out.
> >>>>>>>>> The User Guide does say that you lose mnesia's atomicity and
> >>>>>>>>> isolation properties if you use dirty operations, which does
> >>>>>>>>> seem to contradict what is said about dirty_update_counter(),
> >>>>>>>>> but what the Reference Manual (*and* User Guide) should say is
> >>>>>>>>> that dirty_update_counter() is atomic only in a very limited
> >>>>>>>>> context. The User Guide does say something else that is,
> >>>>>>>>> strictly speaking, wrong: "It is not possible to have
> >>>>>>>>> transaction protected updates of counter records."
> >>>>>>>>> It is of course possible - they are only records after all.
> >>>>>>>>> You can do an update counter inside a transaction through a
> >>>>>>>>> combination of read() and write().
> >>>>>>>>> BR,
> >>>>>>>>> Ulf W
> >>>>>>>>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> >>>>>>>>> http://erlang-solutions.com
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> ________________________________________________________________
> >>>>>>>> erlang-questions (at) erlang.org mailing list.
> >>>>>>>> See http://www.erlang.org/faq.html
> >>>>>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> ________________________________________________________________
> >>>>>> erlang-questions (at) erlang.org mailing list.
> >>>>>> See http://www.erlang.org/faq.html
> >>>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>> ________________________________________________________________
> >>> erlang-questions (at) erlang.org mailing list.
> >>> See http://www.erlang.org/faq.html
> >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >>>
> >>
> >>
> >> ________________________________________________________________
> >> erlang-questions (at) erlang.org mailing list.
> >> See http://www.erlang.org/faq.html
> >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >>
> >>
> >
> > ________________________________________________________________
> > erlang-questions (at) erlang.org mailing list.
> > See http://www.erlang.org/faq.html
> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >
>
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
>
>
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>