[erlang-questions] non-atomic nature of mnesia:dirty_update_counter

Mon Mar 7 16:08:46 CET 2011

Thanks Ulf, that's a very useful bit of information.  A DBMS in maintenance
mode for 10 years really gets me excited.  We have very simple needs for
Mnesia, and are using a separate data store for the eventual output of our
system.  But databases being what they are I don't want to rely on them
being up any more than I have to, so mnesia seems to fit the bill.
On Mar 7, 2011 7:31 AM, "Ulf Wiger" <ulf.wiger@REDACTED> wrote:
>
> A key issue here is that mnesia has basically been in maintenance mode for
about a decade now. The main reason for this is that Ericsson itself is
using mnesia mainly as a configuration database, and is pretty comfortable
with its current capabilities.
>
> At the Erlang User Group meeting at QCon London this Thursday, I will give
a short talk about what we at Erlang Solutions are doing with mnesia right
now, together with some of our customers, to address the kind of problems
Bob is referring to.
>
> http://www.erlang-solutions.com/etc/usergroup/london
>
> BR,
> Ulf
>
> On 6 Mar 2011, at 20:33, Bob Ippolito wrote:
>
>> Well that's the problem. There is nothing that does a better job of
>> exactly what Mnesia does. However, with a few years of operational
>> experience with Mnesia, you will get network partitions and/or you'll
>> run into the 2GB limit (or run out of RAM) and if any of those things
>> happen you'll be in pain. Node restarts also get painfully slow when
>> the indexes are large.
>>
>> If there is another data model or database that can fit your
>> application, you might be better off without Mnesia.
>>
>> On Sun, Mar 6, 2011 at 11:17 AM, <bile@REDACTED> wrote:
>>> Such as? There are lots of random DBs available but any with the ease
>>> of use and speed mnesia seems to offer?
>>>
>>> On Sun, 6 Mar 2011 10:01:24 -0800
>>> Bob Ippolito <bob@REDACTED> wrote:
>>>
>>>> Well, I'm sure you do know what you're doing in general, but not with
>>>> Mnesia. Mnesia is quirky. You will probably eventually regret using
>>>> Mnesia. We try and avoid it as much as possible, because of its
>>>> inability to do the right thing with network partitions and the dets
>>>> limitations for disk tables. If you can afford to, I'd look at using
>>>> something else instead.
>>>>
>>>> On Sun, Mar 6, 2011 at 9:54 AM, Ken Ellis <kaellis@REDACTED> wrote:
>>>>> Yeah but to be blunt, that's two responses with "if you know what
>>>>> you're doing". I know what I'm doing, I just have no idea what
>>>>> Mnesia is doing. The documentation is inconsistent with itself and
>>>>> with Gudmundsson's remarks. Perhaps switch it off for any table
>>>>> spread across multiple nodes if no guarantees can be made in that
>>>>> case?
>>>>>
>>>>>
>>>>> On Sun, Mar 6, 2011 at 12:07 PM, Bob Ippolito <bob@REDACTED>
>>>>> wrote:
>>>>>> Well, they're actually quite useful (for performance) if you know
>>>>>> what you're doing and the application can tolerate it. Often times
>>>>>> the return value of a counter is not important. I suppose if they
>>>>>> were ripped out, maybe fewer people would use Mnesia, which might
>>>>>> be a net good :)
>>>>>>
>>>>>> On Sun, Mar 6, 2011 at 9:00 AM, Ken Ellis <kaellis@REDACTED>
>>>>>> wrote:
>>>>>>> Amen, the whole dirty_* set of functions should be ripped out of
>>>>>>> the release.
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 6, 2011 at 11:28 AM, Dan Gudmundsson
>>>>>>> <dgud@REDACTED> wrote:
>>>>>>>> Nothing is promised with dirty_write don't use it.
>>>>>>>>
>>>>>>>> Dirty read may be ok if you know what you are doing
>>>>>>>> but avoid dirty_write in multiple node system.
>>>>>>>>
>>>>>>>> /Dan
>>>>>>>>
>>>>>>>> On Sun, Mar 6, 2011 at 5:12 PM, Ken Ellis <kaellis@REDACTED>
>>>>>>>> wrote:
>>>>>>>>> Hi Ulf,
>>>>>>>>>
>>>>>>>>> Thanks for the clarification. I also went back to that User
>>>>>>>>> Guide and it does an admirable job of defining atomic as
>>>>>>>>> meaning the ACID all-or-nothing variety and not what for
>>>>>>>>> example you'd call atomic when incrementing counters within the
>>>>>>>>> linux kernel. My bad, forgot I was dealing with a database.
>>>>>>>>> So I think in that sense the dirty_update_counter is atomic
>>>>>>>>> across all nodes. I guess the question is when the users guide
>>>>>>>>> says atomicity is lost with dirty operations, what exactly is
>>>>>>>>> non-atomic about dirty_write or any dirty operation given that
>>>>>>>>> "Mnesia also ensures that all replicas of a table are updated
>>>>>>>>> if a dirty write operation is [successfully?] performed on a
>>>>>>>>> [any?] table". Isn't that the definition of all-or-nothing
>>>>>>>>> atomicity? Unless that statement is inaccurate, that would seem
>>>>>>>>> to mean all dirty_* operations are atomic, and that dirty is a
>>>>>>>>> reference to what an RDBMAS would call dirty reads, with an
>>>>>>>>> additional qualification that it can break isolation of
>>>>>>>>> transactions. Which by the way means there is no isolation
>>>>>>>>> guarantee for any transaction, since dirty operations
>>>>>>>>> compromise their isolation. (IMaybe filthy_* would be a better
>>>>>>>>> prefix :). But I suppose for now I'll take the statement about
>>>>>>>>> lack of atomicity as true, and that dirty writes can be only
>>>>>>>>> partially executed and can leave tables on different nodes in
>>>>>>>>> an inconsistent state.
>>>>>>>>>
>>>>>>>>> Ken
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 6, 2011 at 2:19 AM, Ulf Wiger
>>>>>>>>> <ulf.wiger@REDACTED> wrote:
>>>>>>>>>>
>>>>>>>>>> On 6 Mar 2011, at 06:54, Ken Ellis wrote:
>>>>>>>>>>
>>>>>>>>>> After running test_counter:run(5000,1) on each node
>>>>>>>>>> simultaneously, and comparing the files, i get 2893 duplicates
>>>>>>>>>> out of 5000. Although the counter has been incremented by
>>>>>>>>>> 10000. I can't repro running on a single node with a large
>>>>>>>>>> number of processes, so it seems atomic in that case.
>>>>>>>>>>
>>>>>>>>>> It is exactly the single-node case that is atomic. The
>>>>>>>>>> documentation should spell this out.
>>>>>>>>>> The User Guide does say that you lose mnesia's atomicity and
>>>>>>>>>> isolation properties if you use dirty operations, which does
>>>>>>>>>> seem to contradict what is said about dirty_update_counter(),
>>>>>>>>>> but what the Reference Manual (*and* User Guide) should say is
>>>>>>>>>> that dirty_update_counter() is atomic only in a very limited
>>>>>>>>>> context. The User Guide does say something else that is,
>>>>>>>>>> strictly speaking, wrong: "It is not possible to have
>>>>>>>>>> transaction protected updates of counter records."
>>>>>>>>>> It is of course possible - they are only records after all.
>>>>>>>>>> You can do an update counter inside a transaction through a
>>>>>>>>>> combination of read() and write().
>>>>>>>>>> BR,
>>>>>>>>>> Ulf W
>>>>>>>>>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
>>>>>>>>>> http://erlang-solutions.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ________________________________________________________________
>>>>>>>>> erlang-questions (at) erlang.org mailing list.
>>>>>>>>> See http://www.erlang.org/faq.html
>>>>>>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ________________________________________________________________
>>>>>>> erlang-questions (at) erlang.org mailing list.
>>>>>>> See http://www.erlang.org/faq.html
>>>>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ________________________________________________________________
>>>> erlang-questions (at) erlang.org mailing list.
>>>> See http://www.erlang.org/faq.html
>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>>>
>>>
>>>
>>> ________________________________________________________________
>>> erlang-questions (at) erlang.org mailing list.
>>> See http://www.erlang.org/faq.html
>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>>
>>>
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>
>
> Ulf Wiger, CTO, Erlang Solutions, Ltd.
> http://erlang-solutions.com
>
>
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>