[erlang-questions] non-atomic nature of mnesia:dirty_update_counter

Edmond Begumisa ebegumisa@REDACTED
Mon Mar 7 15:54:48 CET 2011 

On Tue, 08 Mar 2011 01:39:29 +1100, Chandru  
<chandrashekhar.mullaparthi@REDACTED> wrote:

> FWIW, we use mnesia in about 20 different systems all doing various  
> stuff in
> the network. Yes, mnesia doesn't handle partitioned networks very well,  
> but
> we've just learned to work around this. The 2G limit is not a limit at  
> all -
> just fragment your table. Our customer database of about 60 million  
> entries
> is partitioned into 128 fragments and it handles about 350 million  
> lookups a
> day, and the CPUs are rather bored on that machine.
>

At T-mobile I presume?

- Edmond -

> What it does, it does amazingly well. One can keep searching for the  
> perfect
> database, or one can use what one's got.
>
> I can't wait to see what new stuff is being done with mnesia, then I can  
> use
> it even more :-)
>
> cheers
> Chandru
>
> On 7 March 2011 12:29, Ulf Wiger <ulf.wiger@REDACTED> wrote:
>
>>
>> A key issue here is that mnesia has basically been in maintenance mode  
>> for
>> about a decade now. The main reason for this is that Ericsson itself is
>> using mnesia mainly as a configuration database, and is pretty  
>> comfortable
>> with its current capabilities.
>>
>> At the Erlang User Group meeting at QCon London this Thursday, I will  
>> give
>> a short talk about what we at Erlang Solutions are doing with mnesia  
>> right
>> now, together with some of our customers, to address the kind of  
>> problems
>> Bob is referring to.
>>
>> http://www.erlang-solutions.com/etc/usergroup/london
>>
>> BR,
>> Ulf
>>
>> On 6 Mar 2011, at 20:33, Bob Ippolito wrote:
>>
>> > Well that's the problem. There is nothing that does a better job of
>> > exactly what Mnesia does. However, with a few years of operational
>> > experience with Mnesia, you will get network partitions and/or you'll
>> > run into the 2GB limit (or run out of RAM) and if any of those things
>> > happen you'll be in pain. Node restarts also get painfully slow when
>> > the indexes are large.
>> >
>> > If there is another data model or database that can fit your
>> > application, you might be better off without Mnesia.
>> >
>> > On Sun, Mar 6, 2011 at 11:17 AM,  <bile@REDACTED> wrote:
>> >> Such as? There are lots of random DBs available but any with the ease
>> >> of use and speed mnesia seems to offer?
>> >>
>> >> On Sun, 6 Mar 2011 10:01:24 -0800
>> >> Bob Ippolito <bob@REDACTED> wrote:
>> >>
>> >>> Well, I'm sure you do know what you're doing in general, but not  
>> with
>> >>> Mnesia. Mnesia is quirky. You will probably eventually regret using
>> >>> Mnesia. We try and avoid it as much as possible, because of its
>> >>> inability to do the right thing with network partitions and the dets
>> >>> limitations for disk tables. If you can afford to, I'd look at using
>> >>> something else instead.
>> >>>
>> >>> On Sun, Mar 6, 2011 at 9:54 AM, Ken Ellis <kaellis@REDACTED> wrote:
>> >>>> Yeah but to be blunt, that's two responses with "if you know what
>> >>>> you're doing".  I know what I'm doing, I just have no idea what
>> >>>> Mnesia is doing.  The documentation is inconsistent with itself and
>> >>>> with Gudmundsson's remarks.  Perhaps switch it off for any table
>> >>>> spread across multiple nodes if no guarantees can be made in that
>> >>>> case?
>> >>>>
>> >>>>
>> >>>> On Sun, Mar 6, 2011 at 12:07 PM, Bob Ippolito <bob@REDACTED>
>> >>>> wrote:
>> >>>>> Well, they're actually quite useful (for performance) if you know
>> >>>>> what you're doing and the application can tolerate it. Often times
>> >>>>> the return value of a counter is not important. I suppose if they
>> >>>>> were ripped out, maybe fewer people would use Mnesia, which might
>> >>>>> be a net good :)
>> >>>>>
>> >>>>> On Sun, Mar 6, 2011 at 9:00 AM, Ken Ellis <kaellis@REDACTED>
>> >>>>> wrote:
>> >>>>>> Amen, the whole dirty_* set of functions should be ripped out of
>> >>>>>> the release.
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sun, Mar 6, 2011 at 11:28 AM, Dan Gudmundsson
>> >>>>>> <dgud@REDACTED> wrote:
>> >>>>>>> Nothing is promised with dirty_write don't use it.
>> >>>>>>>
>> >>>>>>> Dirty read may be ok if you know what you are doing
>> >>>>>>> but avoid dirty_write in multiple node system.
>> >>>>>>>
>> >>>>>>> /Dan
>> >>>>>>>
>> >>>>>>> On Sun, Mar 6, 2011 at 5:12 PM, Ken Ellis <kaellis@REDACTED>
>> >>>>>>> wrote:
>> >>>>>>>> Hi Ulf,
>> >>>>>>>>
>> >>>>>>>> Thanks for the clarification.  I also went back to that User
>> >>>>>>>> Guide and it does an admirable job of defining atomic as
>> >>>>>>>> meaning the ACID all-or-nothing variety and not what for
>> >>>>>>>> example you'd call atomic when incrementing counters within the
>> >>>>>>>> linux kernel.  My bad, forgot I was dealing with a database.
>> >>>>>>>>  So I think in that sense the dirty_update_counter is atomic
>> >>>>>>>> across all nodes.  I guess the question is when the users guide
>> >>>>>>>> says atomicity is lost with dirty operations, what exactly is
>> >>>>>>>> non-atomic about dirty_write or any dirty operation given that
>> >>>>>>>> "Mnesia also ensures that all replicas of a table are updated
>> >>>>>>>> if a dirty write operation is [successfully?] performed on a
>> >>>>>>>> [any?] table".  Isn't that the definition of all-or-nothing
>> >>>>>>>> atomicity? Unless that statement is inaccurate, that would seem
>> >>>>>>>> to mean all dirty_* operations are atomic, and that dirty is a
>> >>>>>>>> reference to what an RDBMAS would call dirty reads, with an
>> >>>>>>>> additional qualification that it can break isolation of
>> >>>>>>>> transactions.  Which by the way means there is no isolation
>> >>>>>>>> guarantee for any transaction, since dirty operations
>> >>>>>>>> compromise their isolation.  (IMaybe filthy_* would be a better
>> >>>>>>>> prefix :).  But I suppose for now I'll take the statement about
>> >>>>>>>> lack of atomicity as true, and that dirty writes can be only
>> >>>>>>>> partially executed and can leave tables on different nodes in
>> >>>>>>>> an inconsistent state.
>> >>>>>>>>
>> >>>>>>>> Ken
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Sun, Mar 6, 2011 at 2:19 AM, Ulf Wiger
>> >>>>>>>> <ulf.wiger@REDACTED> wrote:
>> >>>>>>>>>
>> >>>>>>>>> On 6 Mar 2011, at 06:54, Ken Ellis wrote:
>> >>>>>>>>>
>> >>>>>>>>> After running test_counter:run(5000,1) on each node
>> >>>>>>>>> simultaneously, and comparing the files, i get 2893 duplicates
>> >>>>>>>>> out of 5000.  Although the counter has been incremented by
>> >>>>>>>>> 10000.  I can't repro running on a single node with a large
>> >>>>>>>>> number of processes, so it seems atomic in that case.
>> >>>>>>>>>
>> >>>>>>>>> It is exactly the single-node case that is atomic. The
>> >>>>>>>>> documentation should spell this out.
>> >>>>>>>>> The User Guide does say that you lose mnesia's atomicity and
>> >>>>>>>>> isolation properties if you use dirty operations, which does
>> >>>>>>>>> seem to contradict what is said about dirty_update_counter(),
>> >>>>>>>>> but what the Reference Manual (*and* User Guide) should say is
>> >>>>>>>>> that dirty_update_counter() is atomic only in a very limited
>> >>>>>>>>> context. The User Guide does say something else that is,
>> >>>>>>>>> strictly speaking, wrong: "It is not possible to have
>> >>>>>>>>> transaction protected updates of counter records."
>> >>>>>>>>> It is of course possible - they are only records after all.
>> >>>>>>>>> You can do an update counter inside a transaction through a
>> >>>>>>>>> combination of read() and write().
>> >>>>>>>>> BR,
>> >>>>>>>>> Ulf W
>> >>>>>>>>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
>> >>>>>>>>> http://erlang-solutions.com
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>  
>> ________________________________________________________________
>> >>>>>>>> erlang-questions (at) erlang.org mailing list.
>> >>>>>>>> See http://www.erlang.org/faq.html
>> >>>>>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> ________________________________________________________________
>> >>>>>> erlang-questions (at) erlang.org mailing list.
>> >>>>>> See http://www.erlang.org/faq.html
>> >>>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>> ________________________________________________________________
>> >>> erlang-questions (at) erlang.org mailing list.
>> >>> See http://www.erlang.org/faq.html
>> >>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>> >>>
>> >>
>> >>
>> >> ________________________________________________________________
>> >> erlang-questions (at) erlang.org mailing list.
>> >> See http://www.erlang.org/faq.html
>> >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>> >>
>> >>
>> >
>> > ________________________________________________________________
>> > erlang-questions (at) erlang.org mailing list.
>> > See http://www.erlang.org/faq.html
>> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>> >
>>
>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
>> http://erlang-solutions.com
>>
>>
>>
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>
>>


-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/

More information about the erlang-questions mailing list