Mnesia and Oracle

[Valentin Micic]

So, I guess, it all comes down to what are you trying to do ;-) and I 
certainly do not know what you mean by session data and a need to replicate 
it: my experience with mnesia replication, however,  is somewhat negative 
with respect to volatile network environment. IMHO, if your data model is 
relatively simple, you would be better of doing a custom replication.

As far as increased memeory consumption with dets data fragmentation (i.e. 
imposed by record deletions), you are right: it is stated in a manual that 
this might be a concern. However, one should also consider a time when this 
was written -- memory concerns from 10 years ago are certainly not the same 
today. I would even dare to say that memory is relatively cheap.

V.

----- Original Message ----- 
From: "Yariv Sadan" <yarivvv@REDACTED>
To: "Valentin Micic" <valentin@REDACTED>
Cc: "Ryan Rawson" <ryanobjc@REDACTED>; "Inswitch Solutions" 
<erlang@REDACTED>; <erlang-questions@REDACTED>
Sent: Thursday, August 10, 2006 2:07 PM
Subject: Re: Mnesia and Oracle


> Hi,
>
> Thanks for the feedback! The reason I intended on using both Mnesia
> and MySQL is because they would store different kinds of data: MySQL
> would store long-term, high-volume data and Mnesia would store session
> data. I think storing session data in Mnesia makes sense because
> Mnesia makes it easy to replicate this data, which would make the
> architecture resilient to a front-end server crash (as opposed to
> storing session data in ETS).
>
> IIRC, ejabberd does something similar when using a SQL RDMBS.
>
> Could I store session data in MySQL? Yes, and I'm still considering
> this option, actually. However, the advantage of using Mnesia is I can
> easily run it on the (Yaws) front-end boxes, which would make session
> data access much faster.
>
> Another problem with dets fragmentation, besides the long time it
> takes to write it to disk, is the growing memory consumption. This can
> affect performance if it gets out of hand. AFAIK, the only way to
> defragment a dets table is to take if offline and reopen it with a
> forced repair flag, which could take a long time (30 minutes?).
>
> Thank
> Yariv
>
>
>
> Interesting... maybe I should have explained my reasoning better:
>
> On 8/10/06, Valentin Micic <valentin@REDACTED> wrote:
>> Well, we're using mnesia with dets stroring there-about 80GB data set.
>> IMHO, if you want to develop a cache to front RDBMS, what would mnesia do
>> for you that ETS wouldn't? Other than processing overhead, of course.
>> And my main issue would be data-integrity.
>>
>> V.
>>
>> ----- Original Message -----
>> From: "Ryan Rawson" <ryanobjc@REDACTED>
>> To: "Valentin Micic" <valentin@REDACTED>
>> Cc: "Yariv Sadan" <yarivvv@REDACTED>; "Inswitch Solutions"
>> <erlang@REDACTED>; <erlang-questions@REDACTED>
>> Sent: Thursday, August 10, 2006 9:54 AM
>> Subject: Re: Mnesia and Oracle
>>
>>
>> > What about my 50 gb data set?
>> >
>> > What about my 100 gb data set?
>> >
>> > Ultimately I think a major value is using mnesia as a coherent cache
>> > and some SQL backend as a master storage.
>> >
>> > -ryan
>> >
>> >
>> > On 8/10/06, Valentin Micic <valentin@REDACTED> wrote:
>> >> I do not think combining mnesia and MySQL is a good choice, no matter
>> >> what
>> >> justification one may put forward. If nothing else, it complicates
>> >> maintenance, confuses people, and may (potentialy) compromise both
>> >> products... not to mention a data integrity issues intrinsic to such 
>> >> an
>> >> implementation.
>> >> Mnesia's dets tables work reasonably well, even with huge data sets. 
>> >> The
>> >> problem(s) starts when you start deleting records -- free list grows 
>> >> to
>> >> the
>> >> point that (IMHO) takes quite a long time to write it to disk. This
>> >> becomes
>> >> more obvious when one uses a lots of fragments. Thus, even if you shut
>> >> the
>> >> database regularly, it might corrupt few fragments. My guess: mensia
>> >> controller does not give enough time to all dets processes (one per
>> >> fragment) to flush their respective free lists to dist.
>> >>
>> >> A question for Erlang/OTP team: how can one prevent this from 
>> >> happening?
>> >>
>> >> I'm looking more (and more) to Berkeley DB for storage. Anybody, how 
>> >> does
>> >> it
>> >> compare to dets?
>> >>
>> >> Valentin.
>> >>
>> >>
>> >> ----- Original Message -----
>> >> From: "Yariv Sadan" <yarivvv@REDACTED>
>> >> To: "Inswitch Solutions" <erlang@REDACTED>
>> >> Cc: <erlang-questions@REDACTED>
>> >> Sent: Tuesday, August 08, 2006 11:51 PM
>> >> Subject: Re: Mnesia and Oracle
>> >>
>> >>
>> >> > Hi Eduardo,
>> >> >
>> >> > I haven't had production experience with these databases, but there
>> >> > are a couple of things I found by research that are keeping me from
>> >> > using Mnesia exclusively in my application:
>> >> >
>> >> > - Mnesia disc storage, based on dets, has a couple of drawbacks when
>> >> > handling very large (many gigs) datasets: potentially long repair
>> >> > times and memory consumption that grows with data fragmentation.
>> >> > - QLC, the query engine for Mnesia, doesn't currently optimize 
>> >> > joins.
>> >> > If your queries involve joining big tables, they can take a long 
>> >> > time
>> >> > to execute.
>> >> >
>> >> > The join optimizations are planned for a future R11 OTP release, but
>> >> > there are no plans to change dets AFAIK.
>> >> >
>> >> > Depending on your application, these issues may not be a big 
>> >> > problem.
>> >> > For the application I'm building, I'm planning on using both MySQL 
>> >> > and
>> >> > Mnesia, where MySQL will be used for storing high-volume data and
>> >> > Mnesia for "live" session data.
>> >> >
>> >> > Hope this helps!
>> >> >
>> >> > Regards,
>> >> > Yariv
>> >> >
>> >> > On 8/8/06, Inswitch Solutions <erlang@REDACTED> wrote:
>> >> >>
>> >> >>
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I'm already working with Oracle and Mnesia, and I'd  like to hear
>> >> >> experiences of the Erlang community about these  databases.
>> >> >> When deciding over Oracle or Mnesia database for an  Erlang, or non
>> >> >> Erlang, based real-time system which factors are in favour  in one
>> >> >> over
>> >> >> the other (performance...?) ?.
>> >> >>
>> >> >>
>> >> >> thanks, Eduardo
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> Prepaid Expertise -        Programmable Switches
>> >> >> Powered by Ericsson Licensed Technology
>> >> >> Eng.        Eduardo Figoli - Development Center - IN Switch 
>> >> >> Solutions
>> >> >> Inc.
>> >> >> Headquarters - Miami-U.S.A. Tel: 1305-3578076 Fax: 
>> >> >> 1305-7686260
>> >> >> Development Center - Montevideo - Uruguay Tel/Fax: 
>> >> >> 5982-7104457
>> >> >> e-mail: eduardo@REDACTED
>> >> >>
>> >> >>
>> >>
>> >>
>> >
>>
>>
>