Mnesia and Oracle

[Yariv Sadan]

Hi,

Thanks for the feedback! The reason I intended on using both Mnesia
and MySQL is because they would store different kinds of data: MySQL
would store long-term, high-volume data and Mnesia would store session
data. I think storing session data in Mnesia makes sense because
Mnesia makes it easy to replicate this data, which would make the
architecture resilient to a front-end server crash (as opposed to
storing session data in ETS).

IIRC, ejabberd does something similar when using a SQL RDMBS.

Could I store session data in MySQL? Yes, and I'm still considering
this option, actually. However, the advantage of using Mnesia is I can
easily run it on the (Yaws) front-end boxes, which would make session
data access much faster.

Another problem with dets fragmentation, besides the long time it
takes to write it to disk, is the growing memory consumption. This can
affect performance if it gets out of hand. AFAIK, the only way to
defragment a dets table is to take if offline and reopen it with a
forced repair flag, which could take a long time (30 minutes?).

Thank
Yariv



Interesting... maybe I should have explained my reasoning better:

On 8/10/06, Valentin Micic <valentin@REDACTED> wrote:
> Well, we're using mnesia with dets stroring there-about 80GB data set.
> IMHO, if you want to develop a cache to front RDBMS, what would mnesia do
> for you that ETS wouldn't? Other than processing overhead, of course.
> And my main issue would be data-integrity.
>
> V.
>
> ----- Original Message -----
> From: "Ryan Rawson" <ryanobjc@REDACTED>
> To: "Valentin Micic" <valentin@REDACTED>
> Cc: "Yariv Sadan" <yarivvv@REDACTED>; "Inswitch Solutions"
> <erlang@REDACTED>; <erlang-questions@REDACTED>
> Sent: Thursday, August 10, 2006 9:54 AM
> Subject: Re: Mnesia and Oracle
>
>
> > What about my 50 gb data set?
> >
> > What about my 100 gb data set?
> >
> > Ultimately I think a major value is using mnesia as a coherent cache
> > and some SQL backend as a master storage.
> >
> > -ryan
> >
> >
> > On 8/10/06, Valentin Micic <valentin@REDACTED> wrote:
> >> I do not think combining mnesia and MySQL is a good choice, no matter
> >> what
> >> justification one may put forward. If nothing else, it complicates
> >> maintenance, confuses people, and may (potentialy) compromise both
> >> products... not to mention a data integrity issues intrinsic to such an
> >> implementation.
> >> Mnesia's dets tables work reasonably well, even with huge data sets. The
> >> problem(s) starts when you start deleting records -- free list grows to
> >> the
> >> point that (IMHO) takes quite a long time to write it to disk. This
> >> becomes
> >> more obvious when one uses a lots of fragments. Thus, even if you shut
> >> the
> >> database regularly, it might corrupt few fragments. My guess: mensia
> >> controller does not give enough time to all dets processes (one per
> >> fragment) to flush their respective free lists to dist.
> >>
> >> A question for Erlang/OTP team: how can one prevent this from happening?
> >>
> >> I'm looking more (and more) to Berkeley DB for storage. Anybody, how does
> >> it
> >> compare to dets?
> >>
> >> Valentin.
> >>
> >>
> >> ----- Original Message -----
> >> From: "Yariv Sadan" <yarivvv@REDACTED>
> >> To: "Inswitch Solutions" <erlang@REDACTED>
> >> Cc: <erlang-questions@REDACTED>
> >> Sent: Tuesday, August 08, 2006 11:51 PM
> >> Subject: Re: Mnesia and Oracle
> >>
> >>
> >> > Hi Eduardo,
> >> >
> >> > I haven't had production experience with these databases, but there
> >> > are a couple of things I found by research that are keeping me from
> >> > using Mnesia exclusively in my application:
> >> >
> >> > - Mnesia disc storage, based on dets, has a couple of drawbacks when
> >> > handling very large (many gigs) datasets: potentially long repair
> >> > times and memory consumption that grows with data fragmentation.
> >> > - QLC, the query engine for Mnesia, doesn't currently optimize joins.
> >> > If your queries involve joining big tables, they can take a long time
> >> > to execute.
> >> >
> >> > The join optimizations are planned for a future R11 OTP release, but
> >> > there are no plans to change dets AFAIK.
> >> >
> >> > Depending on your application, these issues may not be a big problem.
> >> > For the application I'm building, I'm planning on using both MySQL and
> >> > Mnesia, where MySQL will be used for storing high-volume data and
> >> > Mnesia for "live" session data.
> >> >
> >> > Hope this helps!
> >> >
> >> > Regards,
> >> > Yariv
> >> >
> >> > On 8/8/06, Inswitch Solutions <erlang@REDACTED> wrote:
> >> >>
> >> >>
> >> >>
> >> >> Hi,
> >> >>
> >> >> I'm already working with Oracle and Mnesia, and I'd  like to hear
> >> >> experiences of the Erlang community about these  databases.
> >> >> When deciding over Oracle or Mnesia database for an  Erlang, or non
> >> >> Erlang, based real-time system which factors are in favour  in one
> >> >> over
> >> >> the other (performance...?) ?.
> >> >>
> >> >>
> >> >> thanks, Eduardo
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> Prepaid Expertise -        Programmable Switches
> >> >> Powered by Ericsson Licensed Technology
> >> >> Eng.        Eduardo Figoli - Development Center - IN Switch Solutions
> >> >> Inc.
> >> >> Headquarters - Miami-U.S.A. Tel: 1305-3578076 Fax:        1305-7686260
> >> >> Development Center - Montevideo - Uruguay Tel/Fax:        5982-7104457
> >> >> e-mail: eduardo@REDACTED
> >> >>
> >> >>
> >>
> >>
> >
>
>