Mnesia and Oracle

[Yariv Sadan]

Hi,

On 8/10/06, Valentin Micic <valentin@REDACTED> wrote:
> So, I guess, it all comes down to what are you trying to do ;-) and I
> certainly do not know what you mean by session data and a need to replicate
> it: my experience with mnesia replication, however,  is somewhat negative
> with respect to volatile network environment. IMHO, if your data model is
> relatively simple, you would be better of doing a custom replication.

I'm building a webapp with a backend MySQL database. By "session data"
I mean the data that is associated for each user by a cookie.
Currently, Yaws has an API for storing session data in ETS, but I want
to use Mnesia instead so if a web server crashes, the session data
isn't lost. I hope this makes sense... :)

>
> As far as increased memeory consumption with dets data fragmentation (i.e.
> imposed by record deletions), you are right: it is stated in a manual that
> this might be a concern. However, one should also consider a time when this
> was written -- memory concerns from 10 years ago are certainly not the same
> today. I would even dare to say that memory is relatively cheap.

One problem I have with dets is simply the lack of concrete estimates
on the fragmentation cost and repair time in different scenarios. I
know that memory is "relatively cheap" and that dets repair time is
"shorter than it used to be," but I don't know for certain the degree
to which these issues will affect my app and that they won't become a
major pain in the ass a few months down the road. Since switching
databases -- if the need arises -- would be a huge pain, I prefer to
suck it up now and use MySQL just to avoid that risk.

Cheers
Yariv

>
> V.
>
> ----- Original Message -----
> From: "Yariv Sadan" <yarivvv@REDACTED>
> To: "Valentin Micic" <valentin@REDACTED>
> Cc: "Ryan Rawson" <ryanobjc@REDACTED>; "Inswitch Solutions"
> <erlang@REDACTED>; <erlang-questions@REDACTED>
> Sent: Thursday, August 10, 2006 2:07 PM
> Subject: Re: Mnesia and Oracle
>
>
> > Hi,
> >
> > Thanks for the feedback! The reason I intended on using both Mnesia
> > and MySQL is because they would store different kinds of data: MySQL
> > would store long-term, high-volume data and Mnesia would store session
> > data. I think storing session data in Mnesia makes sense because
> > Mnesia makes it easy to replicate this data, which would make the
> > architecture resilient to a front-end server crash (as opposed to
> > storing session data in ETS).
> >
> > IIRC, ejabberd does something similar when using a SQL RDMBS.
> >
> > Could I store session data in MySQL? Yes, and I'm still considering
> > this option, actually. However, the advantage of using Mnesia is I can
> > easily run it on the (Yaws) front-end boxes, which would make session
> > data access much faster.
> >
> > Another problem with dets fragmentation, besides the long time it
> > takes to write it to disk, is the growing memory consumption. This can
> > affect performance if it gets out of hand. AFAIK, the only way to
> > defragment a dets table is to take if offline and reopen it with a
> > forced repair flag, which could take a long time (30 minutes?).
> >
> > Thank
> > Yariv
> >
> >
> >
> > Interesting... maybe I should have explained my reasoning better:
> >
> > On 8/10/06, Valentin Micic <valentin@REDACTED> wrote:
> >> Well, we're using mnesia with dets stroring there-about 80GB data set.
> >> IMHO, if you want to develop a cache to front RDBMS, what would mnesia do
> >> for you that ETS wouldn't? Other than processing overhead, of course.
> >> And my main issue would be data-integrity.
> >>
> >> V.
> >>
> >> ----- Original Message -----
> >> From: "Ryan Rawson" <ryanobjc@REDACTED>
> >> To: "Valentin Micic" <valentin@REDACTED>
> >> Cc: "Yariv Sadan" <yarivvv@REDACTED>; "Inswitch Solutions"
> >> <erlang@REDACTED>; <erlang-questions@REDACTED>
> >> Sent: Thursday, August 10, 2006 9:54 AM
> >> Subject: Re: Mnesia and Oracle
> >>
> >>
> >> > What about my 50 gb data set?
> >> >
> >> > What about my 100 gb data set?
> >> >
> >> > Ultimately I think a major value is using mnesia as a coherent cache
> >> > and some SQL backend as a master storage.
> >> >
> >> > -ryan
> >> >
> >> >
> >> > On 8/10/06, Valentin Micic <valentin@REDACTED> wrote:
> >> >> I do not think combining mnesia and MySQL is a good choice, no matter
> >> >> what
> >> >> justification one may put forward. If nothing else, it complicates
> >> >> maintenance, confuses people, and may (potentialy) compromise both
> >> >> products... not to mention a data integrity issues intrinsic to such
> >> >> an
> >> >> implementation.
> >> >> Mnesia's dets tables work reasonably well, even with huge data sets.
> >> >> The
> >> >> problem(s) starts when you start deleting records -- free list grows
> >> >> to
> >> >> the
> >> >> point that (IMHO) takes quite a long time to write it to disk. This
> >> >> becomes
> >> >> more obvious when one uses a lots of fragments. Thus, even if you shut
> >> >> the
> >> >> database regularly, it might corrupt few fragments. My guess: mensia
> >> >> controller does not give enough time to all dets processes (one per
> >> >> fragment) to flush their respective free lists to dist.
> >> >>
> >> >> A question for Erlang/OTP team: how can one prevent this from
> >> >> happening?
> >> >>
> >> >> I'm looking more (and more) to Berkeley DB for storage. Anybody, how
> >> >> does
> >> >> it
> >> >> compare to dets?
> >> >>
> >> >> Valentin.
> >> >>
> >> >>
> >> >> ----- Original Message -----
> >> >> From: "Yariv Sadan" <yarivvv@REDACTED>
> >> >> To: "Inswitch Solutions" <erlang@REDACTED>
> >> >> Cc: <erlang-questions@REDACTED>
> >> >> Sent: Tuesday, August 08, 2006 11:51 PM
> >> >> Subject: Re: Mnesia and Oracle
> >> >>
> >> >>
> >> >> > Hi Eduardo,
> >> >> >
> >> >> > I haven't had production experience with these databases, but there
> >> >> > are a couple of things I found by research that are keeping me from
> >> >> > using Mnesia exclusively in my application:
> >> >> >
> >> >> > - Mnesia disc storage, based on dets, has a couple of drawbacks when
> >> >> > handling very large (many gigs) datasets: potentially long repair
> >> >> > times and memory consumption that grows with data fragmentation.
> >> >> > - QLC, the query engine for Mnesia, doesn't currently optimize
> >> >> > joins.
> >> >> > If your queries involve joining big tables, they can take a long
> >> >> > time
> >> >> > to execute.
> >> >> >
> >> >> > The join optimizations are planned for a future R11 OTP release, but
> >> >> > there are no plans to change dets AFAIK.
> >> >> >
> >> >> > Depending on your application, these issues may not be a big
> >> >> > problem.
> >> >> > For the application I'm building, I'm planning on using both MySQL
> >> >> > and
> >> >> > Mnesia, where MySQL will be used for storing high-volume data and
> >> >> > Mnesia for "live" session data.
> >> >> >
> >> >> > Hope this helps!
> >> >> >
> >> >> > Regards,
> >> >> > Yariv
> >> >> >
> >> >> > On 8/8/06, Inswitch Solutions <erlang@REDACTED> wrote:
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Hi,
> >> >> >>
> >> >> >> I'm already working with Oracle and Mnesia, and I'd  like to hear
> >> >> >> experiences of the Erlang community about these  databases.
> >> >> >> When deciding over Oracle or Mnesia database for an  Erlang, or non
> >> >> >> Erlang, based real-time system which factors are in favour  in one
> >> >> >> over
> >> >> >> the other (performance...?) ?.
> >> >> >>
> >> >> >>
> >> >> >> thanks, Eduardo
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Prepaid Expertise -        Programmable Switches
> >> >> >> Powered by Ericsson Licensed Technology
> >> >> >> Eng.        Eduardo Figoli - Development Center - IN Switch
> >> >> >> Solutions
> >> >> >> Inc.
> >> >> >> Headquarters - Miami-U.S.A. Tel: 1305-3578076 Fax:
> >> >> >> 1305-7686260
> >> >> >> Development Center - Montevideo - Uruguay Tel/Fax:
> >> >> >> 5982-7104457
> >> >> >> e-mail: eduardo@REDACTED
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >> >
> >>
> >>
> >
>
>