[erlang-questions] Amazon S3 backend for Mnesia

[Ulf Wiger (TN/EAB)]

I guess you could imagine some stateful callback
module (although the module will have to manage
state through dirty means, e.g. an ets table), 
to determine whether the access functions should
actually update the backend or not.

If you only care about doing it on one node at 
a time, you could use something like 
mnesia_lib:val({Tab,where_to_write}) and compare
that with mnesia_lib:val({Tab,external_copies})
(in the rdbms patch), and simply pick the first
one in the list (the external_copies list will
be in the same order on all nodes, but I don't 
think where_to_write necessarily is.)

I'm not sure whether that's air tight, though.
The where_to_write info is update under the 
protection of a table lock, I believe, so chances
are it might work.

The basic maintenance problem you'll have is of 
course that OTP hasn't incorporated the rdbms
patches related to external copies (some other 
parts of rdbms did make it in, though). You'll
be on the bleeding edge as the only user, which
is of course fun in some sense.

The patches needed are in some pretty sensitive
parts of mnesia, which are also likely to be 
updated in future versions of OTP, so you'll
be playing catch-up unless you can help show 
that this way of plugging in back-ends is really
the way to go (not saying it is - it's one way,
but it's mostly untested.)

BR,
Ulf W

> -----Original Message-----
> From: Joel Reymont [mailto:joelr1@REDACTED] 
> Sent: den 4 juni 2007 15:35
> To: Ulf Wiger (TN/EAB)
> Cc: Erlang Questions
> Subject: Amazon S3 backend for Mnesia
> 
> 
> On Jun 4, 2007, at 1:04 PM, Ulf Wiger (TN/EAB) wrote:
> 
> > But you basically only need to do this if it's important for you to 
> > amortize the updates. It might be important in other cases 
> _not to_, 
> > e.g. if you have other readers of the custom backend. In that case, 
> > data wouldn't be visible after a commit, but only after the 
> next log 
> > dump, which might result in inconsistencies at the 
> application level.
> 
> I can now reveal that I'm working on a Amazon S3 backend for Mnesia.  
> The goal is to add transactions and locking, as well as make 
> the data just written to S3 available right away.
> 
> S3 provides neither transactions nor locking. Data written to 
> S3 may become available for reading at some point in the 
> future. S3 is conceptually a bunch of ets tables, at least I 
> think it can be viewed this way.
> 
> I'm looking to have a bunch of Mnesia nodes up and a bunch of 
> s3_copies tables on each.
> 
> Whenever data is written to one of the Mnesia nodes, I'm 
> looking to have _that_ node commit the data to S3 and 
> replicate it to other nodes as Mnesia does normally. I would 
> also keep a cache of most recently written data in a regular 
> Mnesia table.
> 
> I understand that Synapse added a bdb_copies type of table 
> and found this hard to maintain. Would rdbms be easier to 
> maintain in my scenario?
> 
> 	Thanks, Joel
> 
> --
> http://topdog.cc      - EasyLanguage to C# translator
> http://wagerlabs.com  - Blog
> 
> 
> 
> 
> 
>