[erlang-questions] mnesia -- a naive question

[Jesper Louis Andersen]

A couple of points:

* Mnesia protects you against the scenario where one of your nodes fail. It
doesn't automatically protect you against the network splitting, and
requires some manual recovery on the flip side of such an event. For rather
small clusters, this is manageable by manual operation. Larger systems will
be far harder to maintain because the risk of netsplits and node loss goes
up whenever you add a new node.

* I don't know about Wasabi, but Amazon's EC2 nodes are ephemeral in the
sense they can go away at a moments notice. And when this happens, the data
on the node is gone. Thus, to achieve persistent storage, you must either
store data off the EC2 node, presumably in S3, RDS, DynamoDB and so on. Or
use an EBS volume, attached to the EC2 node to provide persistent disk
space (on which your mnesia database can reside).

* The game is all about risk mitigation. If you regularly take a mnesia
backup and store it into S3, or something like it, you can get speedy
recovery to that point in time should the accident happen. If you want
better point-in-time-recovery, you can try running two mnesia nodes, but
you need to heed two important caveats:
    - You probably want your nodes to run in different zones so a failure
in one zone doesn't take down everything.
    - Amazons network is brittle and likely to drop connections which are
seen as netsplits.

* Mnesia mitigates risk by assuming the nodes are fairly robust and stable,
as well as the network between them. If you buy good expensive hardware,
this is a likely assumption and the noise of error will be low. So manual
intervention in the case of an error is probably what is needed anyway (to
fix the faulty hardware as well).

* Amazon and other leased environments tend to have brittle network
connections and flaky machines. To mitigate this, your system must make no
assumptions about stability and handle this up front. Mnesia wasn't really
built to work in such an environment.



On Sat, Jul 29, 2017 at 10:23 PM <lloyd@REDACTED> wrote:

> Hello,
>
> Wasabi is a new cloud storage service that promotes lower storage costs
> and greater speed than Amazon S3:
>
> https://wasabi.com/
>
> During the dev phase I'm running mnesia on the back-end of my current web
> project. I much like the seamless way that mnesia integrates into Erlang as
> well as its replication feature. But folks have warned about the hassles of
> mnesia net splits.
>
> Problem is that I have no operations experience to objectively weigh
> options. But I do want to bridge over all points of failure as
> cost-and-time-effectively as possible.
>
> So, my question is if and how I can integrate Wasabi (or Amazon S3 for
> that matter) into my operation to significantly reduce the probability of
> data loss?
>
>
> Many thanks,
>
> LRP
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170730/95296fff/attachment.htm>