mneisa table fragmentation vs manually splitting

[Zohan]

I have no practical experience with using the mnesia 'fragmented
table' but it makes sense to reuse it and not invent it. i wonder how
it scales up, as you stated.

Z.

On Oct 2, 12:33 pm, vim <ofer.aff...@REDACTED> wrote:
> hi Z, thanks for your reply,
>
> > why do you claim the application should know how to use a specific node to write to a specific key?
>
> i need to know in which cluster resides the key i need. i.e. if i have
> partitioned my database to 100 clusters, in need to keep a map which
> tell me on which cluster i can found a specific key.
>
> this is not related to the master-slave scheme, even if i have just 1
> machine on each cluster, i need to do the same. the master-slave
> scheme is a just a possible enhancement, so i can have hot-backup for
> each key-value.
>
> on the other hand, if i use mnesia fragmented database feature, mnesia
> (as far as i understand) should keep track on where each key resides,
> releasing me from keeping that key mapping.
>
> the disadvantage i fear is the up-scale limit of connecting many
> machines together without partitioning into clusters. from what i have
> read here and there, erlang shouldn't be connected together on more
> than few hundreds active machine. * if anyone has numbers of this
> practical limit, please advise *
>
> > What are the master-slave relations you assume in the 10 machine cluster?
>
> master-slave refers to a scheme where i connect all the mnesia
> instances on the relevant machines, so they are synced. but i choose
> split up my dataset, where for a given key i only do writes through
> specific machine/node. on that kind of scheme i get fault-tolerance
> because if the master (the writer) fails, i always have a backup
> (slave) with the most up to date data. and i can assign one of the
> slave to be a master if the master fails. furthermore, i don't get
> into many problems of multiple write, and syncing them, when you use
> master-master scheme (more than 1 writer)
>
> vim.
>
> On Oct 1, 12:31 am, Zohan <matan.l...@REDACTED> wrote:
>
>
>
> > hi,
>
> > For the manual split option you describe, why do you claim
> > the application should know how to use a specific node to write to a
> > specific key,
> > in order to use master-slave duplicated database scheme?
> > What are the master-slave relations you assume in the 10 machine
> > cluster?
>
> > BR,
> > Z.
>
> > On Sep 27, 8:40 am, vim <ofer.aff...@REDACTED> wrote:
>
> > > hi,
>
> > > i am working on a erlang based distributed architecture, at the end i
> > > intend to use it on EC2. my application included a database, and since
> > > i want to spread the load on the database, i thought of splitting the
> > > table.
>
> > > the question is should i choose mnesia built-in feature 'fragmented
> > > table' over manually splitting the database into clusters?
>
> > > i.e. suppose i have a simple key-value sets which i need to store in
> > > my database. and suppose i have 1000 keys, and i did some measurements
> > > and i want only 10 keys on each machine (for better performance) so i
> > > need 100 machines.  thus, if i understand correctly the
> > > documentation, i have at least two options:
>
> > > 1. manual split:
> > > i can set up clusters of 10 machines each. on each machine i will have
> > > a node with a running mnesia, and on a such cluster all the mnesia
> > > nodes will be connected. furthermore, i can use master-slave
> > > duplicated database scheme, if my application will know how to use a
> > > specific node to write to a specific key. i assume here that *all* the
> > > nodes in my world (100) are erlang connected, and on each 10 machine
> > > cluster there is a connected (shared) mnesia database.
>
> > > 2. table fragmentation:
> > > i can connect all my 100 machines with single mnesia schema, and
> > > fragment the table into 10 pieces, where each cluster of 10 machines
> > > will share each piece. again, i can use the same master-slave scheme i
> > > described above to access my database.
>
> > > am i correct with those assumptions?
> > > should i prefer one method over the other?
>
> > > thanks for your help,
> > > vim.
>
> > > ________________________________________________________________
> > > erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> > > erlang-questions (at) erlang.org
>
> > ________________________________________________________________
> > erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> > erlang-questions (at) erlang.org
>
> ________________________________________________________________
> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> erlang-questions (at) erlang.org