Synchronizing Mnesia tables with data compression

[Igor Ribeiro Sucupira]

Hello.

I'm deploying a system that's going to work with many replicated
Mnesia fragments (hundreds of GB per node) and it seems Mnesia needs
to copy some full dets files between nodes if one of them goes down
and is started again.
So, I thought I could alleviate at least the network traffic by
compressing the messages that are sent between nodes when a dets table
is being copied.

I have added a Mnesia option to allow setting the compression level
for that operation. The default level is 0, in which Mnesia behaves
exactly like today (no compression when copying tables).
I believe this change affects table loading (from a master node) and
table replication (add_table_copy).

You can fetch the patch with:
git fetch git://github.com/igorrs/otp.git send_compressed

Full commit comments:

    Add option to compress data when copying tables between Mnesia nodes

    Optionally using data compression for copying Mnesia tables allows
    the system to be tuned to eliminate network bottlenecks from
    deployments where enough CPU is available to use zlib.
    With this patch, running erl -mnesia send_compressed <level>
    will enable compression for sending tables between nodes. The
    compression level can be any integer in [0, 9], with 0 (the
    default) meaning no compression (exactly the previous behaviour)
    and 9 being the highest compression level.
    To set any non-zero compression level at the sender, both nodes
    must have the updated Mnesia modules (the receiver will work
    according to the sender's configuration).
    I will update the documentation as soon as the patch is merged to the
    pu branch.


Igor.