Synchronizing Mnesia tables with data compression
Igor Ribeiro Sucupira
Sun Jan 10 08:08:50 CET 2010
I'm deploying a system that's going to work with many replicated
Mnesia fragments (hundreds of GB per node) and it seems Mnesia needs
to copy some full dets files between nodes if one of them goes down
and is started again.
So, I thought I could alleviate at least the network traffic by
compressing the messages that are sent between nodes when a dets table
is being copied.
I have added a Mnesia option to allow setting the compression level
for that operation. The default level is 0, in which Mnesia behaves
exactly like today (no compression when copying tables).
I believe this change affects table loading (from a master node) and
table replication (add_table_copy).
You can fetch the patch with:
git fetch git://github.com/igorrs/otp.git send_compressed
Full commit comments:
Add option to compress data when copying tables between Mnesia nodes
Optionally using data compression for copying Mnesia tables allows
the system to be tuned to eliminate network bottlenecks from
deployments where enough CPU is available to use zlib.
With this patch, running erl -mnesia send_compressed <level>
will enable compression for sending tables between nodes. The
compression level can be any integer in [0, 9], with 0 (the
default) meaning no compression (exactly the previous behaviour)
and 9 being the highest compression level.
To set any non-zero compression level at the sender, both nodes
must have the updated Mnesia modules (the receiver will work
according to the sender's configuration).
I will update the documentation as soon as the patch is merged to the
More information about the erlang-patches