[erlang-questions] write appending a file on a replicated file store
Jesper Louis Andersen
Tue Sep 4 11:13:09 CEST 2012
On Sep 4, 2012, at 8:36 AM, Joe Armstrong <erlang@REDACTED> wrote:
> Does anybody know if these stores handle write appends in an efficient manner.
> The worse case would be that each new write append treats the new version of
> the file as a totally new file and copies everything.
There are two methods which come to mind: Rsync-like functionality and merkle-trees.
> If the entire file gets copied then I'd need some kind of incremental
> system, create many small files, then occasionally merge them
> has anybody done this?
Merkle trees is the tool you want I think. It will quickly find out that there are new data appended and only transfer the data there. Coincidentally, there is some merkle tree stuff inside riak :)
More information about the erlang-questions