[erlang-questions] write appending a file on a replicated file store

[Jesper Louis Andersen]

On Sep 4, 2012, at 8:36 AM, Joe Armstrong <erlang@REDACTED> wrote:

> Does anybody know if these stores handle write appends in an efficient manner.
> The worse case would be that each new write append treats the new version of
> the file as a totally new file and copies everything.
> 

There are two methods which come to mind: Rsync-like functionality and merkle-trees.
> 
> If the entire file gets copied then I'd need some kind of incremental
> system, create many small files, then occasionally merge them
> has anybody done this?

Merkle trees is the tool you want I think. It will quickly find out that there are new data appended and only transfer the data there. Coincidentally, there is some merkle tree stuff inside riak :)