[erlang-questions] write appending a file on a replicated file store

Jesper Louis Andersen jesper.louis.andersen@REDACTED
Tue Sep 4 11:13:09 CEST 2012

On Sep 4, 2012, at 8:36 AM, Joe Armstrong <erlang@REDACTED> wrote:

> Does anybody know if these stores handle write appends in an efficient manner.
> The worse case would be that each new write append treats the new version of
> the file as a totally new file and copies everything.

There are two methods which come to mind: Rsync-like functionality and merkle-trees.
> If the entire file gets copied then I'd need some kind of incremental
> system, create many small files, then occasionally merge them
> has anybody done this?

Merkle trees is the tool you want I think. It will quickly find out that there are new data appended and only transfer the data there. Coincidentally, there is some merkle tree stuff inside riak :)

More information about the erlang-questions mailing list