Erlang filesystem (was: Re: File versioning)

[Chris Pressey]

On Wed, 7 May 2003 10:18:00 +0200 (MEST)
Bengt Kleberg <eleberg@REDACTED> wrote:

> > see http://plan9.bell-labs.com/sys/doc/venti/venti.html for a
> > write-only file server.
> 
> it has been pointed out to me that write-only is wrong terminology.

http://www.ganssle.com/misc/wom.html

:)

Seriously, it's a good idea.

Vlad wrote:
> In an Erlang environment, we could modify the file server to access
> some kind of repository for some parts of the namespace, which could
> be a database or a CVS repository or something like that. This way
> everything looks like normal files, but they aren't. Maybe Luke's nfs
> server could also be a part of this too...

Everything looks like normal files - in an Erlang environment.  That
strikes me as the biggest drawback, at least for the sake of argument. 
I'd like to be able to use it from other programs with a minimum of
fuss.  I can think of a few ways around it:

1- make Erlang's filesystem mountable as an OS-level filesystem
2- make Erlang an OS
3- have some way to painlessly communicate between external, non-Erlang
   programs, and a running Erlang node

I get the impression that #1, writing a mountable filesystem 'driver',
is tricky even in C.  I could be wrong though.

I have no problem with #2 since Erlang already has many of the
characteristics of a good OS and since I've abandoned the adage of "the
right language for the task" after realizing that Java - no, Haskell -
no, *Erlang* is clearly God's Own Language.  Also, re-writing in Erlang
all the applications I use daily would give me something interesting to
do.  :)

But #3 is probably the most feasible, and useful for a number of other
things too.  I've tried using a named pipe as a port, and I've tried
ad-hoc connections to an Erlang distribution, but neither seems reliable
enough for real use.  That pretty much leaves sockets - probably inet
sockets since Unix-domain sockets aren't as well tested, and inet
sockets would let you connect across the network & can be secured if
need be.

So I envision doing stuff like the following from the command line:

  ls | grep foo | sort -r | store-in-erlfs-as bar.txt

where 'store-in-erlfs-as' is a little program which opens a socket and
talks to the running Erlang distribution that is handling the Erlang
filesystem.  It could maybe even be a shell script that looks for a
temporary file for what machine and port to connect to, then uses
'tcpdump' or similar to connect to that socket and send Erlang it's
input.

Naturally there'd also be a corresponding 'fetch-from-erlfs' program.
With shorter names probably, like 'erlfsstore' & 'erlfsfetch', but you
get the idea.

On a vaguely related note... someone recently asked on one of the
FreeBSD lists if there was any realtime file replication software
available.  rsync and so forth were mentioned, but apparently there
isn't any that is really 'realtime', and what's more it sounds like a
'hard' problem.  Maybe an Erlang filesystem could fit the bill here?

-Chris