[erlang-questions] suitability of erlang for large-scale datastore

[Christian S]

On 5/24/07, Gary Jefferson <garyjefferson123@REDACTED> wrote:
> Would erlang be a good choice for creating, for
> example, a very high-traffic ftp server?

I would store the file namespace in mnesia and the data in the underlying
file system. So with mnesia you would list directories and find the
filesystem name for a ftp filename. The filesystem filenames could be
sha1 hashes of the file content.

The ftp application would make sure that every node participating as a
ftp server has a copy of the data file before changes are made to
mnesia.

Mnesia is neither good at nor designed to being a blob-store. That is
why I wouldn't put the actual data in mnesia tables.

> What would be some of the other
> advantages/disadvantages of using erlang to build such
> a thing?

Pro:
If the application is written in good OTP manner you will get all the
good properties in robustness etc. The risk of introducing buffer
overflow bugs and other security mistakes are minimal.

Con:
Not having sendfile-syscall in erlang might cost you 10-20% in
performance relative some singlenode C hacks. Maybe, I'm just
guessing. Though, I spotted a patch to add sendfile syscall to erlang
posted to this mailinglist a few years ago.


I have pondered an ftp server with an entirely different target: You
see, I work close to banks and they love to "drop files" into ftp
areas as for their information exchange. This is currently handled by
having a cronjob that poll their upload directory for new files.
Wouldnt it be lovelier if one could just have the ftp server notify
you of the change as it happens? Perhaps even refuse the upload if the
file doesnt pass sanity checks for the format.