[erlang-questions] Twoorl: an open source Twitter clone

[Steve Davis]

On Jun 4, 11:45 am, Lev Walkin <v...@REDACTED> wrote:
> Who talks about _raw_ FS solution? You are? I am certainly not.

Lev: Actually, I didn't see anyone mention anything other than a raw
FS solution, hence the "rider", so thanks for expanding on your
initial comment!

> Steve, I run a quite high traffic, distributed, file-system
> based (as opposed to RDBMS based) service, js-kit.com
>
> http://www.techaddress.com/2008/05/29/js-kit-scores-deal-with-worldno...
>
> The service internally provides itself a CID guarantee,
> with "eventual consistency" instead of A, quite like Amazon SimpleDB:
>
> http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
>
> We also use message queues internally to provide reliability and
> sychronization.
>
> One of the most important thing which allowed us to gain scalability
> is our use of file system storage behind all this CID machinery on top
> of it.

At first sight, I don't see the immediate applicability of these
architectures to the "twitter problem", but there's a lot of detail
there so I'll take the time to look more closely and actually fully
read the amazon paper to see what's there. At the very least, it
should be an interesting/informative read.

> When, as Per Melin says, Twitter wrote on their blog Saturday:
> "Our new architecture will move our reliance to a simple, elegant
> filesystem-based approach, rather than a collection of database
> [sic]."
>
> I believe the emphasis was placed on replacement of RDBMS
> with something _on top of a raw FS_, not just _with_ a raw FS.

Whilst elegant, neither your solution nor Amazon's seem "simple". One
can only hope that Twitter have thought it through to the extent you
have for your architecture, and that the "something on top" that they
have in mind does actually address the issues that I raised. For now
and for me, the best candidate for that "something on top of the FS"
remains a Message Queue :)

To Per Melin: I'm thinking subscription queues (pubsub) and messages
having a time-to-live (TTL). Perhaps one subscriber might be a logger
for permanent record.

/s