[erlang-questions] Twoorl: an open source Twitter clone

[Lev Walkin]

Steve Davis wrote:
> On Jun 4, 10:12 am, Lev Walkin <v...@REDACTED> wrote:
>> ... yet, with relaxed ACID properties, making it much more scalable.
> 
> The level to which I disagree with your conclusion here is quite
> profound! I see a large-scale future of lost messages, data
> corruption, file replication and synchronization issues, and
> *downtime* for the FS approach.
> 
> The raw FS solution

Who talks about _raw_ FS solution? You are? I am certainly not.

> IMHO will only "work" up to a point... in a sense
> the FS suggestion parallels (but is not the same as) MySQL's early and
> conscious sacrifice of referential integrity for performance (InnoDB
> does not resolve this btw, for those that know the issues with MySQL).
> This decision at MySQL has resulted in numerous high-profile scaling
> issues for many services that committed to MySQL for their persistence
> needs. Twitter isn't a good example of this kind of failure - it
> shouldn't have been using the RDBMS at all in the way that it did.
> 
> All in all, the fundamental application scope of Twitter simply
> *screams* Message Queue at me. I'm not sure why the "experts" that
> Twitter have scavenged from IBM and Google haven't come to that
> conclusion also. Since Twitter have appeared to commit to a FS
> approach, I guess we'll have to see if future history proves me
> incorrect :)

Steve, I run a quite high traffic, distributed, file-system
based (as opposed to RDBMS based) service, js-kit.com

http://www.techaddress.com/2008/05/29/js-kit-scores-deal-with-worldnow-adds-19-million-potential-users/

The service internally provides itself a CID guarantee,
with "eventual consistency" instead of A, quite like Amazon SimpleDB:

http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

We also use message queues internally to provide reliability and
sychronization.

One of the most important thing which allowed us to gain scalability
is our use of file system storage behind all this CID machinery on top
of it.

When, as Per Melin says, Twitter wrote on their blog Saturday:
"Our new architecture will move our reliance to a simple, elegant
filesystem-based approach, rather than a collection of database
[sic]."

I believe the emphasis was placed on replacement of RDBMS
with something _on top of a raw FS_, not just _with_ a raw FS.

-- 
Lev Walkin
vlm@REDACTED