[erlang-questions] Twoorl: an open source Twitter clone

[Paul Fisher]

On Wed, 2008-06-04 at 09:08 -0700, Steve Davis wrote:
> On Jun 4, 10:12 am, Lev Walkin <v...@REDACTED> wrote:
> > ... yet, with relaxed ACID properties, making it much more scalable.
> 
> The level to which I disagree with your conclusion here is quite
> profound! I see a large-scale future of lost messages, data
> corruption, file replication and synchronization issues, and
> *downtime* for the FS approach.
> 

I promise that the RDBMS wall exists... we deal with more "records" that
you would ever imagine putting into a relational database.  For example,
on one small portion of our infrastructure we still use MySQL databases
with 8-core 32GB machines, and we have to limit each instance to no more
than a few days worth of data in order to keep it going with reasonable
performance.  Imagine having to commission database instances every
other day of 100s of GBs.

Any of the Google papers or recent articles from Stonebreaker discuss
this in directly in terms of the limits of RDBMS systems.

Finally, i am also here to attest that the filesystem approach scales
well past were the RDBMS falls over.  You just need to architect the
system to balance the trade-offs, which naturally exist at extreme
levels of scale.


-- 
paul