[erlang-questions] Twoorl: an open source Twitter clone

[Per Melin]

2008/5/29 Yariv Sadan <yarivsadan@REDACTED>:
> Facebook runs on MySQL -- that's pretty scalable :)

By Twitter's own account the database has been their bottleneck all along.

> The main reason I didn't go with Mnesia for storing most data is the
> dets issues discussed previously on this list (specifically, the long
> repair times and the need for fragmentation over 4gb). I use Mnesia
> for storing session data though.

I wouldn't go for Mnesia either. If you're building something that's
should possibly scale to the level of Twitter, the 2 GB limit just
doesn't cut it, with or without fragmentation. And in the tests I've
done DETS got very slow long before it hit the size limit. If you use
disc_copies (which doesn't use DETS) instead of disc_only_copies
you're of course instead limited by RAM.

I (like everyone else) have built a (web) chat service that I'll
probably (alpha) release this weekend. The single biggest problem I've
been wrestling with is how to persist the messages in an efficient
manner. Right now I'm using term_to_binary and dumping that in a flat
file (much like disk_log), which works for all my needs, except for
full text search. I'd prefer not to build and store my own reverse
index.