[erlang-questions] Twoorl: an open source Twitter clone

[Patrick Logan]

From: Steve Davis <steven.charles.davis@REDACTED>
>> No database is needed. Everything can pretty much be achieved with
>> files containing fixed-length records.
>
> What you and Bob have said is, of course, true. I suspect that as
> the system becomes large scale then the software required to access
> and manage those files would eventually end up looking a lot like
> one of... an RDBMS, a Message Queue, an IM server, etc.

Hmm. Sounds kind of like a "record" has most of the properties of an
Atom format "Entry" and a series of these are like a "Feed". Whereever
these are stored transiently or archived semi-permanently, these are
small, time-ordered, series of information. Not surprising.

From: Steve Davis <steven.charles.davis@REDACTED>

> The level to which I disagree with your conclusion here is quite
> profound! I see a large-scale future of lost messages, data
> corruption, file replication and synchronization issues, and
> *downtime* for the FS approach.
>
> All in all, the fundamental application scope of Twitter simply
> *screams* Message Queue at me. I'm not sure why the "experts" that
> Twitter have scavenged from IBM and Google haven't come to that
> conclusion also. Since Twitter have appeared to commit to a FS
> approach, I guess we'll have to see if future history proves me
> incorrect :)

This decision could probably be broken down into some general "needs"
(e.g. Twoorl needs time-ordering, categorization by author,
persistence of an entry for some length of time, etc.). Then various
solution alternatives could be mapped onto some or all of those
needs. I think it's too much too soon to say either "use a FS
approach" or "use a queue approach". Plus you'd certainly have to get
to the level of which kind of file system or queue for which needs?
e.g. I wrote before about XMPP as a suitable queue for several of the
needs, and for file systems, ZFS may have some nice attributes for
some of these needs.

From: Damien Morton <dmorton@REDACTED>
> Well, the relational database is suitable for the quickly evolving
> part of their business, and there will always be a quickly evolving
> part of any business, so yes - inevitability is the perfect
> description.

Ironic since I hardly ever see the phrases "relational database" and
"quickly evolving" used together except in an inverse
relationship. 8^D

> Question is - can they create such a specialised datastructure such
> that it _can_ evolve, for example, in the case where they wanted to
> handle not just 140 byte tweets, but also photos and video (twotos,
> and twideos)?

I would probably consider something like an Amazon SimpleDB service as
a potential solution alternative.

-Patrick