[erlang-questions] Twoorl: an open source Twitter clone

[Yariv Sadan]

I considered using a reliable queuing mechanism such as RabbitMQ or
Amazon SQS but I don't think it would make the architecture inherently
more scalable (more reliable maybe, but not more scalable). I think a
Twitter like solution can be designed to scale using just Yaws, MySQL,
and Mnesia or memcache (and maybe Ejabberd if you add an XMPP
gateway). RabbitMQ or SQS would provide *reliable* asynchronous
background processing, but if you don't need 100% reliability (Twoorl
isn't a banking application after all), you can just spawn Erlang
processes from Yaws to do background tasks after a user posts a
message. Also, using persistent queues doesn't make the need for a
database go away. When you pull a tweet from a queue you have to put
it somewhere so it can be shown on rendered pages, and a database is
the most reasonable place to put it. The main problem in scaling
Twitter/Twoorl is how you architect your database backend --
partitioning, denormalization, replication, load balancing, caching,
etc, will probably make or break your ability to scale.

Yariv

On Sun, Jun 1, 2008 at 1:41 AM, David Mitchell <monch1962@REDACTED> wrote:
> This is a REALLY interesting discussion, but at this point it's
> becoming obvious that I don't know enough about Twitter...
>
> Are you suggesting that Twoorl should be architected as follows:
> - when they register, every user gets assigned their own RabbitMQ
> incoming and outgoing queues
> - user adds a message via Web/Yaws interface (I know, this could be
> SMS or something else later...)
> - message goes to that user's RabbitMQ incoming queue
> - a backend reads messages from the user's incoming queue, looks up in
> e.g. a Mnesia table to see who should be receiving messages from that
> user and whether they're connected or not.  If "yes" to both, RabbitMQ
> then forwards the message to each of those users' outgoing queues
> - either the receiving users poll their outgoing queue for the
> forwarded message, or a COMET-type Yaws app springs to life and
> forwards the message to their browser (again, ignoring SMS)
>
> This seems like a reasonable approach; I'm just curious if that's what
> you're suggesting, or whether you've got something else in mind.
>
> Great thread, and thanks Yariv for getting this discussion going with Twoorl
>
> Regards
>
> Dave M.
>
> 2008/6/1 Steve <steven.charles.davis@REDACTED>:
>>
>> On May 31, 5:04 pm, "Yariv Sadan" <yarivsa...@REDACTED> wrote:
>>> ...but it's the only way you can scale this kind of service when N is
>>> big.
>>
>> Hmmm, Yariv, aren't you still thinking about this in the way that Dave
>> Smith pointed to as the heart of the issue? i.e.
>> Dave said: "My understanding is that the reason they have such poor
>> uptime is due to the fact that they modeled the problem as a web-app
>> instead of a messaging system."
>>
>> I'm aware that you are likely a good way away from hitting any
>> scalability problems, but some kind of tiering would seem to be
>> appropriate if twoorl is to be "twitter done right". Yaws at the front
>> end, definitely - but rather /RabbitMQ/ at the back end. I believe
>> that you'd then have the flexibility to distribute/cluster as
>> necessary to scale to the required level (whatever that may be).
>>
>> For sure, Twoorl is a great demo of what can be done with Erlang in an
>> incredibly short time. I'm a relative noob to Erlang, and have learned
>> a great deal from your blog/code/examples.
>>
>> Steve
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://www.erlang.org/mailman/listinfo/erlang-questions
>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>