[erlang-questions] Distributed publish/subscribe system

[Kaiduan Xie]

Thanks Jack, Tony, Toby, Joe and Thomas for the helpful comments and
resources. Joe pointed out a lot of excellent points. The requirements
are listed below,

1) The system should be able to support tens to hundreds of millions
users spreading around the world, so the system will consist hundred
of nodes located in different physical locations.

2) Any user can subscribe to the interested topic, but only topic
owner can publish message to the group. We have no limit on the number
of subscribers in each group. It can be huge, for example, the fans of
Rolling Stone around the world.

3) The number of topics/groups in the system is unlimited.

4) User is not always connected to the system, not always connected to
the same node in the system, and the message delivery should be
guaranteed.

5) As to the latency, it should be in the range of 1 minute if
subscriber is connected.

I am doing research on this topic now, the other requirements are not
sorted out yet.

Thanks,

/Kaiduan


On Tue, Jan 25, 2011 at 3:56 AM, Joe Armstrong <erlang@REDACTED> wrote:
> Standard question:
>
> How big? - how scalable - what latency - fault tolerance requirements do you
> have.
>
> Distributed can mean 2 nodes - or 10^9
> There might be ten channels or 10^12
>
> How fault tolerant? - 2 replicas, 100 replicas.
>
> What the latency? - do you want the message to be delivered with 10 ms or
> 100 years
>
> What's the security model - who owns the data - is there centralized
> security or weak
> per-node security policies.
>
> The point is your requirements need to say things like this otherwise they
> are not much use.
>
> The architectures for these different cases will be *completely different*
>
> /Joe
>
> On Tue, Jan 25, 2011 at 12:54 AM, Kaiduan Xie <kaiduanx@REDACTED> wrote:
>>
>> Hi,
>>
>> Any one has experience to build distributed publish/subscribe system?
>> The system has the following requirements,
>>
>> 1. Subscriber subscribes to the interested topic
>> 2. Publisher publishes event related to the topic
>> 3. System delivers the event to all the users subscribed to this
>> topic. If subscriber is connected, the event is delivered immediately.
>>    If the subscriber is not connected, system should persist the
>> event. When the user next time becomes online, system delivers the
>> event to the user.
>>
>> Can you share the experience?
>>
>> Best regards,
>>
>> /Kaiduan
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>
>
>