[erlang-questions] Distributed publish/subscribe system

[Joe Armstrong]

On Tue, Jan 25, 2011 at 4:00 PM, Kaiduan Xie <kaiduanx@REDACTED> wrote:

> Thanks Jack, Tony, Toby, Joe and Thomas for the helpful comments and
> resources. Joe pointed out a lot of excellent points. The requirements
> are listed below,
>
> 1) The system should be able to support tens to hundreds of millions
> users spreading around the world, so the system will consist hundred
> of nodes located in different physical locations.
>

Who own the nodes? are they all owned by the same organisation?

Who is legally responsible for the system?


>
> 2) Any user can subscribe to the interested topic, but only topic
> owner can publish message to the group. We have no limit on the number
> of subscribers in each group. It can be huge, for example, the fans of
> Rolling Stone around the world.
>
>
How do your stop spam - bogus groups etc?


> 3) The number of topics/groups in the system is unlimited.
>

How do you stop name squatting - ie flooding the system with stupid channel
names?

>
> 4) User is not always connected to the system, not always connected to
> the same node in the system, and the message delivery should be
> guaranteed.
>

Uuug - impossible - it's the Byzantine generals problem. How do you know
they got a
message? - send an ack - how do you know if they got the ack?- ack the ack -
how do you know
if they acked the ack, ack the acked ack ...

/Joe



> 5) As to the latency, it should be in the range of 1 minute if
> subscriber is connected.
>
> I am doing research on this topic now, the other requirements are not
> sorted out yet.
>
> Thanks,
>
> /Kaiduan
>
>
> On Tue, Jan 25, 2011 at 3:56 AM, Joe Armstrong <erlang@REDACTED> wrote:
> > Standard question:
> >
> > How big? - how scalable - what latency - fault tolerance requirements do
> you
> > have.
> >
> > Distributed can mean 2 nodes - or 10^9
> > There might be ten channels or 10^12
> >
> > How fault tolerant? - 2 replicas, 100 replicas.
> >
> > What the latency? - do you want the message to be delivered with 10 ms or
> > 100 years
> >
> > What's the security model - who owns the data - is there centralized
> > security or weak
> > per-node security policies.
> >
> > The point is your requirements need to say things like this otherwise
> they
> > are not much use.
> >
> > The architectures for these different cases will be *completely
> different*
> >
> > /Joe
> >
> > On Tue, Jan 25, 2011 at 12:54 AM, Kaiduan Xie <kaiduanx@REDACTED>
> wrote:
> >>
> >> Hi,
> >>
> >> Any one has experience to build distributed publish/subscribe system?
> >> The system has the following requirements,
> >>
> >> 1. Subscriber subscribes to the interested topic
> >> 2. Publisher publishes event related to the topic
> >> 3. System delivers the event to all the users subscribed to this
> >> topic. If subscriber is connected, the event is delivered immediately.
> >>    If the subscriber is not connected, system should persist the
> >> event. When the user next time becomes online, system delivers the
> >> event to the user.
> >>
> >> Can you share the experience?
> >>
> >> Best regards,
> >>
> >> /Kaiduan
> >>
> >> ________________________________________________________________
> >> erlang-questions (at) erlang.org mailing list.
> >> See http://www.erlang.org/faq.html
> >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >>
> >
> >
>