pushing messages to large number of subscribers with minimal latency
Wed Jul 1 18:12:43 CEST 2009
I'm trying to build a solution where I can push a message to hundreds
of thousands of topic subscribers with minimal latency. Topic
subscribers may come and go at a rapid clip but only point of the
solution is to push messages out to subscribers as quickly as possible.
I'm writing in Erlang and wondering if RabbitMQ is really buying me
much given that it relies so much on Erlang. Pushing messages to
subscribers also seems to be a basic telco problem, surely solved a
few times over.
I'm thinking of a design where each node in the cluster keeps track of
subscribers connected to that node (over TCP) and messages are pushed
to each node which then proceeds to push to its share of subscribers.
I could then keep track of topics on each node by wrapping a dict or
ets table in a process and keying on the pid for ease of removal.
There are no locking or replication needs if topic tracking processes
are running on each node and the number of topics in the system would
only be limited by memory if a dict is used instead of ets.
Any comments on my design?
Mac hacker with a performance bent
More information about the erlang-questions