pushing messages to large number of subscribers with minimal latency

[Joel Reymont]

I'm trying to build a solution where I can push a message to hundreds  
of thousands of topic subscribers with minimal latency. Topic  
subscribers may come and go at a rapid clip but only point of the  
solution is to push messages out to subscribers as quickly as possible.

I'm writing in Erlang and wondering if RabbitMQ is really buying me  
much given that it relies so much on Erlang. Pushing messages to  
subscribers also seems to be a basic telco problem, surely solved a  
few times over.

I'm thinking of a design where each node in the cluster keeps track of  
subscribers connected to that node (over TCP) and messages are pushed  
to each node which then proceeds to push to its share of subscribers.

I could then keep track of topics on each node by wrapping a dict or  
ets table in a process and keying on the pid for ease of removal.  
There are no locking or replication needs if topic tracking processes  
are running on each node and the number of topics in the system would  
only be limited by memory if a dict is used instead of ets.

Any comments on my design?

	Thanks, Joel

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont