[erlang-questions] Send message to thousands of clients

[Matthew Sackman]

On Wed, Nov 24, 2010 at 11:16:35PM +0300, Max Lapshin wrote:
> I remember, that there was such a discussion, but can't find it.
> 
> So, I have a problem: about 40 times per second process ems_media gets
> a frame, performs some actions and then retransmit it up to 1500
> clients and more.
> 
> Problem is in bad responsiveness of this process. Clients cannot wait
> when it reply for some tasks such as unsubscribe, etc.
> 
> Simple benchmarks shows that
> [Pid ! Message || Pid <- List]
> 
> can take upto 13 milliseconds when List length is 2000.
> 
> Question is: what should I do?

This is _highly_ experimental, but you could try my gm module which does
guaranteed multicast. It's rapidly changing and indeed may still contain
some bugs.

You need:
http://hg.rabbitmq.com/rabbitmq-ha/file/default/src/gm.erl

and will want to read
http://hg.rabbitmq.com/rabbitmq-ha/file/default/src/gm_test.erl
to figure out how it works. (You can more than ignore all the other
files in that repo!)

Overall latency will be _higher_ (i.e. end-to-end latency), but the
"sending" process does vastly less work and thus will return much
faster. It may or may not do quite what you need, but it might be worth
a look.

Matthew