[erlang-questions] Comet server broadcasting messages to 40k clients
Wed Jan 21 17:39:58 CET 2009
On Wed, Jan 21, 2009 at 5:38 AM, Adriano Bonat <adrianob@REDACTED> wrote:
> Hi Bernard,
> On Wed, Jan 21, 2009 at 2:52 AM, Bernard Duggan <bernie@REDACTED> wrote:
>> My first suggestion would be to check that you're using kernel polling
>> (do you start erl with "+K true"? You should see the string
>> "[kernel-poll:true]" in the version line). If not, that's probably a
>> significant part of it, since for every packet that comes in the VM will
>> be iterating over all 40k sockets to figure out which one it came from
>> before the message even arrives at your code.
> Yes, I ran the tests using kernel polling.
>> Also, be aware that using the -- operator on a big list can be a very
>> expensive operation (there was a thread about it last week, in fact), so
>> using it to remove a single socket from a list that big could be a
>> significant cost.
> That's one of the Joe's advices on his book :)
> But the latency for the message is not caused by that point...
>> Finally, I'd suggest that, if you haven't already tried it, you spawn a
>> separate thread (process, in erlang parlance) to handle each connection
>> rather than trying to deal with them all in one thread. While I don't
>> think this is going to improve your performance here, it makes your code
>> simpler and easier to follow. Remember, erlang processes are not like C
>> threads - they're cheap, plentiful and carry a much lower switching
>> overhead. This would involve a little more work for your case since you
>> want to broadcast to everyone based on a signal from anyone, but it is
>> more in keeping with "The Erlang Way(tm)". (Actual followers of The
>> Erlang Way are of course free to correct my views...).
> Yes, that would go with the Erlang's way, but I wanted a way that
> works faster. Before I implemented my own http server, I was using
> mochiweb, and it uses a thread per connection and has the same
> performance problem as using a single thread holding several
> Any other ideas are welcome, and thanks for you reply :)
Maybe it's a problem with your platform? What version of Erlang and
which OS/distro are you running?
I haven't seen anything like 5 sec latency with mochiweb even with a
lot of sockets connected.
More information about the erlang-questions