[erlang-questions] Comet server broadcasting messages to 40k clients

[Bernard Duggan]

Hi Adriano,

My first suggestion would be to check that you're using kernel polling
(do you start erl with "+K true"?  You should see the string
"[kernel-poll:true]" in the version line).  If not, that's probably a
significant part of it, since for every packet that comes in the VM will
be iterating over all 40k sockets to figure out which one it came from
before the message even arrives at your code.

Also, be aware that using the -- operator on a big list can be a very
expensive operation (there was a thread about it last week, in fact), so
using it to remove a single socket from a list that big could be a
significant cost.

Finally, I'd suggest that, if you haven't already tried it, you spawn a
separate thread (process, in erlang parlance) to handle each connection
rather than trying to deal with them all in one thread.  While I don't
think this is going to improve your performance here, it makes your code
simpler and easier to follow.  Remember, erlang processes are not like C
threads - they're cheap, plentiful and carry a much lower switching
overhead.  This would involve a little more work for your case since you
want to broadcast to everyone based on a signal from anyone, but it is
more in keeping with "The Erlang Way(tm)".  (Actual followers of The
Erlang Way are of course free to correct my views...).

Cheers,

Bernard

Adriano Bonat wrote:
> Hi,
>
> I implemented a simple Comet server using gen_tcp (code follows below
> this text), and I'm doing tests with 40k connected clients. With a
> certain event, a message will be broadcasted to all connected clients,
> in my current implementation you just have to access the URI "/send".
>
> The current problem is that the first connected client will receive
> the message after ~5s (in my implementation a new client goes in the
> list's head), and this time is so high compared to a C version of the
> server. I tried to understand where time is being spent using tools
> like eprof and fprof, but I'm not that experienced trying to
> understand their results.
>
> This makes me think if I'm doing something wrong, if I can optimize it
> in some way... or if this is the normal overhead imposed by running on
> top of a VM...
>
> Any ideas?
>
> Thanks.
>
>
> -module(myhttpd).
>
> -compile(export_all).
>
> -define(LISTEN_PORT, 8000).
> -define(LISTEN_OPTS, [list, {packet, 0}, {keepalive, true}, {active, true}]).
>
> start() ->
>     register(router, spawn(fun handle_connections/0)),
>
>     {ok, Listen} = gen_tcp:listen(?LISTEN_PORT, ?LISTEN_OPTS),
>     spawn(fun() -> accept_connection(Listen) end).
>
> accept_connection(Listen) ->
>     {ok, Socket} = gen_tcp:accept(Listen),
>     router ! {accepted, Socket},
>     gen_tcp:controlling_process(Socket, whereis(router)),
>     accept_connection(Listen).
>
> handle_connections() ->
>     handle_connections([]).
>
> handle_connections(Sockets) ->
>     receive
>         {accepted, Socket} ->
>             io:format("Just accepted ~p~n", [Socket]),
>             handle_connections([Socket|Sockets]);
>         {tcp, Socket, "GET /send" ++ _T} ->
>             io:format("Sending messages.. ~n", []),
>             TStart = now(),
>             spawn(fun() ->
>                 lists:foreach(fun(S) -> send_chunk(S,
> io_lib:format("(~b usecs) Hello world!<br>", [timer:now_diff(now(),
> TStart)])) end, Sockets)
>             end),
>             gen_tcp:send(Socket, "HTTP/1.1 200 OK\r\n\r\nDone."),
>             gen_tcp:close(Socket),
>             handle_connections(Sockets -- [Socket]);
>         {tcp_closed, Socket} ->
>             io:format("Closed connection ~p~n", [Socket]),
>             handle_connections(Sockets -- [Socket]);
>         {tcp, Socket, X} ->
>             io:format("Rcved: ~p~n", [X]),
>             start_response(Socket),
>             handle_connections(Sockets);
>         {numclients} ->
>             io:format("Num clients: ~p~n", [length(Sockets)]),
>             handle_connections(Sockets);
>         Any ->
>             io:format("Just got ~p~n", [Any]),
>             handle_connections(Sockets)
>     end.
>
> start_response(Socket) ->
>     gen_tcp:send(Socket, "HTTP/1.1 200 OK\r\nContent-type:
> text/html;charset=utf-8\r\nTransfer-encoding: chunked\r\n\r\n").
>
> end_response(Socket) ->
>     gen_tcp:send(Socket, "0\r\n\r\n").
>
> send_chunk(Socket, Chunk) ->
>     gen_tcp:send(Socket, io_lib:format("~.16b\r\n~s\r\n",
> [iolist_size(Chunk), Chunk])).
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>