[erlang-questions] my socket can't handle the load

[Pablo Platt]

p.s.
when getting a gen_server:call timeout the debug message of the requester(not the socket gen_server) says:
heap_size: 1597
stack_size: 24
reductions: 1996





________________________________
From: Pablo Platt <pablo.platt@REDACTED>
To: Kaiduan Xie <kaiduanx@REDACTED>; Bernard Duggan <bernie@REDACTED>
Cc: Erlang <erlang-questions@REDACTED>
Sent: Thu, June 3, 2010 9:49:17 AM
Subject: Re: [erlang-questions] my socket can't handle the load


@Bernard, Kaiduan
When I'm testing with a pool of 20 connections instead of 1, the driver and socket works fine.
I think that means that both the receiver (database) and the sender can handle the load.
When sending one request at a time from the same socket it also works fine but don't think this is the correct design.

I don't think {active, once} is relevant here because responses from the db will only arrive when I make requests
so I don't need to protect myself against DOS from untrusted third party.

When getting a timeout on gen_server:call to the socket gen_server, how can I print useful info
about the socket gen_server and the process calling it so I can see the mailbox queue of them and maybe other useful 
stuff that help me find the problem?

Thanks




________________________________
From: Kaiduan Xie <kaiduanx@REDACTED>
To: Bernard Duggan <bernie@REDACTED>
Cc: Pablo Platt <pablo.platt@REDACTED>; Erlang <erlang-questions@REDACTED>
Sent: Thu, June 3, 2010 3:59:02 AM
Subject: Re: [erlang-questions] my socket can't handle the load

Pablo,

Have you considered any chance the call of
gen_tcp:send(State#state.socket, Packet) getting blocked? For example,
a slow receiver. If gen_tcp:send() is getting blocked, hand_call()
will get blocked, and gen_server will slow down. Please look the
example listed in gen_tcp,

http://www.erlang.org/doc/man/gen_tcp.html#examples

Kaiduan

On Wed, Jun 2, 2010 at 5:35 PM, Bernard Duggan <bernie@REDACTED> wrote:
> Hi Pablo,
>    One thing you may want to try is to wherever possible, avoid letting
> the queue on your gen_server grow beyond a couple of messages.  Allowing
> the queue to grow unchecked can cause serious performance degradation on
> selective receives (which, to be fair, I can't see any of in your code,
> but they can crop up in non-obvious library calls at times).
> Off the top of my head, you'd do this by:
> a) changing the handle_cast operation to a handle_call (to keep clients
> using that operation from flooding you with requests) and
> b) changing {active, true} to {active, once} in your connect call (and
> making the corresponding change in handle_info to reactivate the socket.
>
> It may or may not be your problem, but it's a fairly easy change and
> worth a try.
>
> Another possibility is that you have the same issue, but in the calling
> process (I'm guessing now, since you haven't provided that code).  A
> gen_server call does a selective receive while it waits for a response -
> if your message queue is very large when you make the call, there's a
> good chance you'll get a timeout regardless of how quickly the
> gen_server serves the request.
>
> Cheers,
>
> Bernard
>
> On 3/06/2010 1:27 AM, Pablo Platt wrote:
>> Hi,
>>
>> I'm writing a driver to a database that is using a tcp socket.
>> There are two types of messages, "send and forget" and "send and receive".
>> I have a gen_server that is responsible for opening the socket, receive a message from a process and sending it to the socket and receive responses and pass them to the caller.
>> The gen_server saves a list of {request_id, CallerPid} in the state to know who to respond to when a packet is received from the db.
>>
>> Everything works when the rate of messages is low but when increasing it I'm starting to get gen_server timeout error on the requestor.
>> Do I need to add/change parameters when opening the socket?
>> Should I queue requests and wait for a response before sending the next request or is it ok to send several requests one after the other?
>>
>> Thanks
>>
>> The client sends a request using either:
>> Resp = gen_server:call(Conn, {request, Request})
>> or
>> gen_server:cast(Conn, {request, Request})
>>
>>
>> The relevant gen_server code:
>>
>> init([Host, Port]) ->
>>     Socket = open_socket(Host, Port),
>>     {ok, #state{socket=Socket, req_id=1}}.
>>
>> open_socket(Host, Port) ->
>>     case gen_tcp:connect(Host, Port, [binary, {active, true}]) of
>>         {ok, Sock} ->
>>             Sock;
>>         {error, Reason} ->
>>             exit({open_socket_failed, Reason})
>>     end.
>>
>> handle_call({request, Packet}, From, State) ->
>>     ReqID = State#state.req_id + 1,
>>     gen_tcp:send(State#state.socket, Packet),
>>     {noreply, State#state{req_id=ReqID, requests=[{ReqID, From}|State#state.requests]}};
>>
>> handle_cast({request, Packet}, State) ->
>>     ReqID = State#state.req_id + 1,
>>     gen_tcp:send(State#state.socket, Packet),
>>     {noreply, State#state{req_id=ReqID}}.
>>
>> handle_info({tcp, _Socket, Data}, State) ->
>>     RawResp = <<(State#state.resp)/binary, Data/binary>>,
>>     case check_packet:decode_response(RawResp) of
>>         undefined ->
>>             {noreply, State#state{resp = RawResp}};
>>         {Resp, Tail} ->
>>             ResponseTo = get_requestor(Resp),
>>             {value, {ResponseTo, Client}, NewRequests} = lists:keytake(ResponseTo, 1, State#state.requests),
>>             gen_server:reply(Client, Resp),
>>             {noreply, State#state{resp = Tail, requests=NewRequests}}
>>     end;
>>
>> % the following never been called, even when I'm getting the error.
>> handle_info({tcp_closed, _Socket}, State) ->
>>     {noreply, State};
>>
>> handle_info({tcp_error, _Socket, _Reason}, State) ->
>>     {noreply, State}.
>>
>> terminate(_Reason, _State) ->
>>     ok.
>>
>>
>>
>>
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>