[erlang-questions] my socket can't handle the load

[Dmitry Belyaev]

I've had the same problems with my emongo driver. Finally, I found that
under heavy load received packet could consist of *two* responses but my
code did process only first.

Maybe this is your case too.

On Sat, 2010-06-05 at 02:13 -0700, Pablo Platt wrote:
> I was able to identify exactly when the issue happens.
> As long as only one request at a time is sent to the driver it works fine:
> request1, response1, request2, response2...
> 
> If the client sends several requests as fast as it can without waiting for a response it breaks.
> debug messages show the following:
> request1, request2, response1, request3, response2, Error timeout for request3, response3
> 
> This failure is reproducible. The timeout occurs after 5000ms compared to ~10ms for a normal query.
> It can't be related to too many messages in a process mail box because we are only talking about 3 requests.
> I don't know if this issue is related to my driver or the db.
> 
> Is there something related to sockets, or TCP that breaks when you send and receive data in a non-blocking way?
> Is there an erlang driver that sends messages to a socket in a non-blocking way and handle responses according to a unique id tag
> that I can use as a reference to see if any special configuration is needed?
> 
> 
> 
> 
> 
> 
> ________________________________
> From: Kaiduan Xie <kaiduanx@REDACTED>
> To: Pablo Platt <pablo.platt@REDACTED>
> Cc: Bernard Duggan <bernie@REDACTED>; Erlang <erlang-questions@REDACTED>
> Sent: Thu, June 3, 2010 3:14:33 PM
> Subject: Re: [erlang-questions] my socket can't handle the load
> 
> Pablo, you can use erlang:process_info(Pid, message_queue_len) to find
> the mail box size of gen_server.
> 
> On Thu, Jun 3, 2010 at 2:52 AM, Pablo Platt <pablo.platt@REDACTED> wrote:
> > p.s.
> > when getting a gen_server:call timeout the debug message of the
> > requester(not the socket gen_server) says:
> > heap_size: 1597
> > stack_size: 24
> > reductions: 1996
> >
> >
> > ________________________________
> > From: Pablo Platt <pablo.platt@REDACTED>
> > To: Kaiduan Xie <kaiduanx@REDACTED>; Bernard Duggan <bernie@REDACTED>
> > Cc: Erlang <erlang-questions@REDACTED>
> > Sent: Thu, June 3, 2010 9:49:17 AM
> > Subject: Re: [erlang-questions] my socket can't handle the load
> >
> > @Bernard, Kaiduan
> > When I'm testing with a pool of 20 connections instead of 1, the driver and
> > socket works fine.
> > I think that means that both the receiver (database) and the sender can
> > handle the load.
> > When sending one request at a time from the same socket it also works fine
> > but don't think this is the correct design.
> >
> > I don't think {active, once} is relevant here because responses from the db
> > will only arrive when I make requests
> > so I don't need to protect myself against DOS from untrusted third party.
> >
> > When getting a timeout on gen_server:call to the socket gen_server, how can
> > I print useful info
> > about the socket gen_server and the process calling it so I can see the
> > mailbox queue of them and maybe other useful
> > stuff that help me find the problem?
> >
> > Thanks
> >
> > ________________________________
> > From: Kaiduan Xie <kaiduanx@REDACTED>
> > To: Bernard Duggan <bernie@REDACTED>
> > Cc: Pablo Platt <pablo.platt@REDACTED>; Erlang
> > <erlang-questions@REDACTED>
> > Sent: Thu, June 3, 2010 3:59:02 AM
> > Subject: Re: [erlang-questions] my socket can't handle the load
> >
> > Pablo,
> >
> > Have you considered any chance the call of
> > gen_tcp:send(State#state.socket, Packet) getting blocked? For example,
> > a slow receiver. If gen_tcp:send() is getting blocked, hand_call()
> > will get blocked, and gen_server will slow down. Please look the
> > example listed in gen_tcp,
> >
> > http://www.erlang.org/doc/man/gen_tcp.html#examples
> >
> > Kaiduan
> >
> > On Wed, Jun 2, 2010 at 5:35 PM, Bernard Duggan <bernie@REDACTED> wrote:
> >> Hi Pablo,
> >>    One thing you may want to try is to wherever possible, avoid letting
> >> the queue on your gen_server grow beyond a couple of messages.  Allowing
> >> the queue to grow unchecked can cause serious performance degradation on
> >> selective receives (which, to be fair, I can't see any of in your code,
> >> but they can crop up in non-obvious library calls at times).
> >> Off the top of my head, you'd do this by:
> >> a) changing the handle_cast operation to a handle_call (to keep clients
> >> using that operation from flooding you with requests) and
> >> b) changing {active, true} to {active, once} in your connect call (and
> >> making the corresponding change in handle_info to reactivate the socket.
> >>
> >> It may or may not be your problem, but it's a fairly easy change and
> >> worth a try.
> >>
> >> Another possibility is that you have the same issue, but in the calling
> >> process (I'm guessing now, since you haven't provided that code).  A
> >> gen_server call does a selective receive while it waits for a response -
> >> if your message queue is very large when you make the call, there's a
> >> good chance you'll get a timeout regardless of how quickly the
> >> gen_server serves the request.
> >>
> >> Cheers,
> >>
> >> Bernard
> >>
> >> On 3/06/2010 1:27 AM, Pablo Platt wrote:
> >>> Hi,
> >>>
> >>> I'm writing a driver to a database that is using a tcp socket.
> >>> There are two types of messages, "send and forget" and "send and
> >>> receive".
> >>> I have a gen_server that is responsible for opening the socket, receive a
> >>> message from a process and sending it to the socket and receive responses
> >>> and pass them to the caller.
> >>> The gen_server saves a list of {request_id, CallerPid} in the state to
> >>> know who to respond to when a packet is received from the db.
> >>>
> >>> Everything works when the rate of messages is low but when increasing it
> >>> I'm starting to get gen_server timeout error on the requestor.
> >>> Do I need to add/change parameters when opening the socket?
> >>> Should I queue requests and wait for a response before sending the next
> >>> request or is it ok to send several requests one after the other?
> >>>
> >>> Thanks
> >>>
> >>> The client sends a request using either:
> >>> Resp = gen_server:call(Conn, {request, Request})
> >>> or
> >>> gen_server:cast(Conn, {request, Request})
> >>>
> >>>
> >>> The relevant gen_server code:
> >>>
> >>> init([Host, Port]) ->
> >>>     Socket = open_socket(Host, Port),
> >>>     {ok, #state{socket=Socket, req_id=1}}.
> >>>
> >>> open_socket(Host, Port) ->
> >>>     case gen_tcp:connect(Host, Port, [binary, {active, true}]) of
> >>>         {ok, Sock} ->
> >>>             Sock;
> >>>         {error, Reason} ->
> >>>             exit({open_socket_failed, Reason})
> >>>     end.
> >>>
> >>> handle_call({request, Packet}, From, State) ->
> >>>     ReqID = State#state.req_id + 1,
> >>>     gen_tcp:send(State#state.socket, Packet),
> >>>     {noreply, State#state{req_id=ReqID, requests=[{ReqID,
> >>> From}|State#state.requests]}};
> >>>
> >>> handle_cast({request, Packet}, State) ->
> >>>     ReqID = State#state.req_id + 1,
> >>>     gen_tcp:send(State#state.socket, Packet),
> >>>     {noreply, State#state{req_id=ReqID}}.
> >>>
> >>> handle_info({tcp, _Socket, Data}, State) ->
> >>>     RawResp = <<(State#state.resp)/binary, Data/binary>>,
> >>>     case check_packet:decode_response(RawResp) of
> >>>         undefined ->
> >>>             {noreply, State#state{resp = RawResp}};
> >>>         {Resp, Tail} ->
> >>>             ResponseTo = get_requestor(Resp),
> >>>             {value, {ResponseTo, Client}, NewRequests} =
> >>> lists:keytake(ResponseTo, 1, State#state.requests),
> >>>             gen_server:reply(Client, Resp),
> >>>             {noreply, State#state{resp = Tail, requests=NewRequests}}
> >>>     end;
> >>>
> >>> % the following never been called, even when I'm getting the error.
> >>> handle_info({tcp_closed, _Socket}, State) ->
> >>>     {noreply, State};
> >>>
> >>> handle_info({tcp_error, _Socket, _Reason}, State) ->
> >>>     {noreply, State}.
> >>>
> >>> terminate(_Reason, _State) ->
> >>>     ok.
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >> ________________________________________________________________
> >> erlang-questions (at) erlang.org mailing list.
> >> See http://www.erlang.org/faq.html
> >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >>
> >>
> >
> >
> >
> 
> 
> 
>