[erlang-questions] Frequent crashes in inets http client (R12B-5)
Oscar Hellström
oscar@REDACTED
Fri Jun 5 20:55:28 CEST 2009
Hi Chris,
Actually, the OTP team has responded to this, but privately. I did the
mistake of sending it to erlang-bugs and CC:d erlang-questions. This
somehow seem to make the mail end up in the erlang-question's archive,
but was never sent out. An email was however sent out to erlang-bugs'
members. The response from the OTP team is quoted below:
> Hi,
>
> Thanks for the input (good analyzis). I will add it
> to my inets todo-list (which is getting quite long :)
> There is no time to deal with these issues in the
> R13B01 release, but hopefully R13B02.
>
> Regards,
> /BMK
Chris Newcombe wrote:
> Hi Oscar,
>
> I did see your earlier post -- very useful, thanks very much. (It
> would be great to hear from the OTP team on this.)
>
> I normally use ibrowse, but as ibrowse does not yet use binaries
> internally it's not working well for my current project. I have
> exactly the issue you mentioned in your post (on ibrowse) just now --
> I need to concurrently receive many multi-MB response bodies, and due
> to ibrowse's use of lists the memory overhead is crippling -- I'm too
> am running on 64-bit. My current project has to co-exist with other
> important applications, so memory usage (and much higher cpu from all
> of the reversal/flattening/copying/garbage-collection) is a real
> practical issue.
>
> A couple of days ago I asked Chandru how much work it would be to
> change ibrowse to use binaries, and he very kindly said he would look
> at it. (Thankyou again Chandru!)
>
> If other users of ibrowse would like this to happen, then it might be
> useful to reply publicly to this post and say so. (e.g. I heard that
> the CouchDb team are also very interested.) I'm guessing that
> Chandru would appreciate assistance with testing, and perhaps even
> with code (although I have not asked him).
>
> regards,
>
> Chris
>
>
> 2009/6/5 Oscar Hellström <oscar@REDACTED
> <mailto:oscar@REDACTED>>
>
> Not to start a flame war, but I would stay away from the inets http
> client if I were trying to build something serious. You can find my
> reasons here:
> http://www.erlang.org/cgi-bin/ezmlm-cgi?4:mss:43806:200905:gocblgddeplfolmoleep
>
> Best regards
>
> Chris Newcombe wrote:
> > Is there a patch for the following issue?
> >
> >
> >
> > It was reported a while ago:
> >
> http://groups.google.com/group/erlang-programming/browse_thread/thread/4c497978c75ed6a9/18d9a242df81ba3a?lnk=gst&q=badrecord#18d9a242df81ba3a(but
> <http://groups.google.com/group/erlang-programming/browse_thread/thread/4c497978c75ed6a9/18d9a242df81ba3a?lnk=gst&q=badrecord#18d9a242df81ba3a%28but>
> > I didn't see any replies)
> >
> >
> >
> > Here's a bit more detail:
> >
> >
> >
> > httpc_handler is crashing with
> >
> >
> >
> > {badrecord,request}
> >
> >
> >
> > (BTW it would be great if badrecord errors also contained the
> incorrect
> > term, not just the name of the expected record type)
> >
> >
> >
> > It’s crashing in
> >
> >
> >
> > httpc_handler,handle_info,2
> >
> >
> >
> > The last message received by the gen_server is
> >
> >
> >
> > {timeout,#Ref<0.0.0.9038>}
> >
> >
> >
> > The gen_server #state is
> >
> >
> >
> >
> >
> {state,undefined,{tcp_session,{{"my-test-url",8080},<0.709.0>},false,http,#Port<0.1351>,1},undefined,undefined,undefined,undefined,{[],[]},pipeline,[#Ref<0.0.0.5834>],nolimit,nolimit,{options,{undefined,[]},20000,1,100,disabled,enabled,false},{timers,[],#Ref<0.0.0.19293>}
> >
> >
> >
> > I think the relevant element is the first one (request).
> >
> > i.e. request == undefined
> >
> >
> >
> > Given the message, it seems almost certain that the crash is in
> the second
> > timeout clause of handle_info,
> >
> > (marked below with ***).
> >
> > This clause will fire even if request == undefined, but will try
> to use
> > Request#request.from, which crashes with {badrecord,request}
> >
> >
> >
> > %%% Timeouts
> >
> > %% Internaly, to a request handling process, a request
> time out is
> >
> > %% seen as a canceld request.
> >
> > handle_info({timeout, RequestId}, State =
> >
> > #state{request = Request = #request{id =
> RequestId}}) ->
> >
> > httpc_response:send(Request#request.from,
> >
> > httpc_response:error(Request,timeout)),
> >
> > {stop, normal,
> >
> > State#state{canceled = [RequestId |
> State#state.canceled],
> >
> > request = Request#request{from =
> answer_sent}}};
> >
> >
> >
> > *** handle_info({timeout, RequestId}, State = #state{request
> = Request})
> > ->
> >
> > httpc_response:send(Request#request.from,
> >
> >
> httpc_response:error(Request,timeout)),
> >
> > {noreply, State#state{canceled = [RequestId |
> > State#state.canceled]}};
> >
> >
> >
> > handle_info(timeout_pipeline, State = #state{request =
> undefined}) ->
> >
> > {stop, normal, State};
> >
> >
> >
> > It looks like State#state.request is being set to undefined without
> > cancelling an in-progress request timer.
> >
> >
> > I've only glanced at the code, but both of the following clauses
> appear to
> > do that.
> >
> > (But it could easily be something else.)
> >
> >
> >
> > %% On a redirect or retry the current request becomes
> >
> > %% obsolete and the manager will create a new request
> >
> > %% with the same id as the current.
> >
> > {redirect, NewRequest, Data}->
> >
> > ok = httpc_manager:redirect_request(NewRequest,
> ProfileName),
> >
> > handle_pipeline(State#state{request = undefined}, Data);
> >
> > {retry, TimeNewRequest, Data}->
> >
> > ok = httpc_manager:retry_request(TimeNewRequest,
> ProfileName),
> >
> > handle_pipeline(State#state{request = undefined}, Data);
> >
> >
> >
> > thanks,
> >
> >
> >
> > Chris
> >
> >
>
>
> --
> Oscar Hellström, oscar@REDACTED
> <mailto:oscar@REDACTED>
> Office: +44 20 7655 0337
> Mobile: +44 798 45 44 773
> Erlang Training and Consulting Ltd
> http://www.erlang-consulting.com/
>
>
--
Oscar Hellström, oscar@REDACTED
Office: +44 20 7655 0337
Mobile: +44 798 45 44 773
Erlang Training and Consulting Ltd
http://www.erlang-consulting.com/
More information about the erlang-questions
mailing list