[erlang-questions] Frequent crashes in inets http client (R12B-5)

[Chris Newcombe]

Hi Oscar,

I did see your earlier post -- very useful, thanks very much.   (It would be
great to hear from the OTP team on this.)

I normally use ibrowse, but as ibrowse does not yet use binaries internally
it's not working well for my current project.   I have exactly the issue you
mentioned in your post (on ibrowse) just now -- I need to concurrently
receive many multi-MB response bodies, and due to ibrowse's use of lists the
memory overhead is crippling -- I'm too am running on 64-bit.  My current
project has to co-exist with other important applications, so memory usage
(and much higher cpu from all of the
reversal/flattening/copying/garbage-collection) is a real practical issue.

A couple of days ago I asked Chandru how much work it would be to change
ibrowse to use binaries, and he very kindly said he would look at it.
(Thankyou again Chandru!)

If other users of ibrowse would like this to happen, then it might be useful
to reply publicly to this post and say so.  (e.g. I heard that the CouchDb
team are also very interested.)   I'm guessing that Chandru would appreciate
assistance with testing, and perhaps even with code (although I have not
asked him).

regards,

Chris


2009/6/5 Oscar Hellström <oscar@REDACTED>

> Not to start a flame war, but I would stay away from the inets http
> client if I were trying to build something serious. You can find my
> reasons here:
>
> http://www.erlang.org/cgi-bin/ezmlm-cgi?4:mss:43806:200905:gocblgddeplfolmoleep
>
> Best regards
>
> Chris Newcombe wrote:
> > Is there a patch for the following issue?
> >
> >
> >
> > It was reported a while ago:
> >
> http://groups.google.com/group/erlang-programming/browse_thread/thread/4c497978c75ed6a9/18d9a242df81ba3a?lnk=gst&q=badrecord#18d9a242df81ba3a(but<http://groups.google.com/group/erlang-programming/browse_thread/thread/4c497978c75ed6a9/18d9a242df81ba3a?lnk=gst&q=badrecord#18d9a242df81ba3a%28but>
> > I didn't see any replies)
> >
> >
> >
> > Here's a bit more detail:
> >
> >
> >
> > httpc_handler is crashing with
> >
> >
> >
> >        {badrecord,request}
> >
> >
> >
> > (BTW it would be great if badrecord errors also contained the incorrect
> > term, not just the name of the expected record type)
> >
> >
> >
> > It’s crashing in
> >
> >
> >
> >        httpc_handler,handle_info,2
> >
> >
> >
> > The last message received by the gen_server is
> >
> >
> >
> >       {timeout,#Ref<0.0.0.9038>}
> >
> >
> >
> > The gen_server #state is
> >
> >
> >
> >
> >
> {state,undefined,{tcp_session,{{"my-test-url",8080},<0.709.0>},false,http,#Port<0.1351>,1},undefined,undefined,undefined,undefined,{[],[]},pipeline,[#Ref<0.0.0.5834>],nolimit,nolimit,{options,{undefined,[]},20000,1,100,disabled,enabled,false},{timers,[],#Ref<0.0.0.19293>}
> >
> >
> >
> > I think the relevant element is the first one (request).
> >
> > i.e. request == undefined
> >
> >
> >
> > Given the message, it seems almost certain that the crash is in the
> second
> > timeout clause of handle_info,
> >
> > (marked below with ***).
> >
> > This clause will fire even if request == undefined, but will try to use
> > Request#request.from, which crashes with {badrecord,request}
> >
> >
> >
> >        %%% Timeouts
> >
> >        %% Internaly, to a request handling process, a request time out is
> >
> >        %% seen as a canceld request.
> >
> >        handle_info({timeout, RequestId}, State =
> >
> >                    #state{request = Request = #request{id = RequestId}})
> ->
> >
> >            httpc_response:send(Request#request.from,
> >
> >                          httpc_response:error(Request,timeout)),
> >
> >            {stop, normal,
> >
> >             State#state{canceled = [RequestId | State#state.canceled],
> >
> >                         request = Request#request{from = answer_sent}}};
> >
> >
> >
> > ***    handle_info({timeout, RequestId}, State = #state{request =
> Request})
> > ->
> >
> >            httpc_response:send(Request#request.from,
> >
> >                               httpc_response:error(Request,timeout)),
> >
> >            {noreply, State#state{canceled = [RequestId |
> > State#state.canceled]}};
> >
> >
> >
> >        handle_info(timeout_pipeline, State = #state{request = undefined})
> ->
> >
> >            {stop, normal, State};
> >
> >
> >
> > It looks like State#state.request is being set to undefined without
> > cancelling an in-progress request timer.
> >
> >
> > I've only glanced at the code, but both of the following clauses appear
> to
> > do that.
> >
> > (But it could easily be something else.)
> >
> >
> >
> >         %% On a redirect or retry the current request becomes
> >
> >         %% obsolete and the manager will create a new request
> >
> >         %% with the same id as the current.
> >
> >         {redirect, NewRequest, Data}->
> >
> >             ok = httpc_manager:redirect_request(NewRequest, ProfileName),
> >
> >             handle_pipeline(State#state{request = undefined}, Data);
> >
> >         {retry, TimeNewRequest, Data}->
> >
> >             ok = httpc_manager:retry_request(TimeNewRequest,
> ProfileName),
> >
> >             handle_pipeline(State#state{request = undefined}, Data);
> >
> >
> >
> > thanks,
> >
> >
> >
> > Chris
> >
> >
>
>
> --
> Oscar Hellström, oscar@REDACTED
> Office: +44 20 7655 0337
> Mobile: +44 798 45 44 773
> Erlang Training and Consulting Ltd
> http://www.erlang-consulting.com/
>
>