[erlang-questions] httpc manager crashes

[Arun Suresh]

Helo Ingela

I am Suma's colleague. Let me try to give more context to the issue we are
facing..

* Our app make a LOT (in the range of around 300) of http requests
concurrently.
* There are two types of requests
   1) POST requests : These require us to go thru a proxy. These can
themselves hit about 150  - 200concurrent requests.
   2) GET requests :  Standard http requests.

As far as the GET requests are concerned, we start some 300 profiles and
randomly pick one for use.

Since we have to send our POSTs via a proxy, at app initialialization time,
we start 1 httpc profile. We see a lot of httpc_manager crashes pertaining
to this particular profile. I understand that the httpc_manager is supposed
to be a pretty light weight process.. not requiring any synchronization with
the manager.. Is there a chance that the httpc_manager can crash ? Also, I
was wondering, is the Data to be posted also copied to the httpc_managers
message queue before its again copied to the handlers message queue before
actually sending ?


Regards
-Arun



On Fri, Apr 29, 2011 at 12:49 PM, Ingela Andin <ingela@REDACTED> wrote:

> Hi!
>
> A httpc-profile could be view as a "browser core", it saves options
> that you want to use for all requests such as proxy-options,
> it handles cookies, it dispatches requests for pipelining or
> persistent HTTP-connections queues, it could even do caching but that
> is not implemented at the moment.  There is no hard limit on how many
> profiles you can use, a profile is mainly a new dispatcher process.
> 300 erlang-processes is no problem at all, but starting a new profile
> for every request I think is a bit of an odd strategy.  Why do you do
> that? The manager process will spawn a new request-handler process for
> each request unless it can be pipelined, or if pipelining is not
> enabled by setting the pipeline timeout and there is a persistent
> connection that can be used to queue the request on. Length of
> pipeline and persistent
> connection-queues are configurable. After the request has been
> dispatched to the handler process that process will handle the request
> and send the answer to the client without any synchronizing with the
> manager process. The manager process should not be very busy so it
> should not be a big bottleneck and if it gets too busy we might have
> to change its implementation.
>
> We also think we know what is causing your problem and we will fix it
> as soon as possible.
>
> Regards Ingela Erlang/OTP team - Ericsson AB
>
>
> 2011/4/28 Suma Shivaprasad <sumasai.shivaprasad@REDACTED>:
> > Hi,
> >
> > We are using inets - 5.5.1 and HTTP/1.1.  We are creating a new httpc
> > profile for every request that we make and around 300 concurrent requests
> > were running at the same time . Is this fine?  How many maximum profiles
> has
> > httpc been tested with for concurrent use?
> >
> > Thanks
> > Suma
> >
> > On Thu, Apr 28, 2011 at 2:53 PM, Ingela Andin <ingela.andin@REDACTED>
> > wrote:
> >>
> >> Hi!
> >>
> >> What version of inets and Erlang are you using?  What HTTP version
> >> HTTP-1.0 ?
> >> If there is a bug we will of course fix it.
> >>
> >> Regards Ingela Erlang/OTP - Ericsson AB
> >>
> >>
> >> 2011/4/27 Suma Shivaprasad <sumasai.shivaprasad@REDACTED>:
> >> > We have an app that makes a lot of calls to httpc and we are seeing
> the
> >> > following error in sasl logs quite often.
> >> >
> >> > CRASH REPORT==== 27-Apr-2011::05:21:24 ===
> >> >   crasher:
> >> >     initial call: httpc_manager:init/1
> >> >     pid: <0.10297.0>
> >> >     registered_name: httpc_manager_profile_8261
> >> >     exception exit: {{case_clause,
> >> >
> [{handler_info,#Ref<0.0.398.166371>,undefined,
> >> >                               <0.12971.248>,undefined,operational}]},
> >> >                      [{httpc_manager,handle_connect_and_send,5},
> >> >                       {httpc_manager,handle_info,2},
> >> >                       {gen_server,handle_msg,5},
> >> >                       {proc_lib,init_p_do_apply,3}]}
> >> >       in function  gen_server:terminate/6
> >> >     ancestors: [httpc_profile_sup,httpc_sup,inets_sup,<0.46.0>
> >> >
> >> >
> >> >
> >> > and
> >> >
> >> >
> >> > =CRASH REPORT==== 27-Apr-2011::05:30:18 ===
> >> >   crasher:
> >> >     initial call: httpc_manager:init/1
> >> >     pid: <0.22933.12>
> >> >     registered_name: httpc_manager_profile_8122
> >> >     exception exit: {{case_clause,
> >> >
> >> > [{handler_info,#Ref<0.0.418.148452>,<0.20608.257>,
> >> >                               undefined,<0.15267.257>,initiating}]},
> >> >                      [{httpc_manager,handle_connect_and_send,5},
> >> >                       {httpc_manager,handle_info,2},
> >> >                       {gen_server,handle_msg,5},
> >> >                       {proc_lib,init_p_do_apply,3}]}
> >> >       in function  gen_server:terminate/6
> >> >     ancestors: [httpc_profile_sup,httpc_sup,inets_sup,<0.46.0>]
> >> >     messages: [{'EXIT',<0.20574.257>,normal}]
> >> >     links: [<0.50.0>,<0.20608.257>]
> >> >     dictionary: []
> >> >
> >> >
> >> > Has anyone else seen these errors or have a fix for this?
> >> >
> >> > Thanks
> >> > Suma
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > erlang-questions mailing list
> >> > erlang-questions@REDACTED
> >> > http://erlang.org/mailman/listinfo/erlang-questions
> >> >
> >> >
> >
> >
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110429/577773d7/attachment.htm>