Inets HTTP client stability

Michael McDaniel erlang@REDACTED
Sun Nov 13 20:34:05 CET 2005

On Thu, Oct 27, 2005 at 11:38:06AM -0700, Michael McDaniel wrote:
> On Thu, Oct 27, 2005 at 02:34:21PM +0200, Heinrich Venter wrote:
> > Hi all.
> >  
> > I am about to develop an app that will act as an http gateway for some of the
> > services we are running.  For the incoming http requests I plan to use httpd
> > from inets.  For the outgonig leg it would thus make sense to use the http
> > client from inets.
> > My question is wether any one else is using the inets http client for medium to
> > high volume outgoing traffic?  I know of two instances where people chose to
> > create a java node to make http connections, instead of using inets.
> > Is it worth the risk to try the inets solution first, or should I budget the
> > time to make a java node and try to upgrade to inets later?
> > Any one with experience in this regard?
> >  
> > The requirements for the http client will be that many connections will be made
> > to a series of speciffic URLs.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> I have successfully been using http:request/4 for doing POSTs for a 
> few months now in a production system.  I am very happy with the
> results (and Erlang!).
> See code below...
> The production system is not supporting heavy traffic (though the customer
> hopes to make more sales!); however, during testing I was very pleased.
> The destination server is provided by another company, and their
> testing server could not keep up with my load testing.  At one point I
> had between 800-900 http:request/4s backed up retrying as described 
> below.  All the POSTs eventually were successful with a response from the
> test server.  An abbreviated version of the code is below...
> -define(DEFAULT_TIMEOUT, 2000).
> MyFun(Timeout, Body) ->
>   case (catch http:request(post,
>   {"" ,
>   [ 
>      {"Date", httpd_util:rfc1123_date()} ,
>      {"Host", ""} ,
>      {"Accept", "text/xml/html"},
>      {"User-Agent",  "Demo Erlang Client"} 
>   ], "text/xml; charset=utf-8", Body },
>      [{keepalive, false}, {nodelay,true}], [])) of
>        {ok, Result}    -> {ok, Result} ;
>        {error, Reason} -> 
> 		      Result = null , 
>                       mylogger:log_error(?MODULE, ?LINE,
> 		      'MyFun, http:request', "failed, see /tmp/my.log"),
> 		      {ok, Io} = file:open('/tmp/my.log', [append]) ,
> 	 	      io:fwrite(Io, "~26p, ~p~n",
> 					[erlang:localtime(), Reason]) ,
> 		      file:close(Io) ,
> 		      if TimeOut > 8192000 ->  % milliseconds
> 	 		        timer:sleep(TimeOut/8) , 
> 			 true ->
> 	 		        timer:sleep(TimeOut) , 
> 				?MODULE:MyFun(2*TimeOut,Body) 
> 		      end 
>   end ,
> 	%%% Do something with Result ...
> end.
> Of course you could give up eventually but I "know" I'll eventually be
> able to get where I'm POSTing to, hence the endless retries.
> You may prefer to spawn rather than serialize the repeat fun calls as
> I have done.
> Please anyone mention if there are more accepted/efficient ways of
> accomplishing the above.  I still consider myself very novice with
> Erlang and Erlang idioms.  I have not yet started using OTP yet,
> and still am writing "pure" Erlang.  I have a new project coming up
> and will probably dabble with OTP on that one.
> ~Michael

An update regarding the above code in production.  I recently
discovered (see my message with
            "Subject: badmatch with chunked data (LONG)")

that my code does not handle chunked data coming back from
the server as required by rfc2616 in

Section 3.6.1 Chunked Transfer Coding


   All HTTP/1.1 applications MUST be able to receive and decode the
   "chunked" transfer-coding, and MUST ignore chunk-extension
   extensions they do not understand.

I still do not understand why I am getting the badmatch from
http:request/4 as described in my message with
             "Subject: badmatch with chunked data (LONG)"

It appears that http:request/4 itself is crashing.

Having said all that, I tried ibrowse (Chandrus http client) as
suggested by Peter Lund in this thread.  It works with the
chunked data.

So, if you continue to use http:request/4, be sure to test it
with chunked data before using it in production.

Portland, Oregon, USA

More information about the erlang-questions mailing list