Inets HTTP client stability

[Michael McDaniel]

On Thu, Oct 27, 2005 at 11:38:06AM -0700, Michael McDaniel wrote:
> On Thu, Oct 27, 2005 at 02:34:21PM +0200, Heinrich Venter wrote:
> > Hi all.
> >  
> > I am about to develop an app that will act as an http gateway for some of the
> > services we are running.  For the incoming http requests I plan to use httpd
> > from inets.  For the outgonig leg it would thus make sense to use the http
> > client from inets.
> > My question is wether any one else is using the inets http client for medium to
> > high volume outgoing traffic?  I know of two instances where people chose to
> > create a java node to make http connections, instead of using inets.
> > Is it worth the risk to try the inets solution first, or should I budget the
> > time to make a java node and try to upgrade to inets later?
> > Any one with experience in this regard?
> >  
> > The requirements for the http client will be that many connections will be made
> > to a series of speciffic URLs.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> I have successfully been using http:request/4 for doing POSTs for a 
> few months now in a production system.  I am very happy with the
> results (and Erlang!).
> 
> See code below...
> 
> The production system is not supporting heavy traffic (though the customer
> hopes to make more sales!); however, during testing I was very pleased.
> 
> The destination server is provided by another company, and their
> testing server could not keep up with my load testing.  At one point I
> had between 800-900 http:request/4s backed up retrying as described 
> below.  All the POSTs eventually were successful with a response from the
> test server.  An abbreviated version of the code is below...
> 
> 
> -define(DEFAULT_TIMEOUT, 2000).
> MyFun(Timeout, Body) ->
> 
>   case (catch http:request(post,
>   {"http://example.com/path/" ,
>   [ 
>      {"Date", httpd_util:rfc1123_date()} ,
>      {"Host", "example.com"} ,
>      {"Accept", "text/xml/html"},
>      {"User-Agent",  "Demo Erlang Client"} 
>   ], "text/xml; charset=utf-8", Body },
>      [{keepalive, false}, {nodelay,true}], [])) of
>        {ok, Result}    -> {ok, Result} ;
>        {error, Reason} -> 
> 		      Result = null , 
>                       mylogger:log_error(?MODULE, ?LINE,
> 		      'MyFun, http:request', "failed, see /tmp/my.log"),
> 		      {ok, Io} = file:open('/tmp/my.log', [append]) ,
> 	 	      io:fwrite(Io, "~26p, ~p~n",
> 					[erlang:localtime(), Reason]) ,
> 		      file:close(Io) ,
> 		      if TimeOut > 8192000 ->  % milliseconds
> 	 		        timer:sleep(TimeOut/8) , 
> 				?MODULE:MyFun(?DEFAULT_TIMEOUT,Body);
> 			 true ->
> 	 		        timer:sleep(TimeOut) , 
> 				?MODULE:MyFun(2*TimeOut,Body) 
> 		      end 
>   end ,
> 
> 	%%% Do something with Result ...
> 
> end.
> 
> Of course you could give up eventually but I "know" I'll eventually be
> able to get where I'm POSTing to, hence the endless retries.
> 
> You may prefer to spawn rather than serialize the repeat fun calls as
> I have done.
> 
> Please anyone mention if there are more accepted/efficient ways of
> accomplishing the above.  I still consider myself very novice with
> Erlang and Erlang idioms.  I have not yet started using OTP yet,
> and still am writing "pure" Erlang.  I have a new project coming up
> and will probably dabble with OTP on that one.
> 
> ~Michael
> 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

An update regarding the above code in production.  I recently
discovered (see my message with
            "Subject: badmatch with chunked data (LONG)")

that my code does not handle chunked data coming back from
the server as required by rfc2616 in

Section 3.6.1 Chunked Transfer Coding

...

   All HTTP/1.1 applications MUST be able to receive and decode the
   "chunked" transfer-coding, and MUST ignore chunk-extension
   extensions they do not understand.
----

I still do not understand why I am getting the badmatch from
http:request/4 as described in my message with
             "Subject: badmatch with chunked data (LONG)"

It appears that http:request/4 itself is crashing.

Having said all that, I tried ibrowse (Chandrus http client) as
suggested by Peter Lund in this thread.  It works with the
chunked data.

So, if you continue to use http:request/4, be sure to test it
with chunked data before using it in production.

~Michael
Portland, Oregon, USA