Inets HTTP client stability

[Michael McDaniel]

On Thu, Oct 27, 2005 at 02:34:21PM +0200, Heinrich Venter wrote:
> Hi all.
>  
> I am about to develop an app that will act as an http gateway for some of the
> services we are running.  For the incoming http requests I plan to use httpd
> from inets.  For the outgonig leg it would thus make sense to use the http
> client from inets.
> My question is wether any one else is using the inets http client for medium to
> high volume outgoing traffic?  I know of two instances where people chose to
> create a java node to make http connections, instead of using inets.
> Is it worth the risk to try the inets solution first, or should I budget the
> time to make a java node and try to upgrade to inets later?
> Any one with experience in this regard?
>  
> The requirements for the http client will be that many connections will be made
> to a series of speciffic URLs.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I have successfully been using http:request/4 for doing POSTs for a 
few months now in a production system.  I am very happy with the
results (and Erlang!).

See code below...

The production system is not supporting heavy traffic (though the customer
hopes to make more sales!); however, during testing I was very pleased.

The destination server is provided by another company, and their
testing server could not keep up with my load testing.  At one point I
had between 800-900 http:request/4s backed up retrying as described 
below.  All the POSTs eventually were successful with a response from the
test server.  An abbreviated version of the code is below...


-define(DEFAULT_TIMEOUT, 2000).
MyFun(Timeout, Body) ->

  case (catch http:request(post,
  {"http://example.com/path/" ,
  [ 
     {"Date", httpd_util:rfc1123_date()} ,
     {"Host", "example.com"} ,
     {"Accept", "text/xml/html"},
     {"User-Agent",  "Demo Erlang Client"} 
  ], "text/xml; charset=utf-8", Body },
     [{keepalive, false}, {nodelay,true}], [])) of
       {ok, Result}    -> {ok, Result} ;
       {error, Reason} -> 
		      Result = null , 
                      mylogger:log_error(?MODULE, ?LINE,
		      'MyFun, http:request', "failed, see /tmp/my.log"),
		      {ok, Io} = file:open('/tmp/my.log', [append]) ,
	 	      io:fwrite(Io, "~26p, ~p~n",
					[erlang:localtime(), Reason]) ,
		      file:close(Io) ,
		      if TimeOut > 8192000 ->  % milliseconds
	 		        timer:sleep(TimeOut/8) , 
				?MODULE:MyFun(?DEFAULT_TIMEOUT,Body);
			 true ->
	 		        timer:sleep(TimeOut) , 
				?MODULE:MyFun(2*TimeOut,Body) 
		      end 
  end ,

	%%% Do something with Result ...

end.

Of course you could give up eventually but I "know" I'll eventually be
able to get where I'm POSTing to, hence the endless retries.

You may prefer to spawn rather than serialize the repeat fun calls as
I have done.

Please anyone mention if there are more accepted/efficient ways of
accomplishing the above.  I still consider myself very novice with
Erlang and Erlang idioms.  I have not yet started using OTP yet,
and still am writing "pure" Erlang.  I have a new project coming up
and will probably dabble with OTP on that one.

~Michael