confusing returns from httpc:request/4

Wed Oct 27 07:25:56 CEST 2010

Sorry, I missed out on stating that to make the problem go away, I
also set the HTTP version as "HTTP/1.0"

So you may also want to look at the code for handling persistent
connections in general, especially when the server only supports HTTP
1.0 .

  Amit

On Oct 21, 8:54 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
> Hi!
>
> Yes thank you for the information. I think I have located the bug. It seems
> that to avoid crash reports the return value when receiving a tcp_closed
> was changed and this altered the terminate clause that is run,  alas breaking
> the automatic retries that should be done if the server terminates a connection
> before serving all pipelined requests. This will be fixed in an
> upcoming release.
>
> Regards Ingela Erlang/OTP-team,  Ericsson AB
>
> 2010/10/21amitm<amit.mur...@REDACTED>:
>
>
>
>
>
>
>
> >> Humm ... what settings did you have when you had the problem?
>
> > The function making the call in the situation when the problem
> > manifests itself is:
>
> > mkGetReq2(GetUrl) ->
> >    GetRequest = {GetUrl, []},
> >    GetHttpOptions = [{autoredirect, true}],
> >    Options = [ {sync,true}, {headers_as_is,true}, {body_format,
> > binary} ],
> >    httpc:request(get, GetRequest, GetHttpOptions, Options).
>
> > No other options were set, which means they run with their default
> > values.
>
> >> Did you set the pipeline timout ?
>
> > No pipeline timeout was set.
>
> >> What webserver did you connect to (apache, yaws, ...)?
>
> > The webserver is one used by Amazon internally on EC2. Curl in verbose
> > mode, as executed by the command
> > "curl -vhttp://169.254.169.254/2009-04-04/meta-data/instance-id"
> > prints out
>
> >  * About to connect() to 169.254.169.254 port 80 (#0)
> >  *   Trying 169.254.169.254... connected
> >  * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> >  > GET /2009-04-04/meta-data/instance-id HTTP/1.1
> >  > User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/
> > 0.9.8k zlib/1.2.3.3 libidn/1.15
> >  > Host: 169.254.169.254
> >  > Accept: */*
> >  >
> >  * HTTP 1.0, assume close after body
> >  < HTTP/1.0 200 OK
> >  < Content-Type: text/plain
> >  < Accept-Ranges: bytes
> >  < ETag: "-1124024050"
> >  < Last-Modified: Tue, 19 Oct 2010 13:58:54 GMT
> >  < Content-Length: 10
> >  < Connection: close
> >  < Date: Thu, 21 Oct 2010 13:48:19 GMT
> >  < Server: EC2ws
> >  <
> >  * Closing connection #0
>
> >> What version of inets?
>
> > erl -v prints
> > "Erlang R14B (erts-5.8.1) [source] [rq:1] [async-threads:0] [kernel-
> > poll:false]"
>
> > Hope it helps,
>
> >  Amit
>
> > On Oct 21, 6:32 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
> >> Hi!
>
> >> Humm ... what settings did you have when you had the problem?
> >> Did you set the pipeline timout ? What webserver did you connect to
> >> (apache, yaws, ...)? What
> >> version of inets? If there is some bug in the pipeling or
> >> persistenconnection code
> >> of course we would like to know so that we can fix it.
>
> >> Regards Ingela Erlang/OTP team - Ericsson AB
>
> >> 2010/10/20amitm<amit.mur...@REDACTED>:
>
> >> > FWIW, I had a similar problem. It went away when I disabled pipelining
> >> > completely by executing
>
> >> >    ok = httpc:set_options([{max_keep_alive_length, 0},
> >> > {max_pipeline_length, 0}, {max_sessions, 0} ]),
>
> >> > before my http calls. I used to usually see it on the 4th consecutive
> >> > call to the same host when pipelining was enabled. The URL in question
> >> > in my case was an Amazon EC2 url.
>
> >> > The set of URLs in my case were
>
> >> > "http://169.254.169.254/2009-04-04/meta-data/instance-id",
> >> > "http://169.254.169.254/2009-04-04/meta-data/local-hostname",
> >> > "http://169.254.169.254/2009-04-04/meta-data/local-ipv4",
> >> > "http://169.254.169.254/2009-04-04/meta-data/public-hostname",
> >> > "http://169.254.169.254/2009-04-04/meta-data/public-ipv4",
> >> > "http://169.254.169.254/2009-04-04/meta-data/ami-id",
>
> >> > While executing this set, with pipelining enabled, it used to always
> >> > bomb on the 4th URL, i.e. for 'public-hostname' in 25% of the cases
>
> >> > with pipelining disabled, the problem disappears.
>
> >> >  Amit
>
> >> > On Oct 13, 2:58 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
> >> >> 2010/10/13 Chandru <chandrashekhar.mullapar...@REDACTED>:
>
> >> >> > On 13 October 2010 09:53, Ingela Andin <ingela.an...@REDACTED> wrote:
> >> >> >> Hi!
>
> >> >> >> The only reason that the httpc-client will return the value {error,
> >> >> >> socket_closed_remotely}
> >> >> >> is that the connection is prematurely  closed by the server.
> >> >> >> Unfortunately  that something
> >> >> >> is sent on a tcp socket does not mean it has been sent on the wire
> >> >> >> yet, and depending on
> >> >> >> the circumstances it is possible for the socket to be closed before
> >> >> >> all data has been sent
> >> >> >> so occasionally a tcp_close can come before all data is delivered.
>
> >> >> >> When you use ibrowse do you verify that you always get a correct body?
> >> >> >> (compleate body) from what I can tell from the code ibrowse will on a
> >> >> >> tcp_closed  message always return whatever happens to be in the buffer
> >> >> >> compleate or not.
>
> >> >> > ibrowse does this only in the following cases:
> >> >> >  * server returned a "Connection: Close"
> >> >> >  * Server supports HTTP/0.9 or HTTP/1.0
>
> >> >> > In all other cases, it returns an error to the caller.
>
> >> >> > regards,
> >> >> > Chandru
>
> >> >> Well I do not know if any of these things is true in Dans senario. But
> >> >> I do know that
> >> >> getting the problem with a tcp close message before all data is
> >> >> deliverd is a highly timing dependant problem
> >> >> and if it happens in the server there is nothing inets-client or
> >> >> ibrows can do about it except return an error.
>
> >> >> Regards Ingela Erlang/OTP team - Ericsson AB
>
> >> >> ________________________________________________________________
> >> >> erlang-questions (at) erlang.org mailing list.
> >> >> Seehttp://www.erlang.org/faq.html
> >> >> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>
> >> > ________________________________________________________________
> >> > erlang-questions (at) erlang.org mailing list.
> >> > Seehttp://www.erlang.org/faq.html
> >> > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>
> >> ________________________________________________________________
> >> erlang-questions (at) erlang.org mailing list.
> >> Seehttp://www.erlang.org/faq.html
> >> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>
> > ________________________________________________________________
> > erlang-questions (at) erlang.org mailing list.
> > Seehttp://www.erlang.org/faq.html
> > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> Seehttp://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED