confusing returns from httpc:request/4

[amitm]

> Humm ... what settings did you have when you had the problem?

The function making the call in the situation when the problem
manifests itself is:

mkGetReq2(GetUrl) ->
    GetRequest = {GetUrl, []},
    GetHttpOptions = [{autoredirect, true}],
    Options = [ {sync,true}, {headers_as_is,true}, {body_format,
binary} ],
    httpc:request(get, GetRequest, GetHttpOptions, Options).

No other options were set, which means they run with their default
values.

> Did you set the pipeline timout ?

No pipeline timeout was set.

> What webserver did you connect to (apache, yaws, ...)?

The webserver is one used by Amazon internally on EC2. Curl in verbose
mode, as executed by the command
"curl -v http://169.254.169.254/2009-04-04/meta-data/instance-id "
prints out

  * About to connect() to 169.254.169.254 port 80 (#0)
  *   Trying 169.254.169.254... connected
  * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
  > GET /2009-04-04/meta-data/instance-id HTTP/1.1
  > User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/
0.9.8k zlib/1.2.3.3 libidn/1.15
  > Host: 169.254.169.254
  > Accept: */*
  >
  * HTTP 1.0, assume close after body
  < HTTP/1.0 200 OK
  < Content-Type: text/plain
  < Accept-Ranges: bytes
  < ETag: "-1124024050"
  < Last-Modified: Tue, 19 Oct 2010 13:58:54 GMT
  < Content-Length: 10
  < Connection: close
  < Date: Thu, 21 Oct 2010 13:48:19 GMT
  < Server: EC2ws
  <
  * Closing connection #0

> What version of inets?

erl -v prints
"Erlang R14B (erts-5.8.1) [source] [rq:1] [async-threads:0] [kernel-
poll:false]"

Hope it helps,

  Amit


On Oct 21, 6:32 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
> Hi!
>
> Humm ... what settings did you have when you had the problem?
> Did you set the pipeline timout ? What webserver did you connect to
> (apache, yaws, ...)? What
> version of inets? If there is some bug in the pipeling or
> persistenconnection code
> of course we would like to know so that we can fix it.
>
> Regards Ingela Erlang/OTP team - Ericsson AB
>
> 2010/10/20 amitm <amit.mur...@REDACTED>:
>
>
>
>
>
> > FWIW, I had a similar problem. It went away when I disabled pipelining
> > completely by executing
>
> >    ok = httpc:set_options([{max_keep_alive_length, 0},
> > {max_pipeline_length, 0}, {max_sessions, 0} ]),
>
> > before my http calls. I used to usually see it on the 4th consecutive
> > call to the same host when pipelining was enabled. The URL in question
> > in my case was an Amazon EC2 url.
>
> > The set of URLs in my case were
>
> > "http://169.254.169.254/2009-04-04/meta-data/instance-id",
> > "http://169.254.169.254/2009-04-04/meta-data/local-hostname",
> > "http://169.254.169.254/2009-04-04/meta-data/local-ipv4",
> > "http://169.254.169.254/2009-04-04/meta-data/public-hostname",
> > "http://169.254.169.254/2009-04-04/meta-data/public-ipv4",
> > "http://169.254.169.254/2009-04-04/meta-data/ami-id",
>
> > While executing this set, with pipelining enabled, it used to always
> > bomb on the 4th URL, i.e. for 'public-hostname' in 25% of the cases
>
> > with pipelining disabled, the problem disappears.
>
> >  Amit
>
> > On Oct 13, 2:58 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
> >> 2010/10/13 Chandru <chandrashekhar.mullapar...@REDACTED>:
>
> >> > On 13 October 2010 09:53, Ingela Andin <ingela.an...@REDACTED> wrote:
> >> >> Hi!
>
> >> >> The only reason that the httpc-client will return the value {error,
> >> >> socket_closed_remotely}
> >> >> is that the connection is prematurely  closed by the server.
> >> >> Unfortunately  that something
> >> >> is sent on a tcp socket does not mean it has been sent on the wire
> >> >> yet, and depending on
> >> >> the circumstances it is possible for the socket to be closed before
> >> >> all data has been sent
> >> >> so occasionally a tcp_close can come before all data is delivered.
>
> >> >> When you use ibrowse do you verify that you always get a correct body?
> >> >> (compleate body) from what I can tell from the code ibrowse will on a
> >> >> tcp_closed  message always return whatever happens to be in the buffer
> >> >> compleate or not.
>
> >> > ibrowse does this only in the following cases:
> >> >  * server returned a "Connection: Close"
> >> >  * Server supports HTTP/0.9 or HTTP/1.0
>
> >> > In all other cases, it returns an error to the caller.
>
> >> > regards,
> >> > Chandru
>
> >> Well I do not know if any of these things is true in Dans senario. But
> >> I do know that
> >> getting the problem with a tcp close message before all data is
> >> deliverd is a highly timing dependant problem
> >> and if it happens in the server there is nothing inets-client or
> >> ibrows can do about it except return an error.
>
> >> Regards Ingela Erlang/OTP team - Ericsson AB
>
> >> ________________________________________________________________
> >> erlang-questions (at) erlang.org mailing list.
> >> Seehttp://www.erlang.org/faq.html
> >> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>
> > ________________________________________________________________
> > erlang-questions (at) erlang.org mailing list.
> > Seehttp://www.erlang.org/faq.html
> > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> Seehttp://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED