[erlang-questions] Re: confusing returns from httpc:request/4

[Ingela Andin]

Hi!

Yes thank you for the information. I think I have located the bug. It seems
that to avoid crash reports the return value when receiving a tcp_closed
was changed and this altered the terminate clause that is run,  alas breaking
the automatic retries that should be done if the server terminates a connection
before serving all pipelined requests. This will be fixed in an
upcoming release.

Regards Ingela Erlang/OTP-team,  Ericsson AB

2010/10/21 amitm <amit.murthy@REDACTED>:
>
>> Humm ... what settings did you have when you had the problem?
>
> The function making the call in the situation when the problem
> manifests itself is:
>
> mkGetReq2(GetUrl) ->
>    GetRequest = {GetUrl, []},
>    GetHttpOptions = [{autoredirect, true}],
>    Options = [ {sync,true}, {headers_as_is,true}, {body_format,
> binary} ],
>    httpc:request(get, GetRequest, GetHttpOptions, Options).
>
> No other options were set, which means they run with their default
> values.
>
>> Did you set the pipeline timout ?
>
> No pipeline timeout was set.
>
>> What webserver did you connect to (apache, yaws, ...)?
>
> The webserver is one used by Amazon internally on EC2. Curl in verbose
> mode, as executed by the command
> "curl -v http://169.254.169.254/2009-04-04/meta-data/instance-id "
> prints out
>
>  * About to connect() to 169.254.169.254 port 80 (#0)
>  *   Trying 169.254.169.254... connected
>  * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
>  > GET /2009-04-04/meta-data/instance-id HTTP/1.1
>  > User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/
> 0.9.8k zlib/1.2.3.3 libidn/1.15
>  > Host: 169.254.169.254
>  > Accept: */*
>  >
>  * HTTP 1.0, assume close after body
>  < HTTP/1.0 200 OK
>  < Content-Type: text/plain
>  < Accept-Ranges: bytes
>  < ETag: "-1124024050"
>  < Last-Modified: Tue, 19 Oct 2010 13:58:54 GMT
>  < Content-Length: 10
>  < Connection: close
>  < Date: Thu, 21 Oct 2010 13:48:19 GMT
>  < Server: EC2ws
>  <
>  * Closing connection #0
>
>> What version of inets?
>
> erl -v prints
> "Erlang R14B (erts-5.8.1) [source] [rq:1] [async-threads:0] [kernel-
> poll:false]"
>
> Hope it helps,
>
>  Amit
>
>
> On Oct 21, 6:32 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
>> Hi!
>>
>> Humm ... what settings did you have when you had the problem?
>> Did you set the pipeline timout ? What webserver did you connect to
>> (apache, yaws, ...)? What
>> version of inets? If there is some bug in the pipeling or
>> persistenconnection code
>> of course we would like to know so that we can fix it.
>>
>> Regards Ingela Erlang/OTP team - Ericsson AB
>>
>> 2010/10/20 amitm <amit.mur...@REDACTED>:
>>
>>
>>
>>
>>
>> > FWIW, I had a similar problem. It went away when I disabled pipelining
>> > completely by executing
>>
>> >    ok = httpc:set_options([{max_keep_alive_length, 0},
>> > {max_pipeline_length, 0}, {max_sessions, 0} ]),
>>
>> > before my http calls. I used to usually see it on the 4th consecutive
>> > call to the same host when pipelining was enabled. The URL in question
>> > in my case was an Amazon EC2 url.
>>
>> > The set of URLs in my case were
>>
>> > "http://169.254.169.254/2009-04-04/meta-data/instance-id",
>> > "http://169.254.169.254/2009-04-04/meta-data/local-hostname",
>> > "http://169.254.169.254/2009-04-04/meta-data/local-ipv4",
>> > "http://169.254.169.254/2009-04-04/meta-data/public-hostname",
>> > "http://169.254.169.254/2009-04-04/meta-data/public-ipv4",
>> > "http://169.254.169.254/2009-04-04/meta-data/ami-id",
>>
>> > While executing this set, with pipelining enabled, it used to always
>> > bomb on the 4th URL, i.e. for 'public-hostname' in 25% of the cases
>>
>> > with pipelining disabled, the problem disappears.
>>
>> >  Amit
>>
>> > On Oct 13, 2:58 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
>> >> 2010/10/13 Chandru <chandrashekhar.mullapar...@REDACTED>:
>>
>> >> > On 13 October 2010 09:53, Ingela Andin <ingela.an...@REDACTED> wrote:
>> >> >> Hi!
>>
>> >> >> The only reason that the httpc-client will return the value {error,
>> >> >> socket_closed_remotely}
>> >> >> is that the connection is prematurely  closed by the server.
>> >> >> Unfortunately  that something
>> >> >> is sent on a tcp socket does not mean it has been sent on the wire
>> >> >> yet, and depending on
>> >> >> the circumstances it is possible for the socket to be closed before
>> >> >> all data has been sent
>> >> >> so occasionally a tcp_close can come before all data is delivered.
>>
>> >> >> When you use ibrowse do you verify that you always get a correct body?
>> >> >> (compleate body) from what I can tell from the code ibrowse will on a
>> >> >> tcp_closed  message always return whatever happens to be in the buffer
>> >> >> compleate or not.
>>
>> >> > ibrowse does this only in the following cases:
>> >> >  * server returned a "Connection: Close"
>> >> >  * Server supports HTTP/0.9 or HTTP/1.0
>>
>> >> > In all other cases, it returns an error to the caller.
>>
>> >> > regards,
>> >> > Chandru
>>
>> >> Well I do not know if any of these things is true in Dans senario. But
>> >> I do know that
>> >> getting the problem with a tcp close message before all data is
>> >> deliverd is a highly timing dependant problem
>> >> and if it happens in the server there is nothing inets-client or
>> >> ibrows can do about it except return an error.
>>
>> >> Regards Ingela Erlang/OTP team - Ericsson AB
>>
>> >> ________________________________________________________________
>> >> erlang-questions (at) erlang.org mailing list.
>> >> Seehttp://www.erlang.org/faq.html
>> >> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>>
>> > ________________________________________________________________
>> > erlang-questions (at) erlang.org mailing list.
>> > Seehttp://www.erlang.org/faq.html
>> > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> Seehttp://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>