[erlang-questions] Re: confusing returns from httpc:request/4

Thu Oct 28 11:51:51 CEST 2010

Hi!

Thank you for pointing that out, and yes the piepline code and the
persistent connection code has a lot of common code
so it is quite natural that the bug affects them both.

Regards Ingela Erlang/OTP team - Ericsson AB

2010/10/27 amitm <amit.murthy@REDACTED>:
> Sorry, I missed out on stating that to make the problem go away, I
> also set the HTTP version as "HTTP/1.0"
>
> So you may also want to look at the code for handling persistent
> connections in general, especially when the server only supports HTTP
> 1.0 .
>
>
>  Amit
>
>
> On Oct 21, 8:54 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
>> Hi!
>>
>> Yes thank you for the information. I think I have located the bug. It seems
>> that to avoid crash reports the return value when receiving a tcp_closed
>> was changed and this altered the terminate clause that is run,  alas breaking
>> the automatic retries that should be done if the server terminates a connection
>> before serving all pipelined requests. This will be fixed in an
>> upcoming release.
>>
>> Regards Ingela Erlang/OTP-team,  Ericsson AB
>>
>> 2010/10/21amitm<amit.mur...@REDACTED>:
>>
>>
>>
>>
>>
>>
>>
>> >> Humm ... what settings did you have when you had the problem?
>>
>> > The function making the call in the situation when the problem
>> > manifests itself is:
>>
>> > mkGetReq2(GetUrl) ->
>> >    GetRequest = {GetUrl, []},
>> >    GetHttpOptions = [{autoredirect, true}],
>> >    Options = [ {sync,true}, {headers_as_is,true}, {body_format,
>> > binary} ],
>> >    httpc:request(get, GetRequest, GetHttpOptions, Options).
>>
>> > No other options were set, which means they run with their default
>> > values.
>>
>> >> Did you set the pipeline timout ?
>>
>> > No pipeline timeout was set.
>>
>> >> What webserver did you connect to (apache, yaws, ...)?
>>
>> > The webserver is one used by Amazon internally on EC2. Curl in verbose
>> > mode, as executed by the command
>> > "curl -vhttp://169.254.169.254/2009-04-04/meta-data/instance-id"
>> > prints out
>>
>> >  * About to connect() to 169.254.169.254 port 80 (#0)
>> >  *   Trying 169.254.169.254... connected
>> >  * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
>> >  > GET /2009-04-04/meta-data/instance-id HTTP/1.1
>> >  > User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/
>> > 0.9.8k zlib/1.2.3.3 libidn/1.15
>> >  > Host: 169.254.169.254
>> >  > Accept: */*
>> >  >
>> >  * HTTP 1.0, assume close after body
>> >  < HTTP/1.0 200 OK
>> >  < Content-Type: text/plain
>> >  < Accept-Ranges: bytes
>> >  < ETag: "-1124024050"
>> >  < Last-Modified: Tue, 19 Oct 2010 13:58:54 GMT
>> >  < Content-Length: 10
>> >  < Connection: close
>> >  < Date: Thu, 21 Oct 2010 13:48:19 GMT
>> >  < Server: EC2ws
>> >  <
>> >  * Closing connection #0
>>
>> >> What version of inets?
>>
>> > erl -v prints
>> > "Erlang R14B (erts-5.8.1) [source] [rq:1] [async-threads:0] [kernel-
>> > poll:false]"
>>
>> > Hope it helps,
>>
>> >  Amit
>>
>> > On Oct 21, 6:32 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
>> >> Hi!
>>
>> >> Humm ... what settings did you have when you had the problem?
>> >> Did you set the pipeline timout ? What webserver did you connect to
>> >> (apache, yaws, ...)? What
>> >> version of inets? If there is some bug in the pipeling or
>> >> persistenconnection code
>> >> of course we would like to know so that we can fix it.
>>
>> >> Regards Ingela Erlang/OTP team - Ericsson AB
>>
>> >> 2010/10/20amitm<amit.mur...@REDACTED>:
>>
>> >> > FWIW, I had a similar problem. It went away when I disabled pipelining
>> >> > completely by executing
>>
>> >> >    ok = httpc:set_options([{max_keep_alive_length, 0},
>> >> > {max_pipeline_length, 0}, {max_sessions, 0} ]),
>>
>> >> > before my http calls. I used to usually see it on the 4th consecutive
>> >> > call to the same host when pipelining was enabled. The URL in question
>> >> > in my case was an Amazon EC2 url.
>>
>> >> > The set of URLs in my case were
>>
>> >> > "http://169.254.169.254/2009-04-04/meta-data/instance-id",
>> >> > "http://169.254.169.254/2009-04-04/meta-data/local-hostname",
>> >> > "http://169.254.169.254/2009-04-04/meta-data/local-ipv4",
>> >> > "http://169.254.169.254/2009-04-04/meta-data/public-hostname",
>> >> > "http://169.254.169.254/2009-04-04/meta-data/public-ipv4",
>> >> > "http://169.254.169.254/2009-04-04/meta-data/ami-id",
>>
>> >> > While executing this set, with pipelining enabled, it used to always
>> >> > bomb on the 4th URL, i.e. for 'public-hostname' in 25% of the cases
>>
>> >> > with pipelining disabled, the problem disappears.
>>
>> >> >  Amit
>>
>> >> > On Oct 13, 2:58 pm, Ingela Andin <ingela.an...@REDACTED> wrote:
>> >> >> 2010/10/13 Chandru <chandrashekhar.mullapar...@REDACTED>:
>>
>> >> >> > On 13 October 2010 09:53, Ingela Andin <ingela.an...@REDACTED> wrote:
>> >> >> >> Hi!
>>
>> >> >> >> The only reason that the httpc-client will return the value {error,
>> >> >> >> socket_closed_remotely}
>> >> >> >> is that the connection is prematurely  closed by the server.
>> >> >> >> Unfortunately  that something
>> >> >> >> is sent on a tcp socket does not mean it has been sent on the wire
>> >> >> >> yet, and depending on
>> >> >> >> the circumstances it is possible for the socket to be closed before
>> >> >> >> all data has been sent
>> >> >> >> so occasionally a tcp_close can come before all data is delivered.
>>
>> >> >> >> When you use ibrowse do you verify that you always get a correct body?
>> >> >> >> (compleate body) from what I can tell from the code ibrowse will on a
>> >> >> >> tcp_closed  message always return whatever happens to be in the buffer
>> >> >> >> compleate or not.
>>
>> >> >> > ibrowse does this only in the following cases:
>> >> >> >  * server returned a "Connection: Close"
>> >> >> >  * Server supports HTTP/0.9 or HTTP/1.0
>>
>> >> >> > In all other cases, it returns an error to the caller.
>>
>> >> >> > regards,
>> >> >> > Chandru
>>
>> >> >> Well I do not know if any of these things is true in Dans senario. But
>> >> >> I do know that
>> >> >> getting the problem with a tcp close message before all data is
>> >> >> deliverd is a highly timing dependant problem
>> >> >> and if it happens in the server there is nothing inets-client or
>> >> >> ibrows can do about it except return an error.
>>
>> >> >> Regards Ingela Erlang/OTP team - Ericsson AB
>>
>> >> >> ________________________________________________________________
>> >> >> erlang-questions (at) erlang.org mailing list.
>> >> >> Seehttp://www.erlang.org/faq.html
>> >> >> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>>
>> >> > ________________________________________________________________
>> >> > erlang-questions (at) erlang.org mailing list.
>> >> > Seehttp://www.erlang.org/faq.html
>> >> > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>>
>> >> ________________________________________________________________
>> >> erlang-questions (at) erlang.org mailing list.
>> >> Seehttp://www.erlang.org/faq.html
>> >> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>>
>> > ________________________________________________________________
>> > erlang-questions (at) erlang.org mailing list.
>> > Seehttp://www.erlang.org/faq.html
>> > To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> Seehttp://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscr...@REDACTED
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>