[erlang-bugs] httpc / http 1.1. closes socket

[jim rosenblum]

In http://erlang.org/pipermail/erlang-questions/2014-February/077788.html
and elsewhere, users observed that httpc would occasionally encounter
`{error,socket_closed_remotely}` when querying a web service of some sort.
The workaround was to use http 1.0, as opposed to 1.1, which is somewhat
of an unsatisfying resolution.

I have encountered this same problem and have tried to create a minimal
example that reproduces the issue using a public url to post a simple JSON
body. I am hoping that the following provides enough information to
constitute a useful bug report.

The following code will quickly cause the problem to appear; however,
the problem *disappears* when  {"connection", "close"} is added to
the header. In fact I cannot reproduce the problem ever when this
header is used.

My code is as follows:

-module(httpc_bug).
-export([test/0]).

test()->
    httpc:set_options([{max_keep_alive_length, 0},
                                 {max_pipeline_length, 0},
                                 {max_sessions, 0}]),
    test_put(0).

test_put(N) ->
    Url = "https://posttestserver.com/post.php",
    ContentType= "application/json",
    Body = <<"{\"type\":\"body type\",\"value\":2}">>,
    Header = [],
    HttpOptions = [],
    Options = [],

    case httpc:request(post, {Url, Header, ContentType, Body},
                                  HttpOptions,
                                  Options) of
        {ok, {{_V, _Cd, _R}, _RespHeader, B }} ->
                io:format("~p[~p]:~p~n”, [self(), N, B]),
                test_put(N+1);
        {error, Reason} ->
                io:format("~p: died after ~p iterations with: ~p. ~n.",
                               [self(), N, Reason])
        end.


A typical run looks like this:

Erlang/OTP 17 [erts-6.2] [source-aaaefb3] [64-bit] [smp:8:8]
[async-threads:10] [hipe] [kernel-poll:false]


Eshell V6.2  (abort with ^G)
1> inets:start().
ok
2> ssl:start().
ok
3> code:load_file(httpc_bug).
{module,httpc_bug}
4> httpc_bug:test(1).
[<0.15201.0>]
<0.15201.0>[0]:"Successfully dumped 0 post variables.\nView it at
http://www.posttestserver.com/data/2015/03/13/19.21.45157935823\nPost body
was 30 chars long."
<0.15201.0>[1]:"Successfully dumped 0 post variables.\nView it at
http://www.posttestserver.com/data/2015/03/13/19.21.451784952425\nPost body
was 30 chars long."
.
. SNIP
.
<0.15201.0>: died after 102 iterations with: socket_closed_remotely.


I just did five runs, four died in under 410 iterations, and the fifth took
809 iterations.


Again, adding {"connection", "close"} to the header causes the problem to
disappear.

By examining the contents of the http_manager__session_db before each call
to the httpc:request, I am pretty sure that the flow goes:


httpc_manager.erl line 759 — within a handle_request clause

handle_request(Request, State = #state{options = Options}) ->
    NewRequest = handle_cookies(generate_request_id(Request), State),
    SessionType = session_type(Options),
    case select_session(Request#request.method,
                        Request#request.address,
                        Request#request.scheme, SessionType, State) of
    {ok, HandlerPid} ->
        -> pipeline_or_keep_alive(NewRequest, HandlerPid, State); <-
        .
        . SNIP
        .

To

httpc_handler.erl line 323 — within a handle_call where keep_alive requests
are handled.

Hope this is useful.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-bugs/attachments/20150327/86aaaa05/attachment.htm>