[erlang-questions] httpc considered harmful

[rytong-caoxu]

I’ve reported the bug a few years ago:

http://erlang.org/pipermail/erlang-bugs/2011-November/002647.html

And the reason was explained in

http://stackoverflow.com/questions/14473416/persistent-keepalive-http-client-implementation-of-erlang

With pipeline(or earlier keepalive with problem), httpc client selects the response according to the requests’  sending order, which may cause MIXED UP responses if the server fails to response some requests in the sending order.

It seems that the bug has already been fixed, but not confirmed by myself. You can use HTTP/0.9 instead, or set the max keepalive queue length to be 1 to work around.


在 2014年2月14日，上午3:39，Felix Gallo <felixgallo@REDACTED> 写道：

> I recently ran into a very scary issue that appears to be related to httpc.
> 
> I was hitting a web API millions of times, with varying URLs; e.g., /users/9000000, /users/9000001, etc., at a rate of around 100-400 requests/sec, using httpc:request, each request spawned by a different worker:
> 
> get_user(UserID) ->
>   get_user_r(UserID, 10).
> get_user_r(UserID, 0) ->
>   io:format("dying because ran out of retries on ~p~n",[UserID]);
> get_user_r(UserID, Retries) ->
>   Url = lists:concat(["http://example.com/users/", UserID]),
>   Filename = lists:concat(["users/-", UserID, ".json"]),
>   io:format("requesting user: ~p~n", [UserID]),
>   case httpc:request(Url) of
>     {ok, Result} ->
>       {_, _, Body} = Result,
>       file:write_file(Filename, Body),
>       userscrapemaster ! {ok, ClanID};
>     {error, Reason} ->
>       io:format("error for user ~p: ~p~n",[UserID, Reason]),
>       get_members_r(UserID, Retries - 1)
> 
> A small (< 0.1%) but significant percentage of the time, the httpc:request call for completely different workers MIXED UP THEIR RESPONSES with other concurrent requests.
> 
> For example, sometimes /users/5000 returned success but provided the body that /users/5001 should have returned, and /users/5001 returned the body that /users/5002 should have returned, and /users/5002 returned the body that /users/5000 should have returned.  Or, /users/5009 returned the response for /users/5010, and vice versa.
> 
> There appeared to be no obvious pattern except that all those calls were concurrent, and pragmatically I didn't have the time to go chasing into httpc to try to figure out where the state was getting scrambled, but as a test I moved the call over to lhttpc without changing the structure of the code otherwise, and the mixed responses went away.
> 
> If I get some time I'll try to dig into httpc to understand what happened there, but as a warning to others: httpc looks like it has a hidden race condition or other bug, and lhttpc does not.
> 
> F.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140217/7b8750a4/attachment.htm>