[erlang-questions] httpc considered harmful

[Felix Gallo]

I recently ran into a very scary issue that appears to be related to httpc.

I was hitting a web API millions of times, with varying URLs; e.g.,
/users/9000000, /users/9000001, etc., at a rate of around 100-400
requests/sec, using httpc:request, each request spawned by a different
worker:

get_user(UserID) ->
  get_user_r(UserID, 10).
get_user_r(UserID, 0) ->
  io:format("dying because ran out of retries on ~p~n",[UserID]);
get_user_r(UserID, Retries) ->
  Url = lists:concat(["http://example.com/users/", UserID]),
  Filename = lists:concat(["users/-", UserID, ".json"]),
  io:format("requesting user: ~p~n", [UserID]),
  case httpc:request(Url) of
    {ok, Result} ->
      {_, _, Body} = Result,
      file:write_file(Filename, Body),
      userscrapemaster ! {ok, ClanID};
    {error, Reason} ->
      io:format("error for user ~p: ~p~n",[UserID, Reason]),
      get_members_r(UserID, Retries - 1)

A small (< 0.1%) but significant percentage of the time, the httpc:request
call for completely different workers MIXED UP THEIR RESPONSES with other
concurrent requests.

For example, sometimes /users/5000 returned success but provided the body
that /users/5001 should have returned, and /users/5001 returned the body
that /users/5002 should have returned, and /users/5002 returned the body
that /users/5000 should have returned.  Or, /users/5009 returned the
response for /users/5010, and vice versa.

There appeared to be no obvious pattern except that all those calls were
concurrent, and pragmatically I didn't have the time to go chasing into
httpc to try to figure out where the state was getting scrambled, but as a
test I moved the call over to lhttpc without changing the structure of the
code otherwise, and the mixed responses went away.

If I get some time I'll try to dig into httpc to understand what happened
there, but as a warning to others: httpc looks like it has a hidden race
condition or other bug, and lhttpc does not.

F.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140213/e961177f/attachment.htm>