[erlang-questions] Erlang HTTP client libraries- pros/cons

[Martin Karlsson]

I have done some benchmarking for our specific use case.

* Main web server with high number of concurrent requests coming in
  (~1000 concurrent requests per second but benchmarked up to 10,000
  requests)
* Most requests requires a call out to an back-end HTTP server (only
  GET)
* There are few back-end endpoints (3).

It is not clear how many different kind of servers you are going to call
out to and this will likely affect your choice of library. 

For our use case  I benchmarked httpc, gun, hackney, dlhttpc, ibrowse.

All of them behaves well when the endpoint replies quickly. However,
once the latency of the response goes up we have problems with all "traditional
clients". The only one that worked for us in this use case was dlhttpc
(https://github.com/ferd/dlhttpc) and this is likely because of the
dispcount "router". 

I know hackney has experimental dispcount pooling so that would likely
work as well.

* httpc is known to have a few quirks. For our use case which only do HTTP
  get requests, no cookies, pipelines or other it works well but there is
  a gen_server bottleneck in there somewhere when calling only few
  endpoints (have not tried with many endpoints). The biggest pro is that
  it is built-in, but behaves a bit strange under load.

* gun - was discarded quickly as I couldn't get anywhere near the
  performance I was after and I didn't have the time to trouble-shoot
  with so many other alternatives.

* hackney and ibrowse - seemed like the most stable clients in term of
  response times. hackney is also used lots in the elixir community so
  it should have some good production usage. ibrowse has been around
  for a long time.

* dlhttpc - works best for us. Not updated in a while so can't say
  anything for more advanced usage. Again, this is becuase of dispcount
  routing, not the actual http client in it self.


Cheers
Martin