[erlang-questions] better HTTP client?
Mon Sep 9 14:14:36 CEST 2013
You might be interested in my fork of lhttpc:
It's pretty much regular lhttpc, but I added sharded connection pools to
it. It has been especially optimized for doing a lot of requests to a
limited set of endpoints, and was developed during my stay at AdGear for
real time bidding purposes. It's also used at other real time bidding
It will not do request cancelling the way you mention it. It has a
central load-balancer per endpoint, where you ask for sockets. It's easy
to cancel a task once you have the connection yourself, but not during
the time it's in the queue to the manager.
If you kill the process that made the request, the manager will find out
once it tries to give away the process and keep going until it hits a
request that works
That might be suboptimal behavior in your case.
I also have a fork of that very load balancer done using dispcount
called dlhttpc: https://github.com/ferd/dlhttpc
Dispcount is a low-latency pool (https://github.com/ferd/dispcount) that
basically will tell you right away if you can have a resource or not.
There will be no such thing as a blocking request using dlhttpc because
the pooling model doesn't allow for an explicit request queue. It
instead uses a pool of workers that may or may not be connected (in the
case of dlhttpc, each worker will only connect once asked for it, but
processes will be started already), and assumes you will retry until you
hit one you need. It's been designed with cases of constant overload in
mind, where you may arbitrarily decide to drop a request from the
Depending on how flexible your requirements are, I'd try at least one of
these libraries to see how they fare.
lhttpc has been in production for a long while now and has been fairly
stable as far as I can tell, everywhere it's been deployed. dlhttpc has
seen no real production use, but dispcount has, so it's not too likely
to be a terrible thing to deploy.
On 09/08, Chris King wrote:
> On Sun, 08 Sep 2013 19:33:05 -0400, Bob Ippolito <bob@REDACTED> wrote:
> >Async cancelable and synchronous are basically isomorphic in
> >Erlang, just start a process to wrap the sync call and kill >it if
> >you want to cancel.
> Not quite -- the HTTP request remains outstanding and blocks the
> request queue. I need to clear those requests out, as I need to
> process new requests ASAP. This should be handled by the HTTP
> client itself, as only it knows how many requests it has actually
> sent to the server. (The alternative would be for the HTTP client
> to expose a crediting mechanism; though this would complicate the
> erlang-questions mailing list
More information about the erlang-questions