[erlang-questions] better HTTP client?

[Fred Hebert]

You might be interested in my fork of lhttpc:
https://github.com/ferd/lhttpc

It's pretty much regular lhttpc, but I added sharded connection pools to
it. It has been especially optimized for doing a lot of requests to a
limited set of endpoints, and was developed during my stay at AdGear for
real time bidding purposes. It's also used at other real time bidding
shops now.

It will not do request cancelling the way you mention it. It has a
central load-balancer per endpoint, where you ask for sockets. It's easy
to cancel a task once you have the connection yourself, but not during
the time it's in the queue to the manager.

If you kill the process that made the request, the manager will find out
once it tries to give away the process and keep going until it hits a
request that works
(https://github.com/ferd/lhttpc/blob/master/src/lhttpc_lb.erl#L98)

That might be suboptimal behavior in your case.

I also have a fork of that very load balancer done using dispcount
called dlhttpc: https://github.com/ferd/dlhttpc

Dispcount is a low-latency pool (https://github.com/ferd/dispcount) that
basically will tell you right away if you can have a resource or not.
There will be no such thing as a blocking request using dlhttpc because
the pooling model doesn't allow for an explicit request queue. It
instead uses a pool of workers that may or may not be connected (in the
case of dlhttpc, each worker will only connect once asked for it, but
processes will be started already), and assumes you will retry until you
hit one you need. It's been designed with cases of constant overload in
mind, where you may arbitrarily decide to drop a request from the
call-site.

Depending on how flexible your requirements are, I'd try at least one of
these libraries to see how they fare.

lhttpc has been in production for a long while now and has been fairly
stable as far as I can tell, everywhere it's been deployed. dlhttpc has
seen no real production use, but dispcount has, so it's not too likely
to be a terrible thing to deploy.

Regards,
Fred.

On 09/08, Chris King wrote:
> On Sun, 08 Sep 2013 19:33:05 -0400, Bob Ippolito <bob@REDACTED> wrote:
> 
> >Async cancelable and synchronous are basically isomorphic in
> >Erlang, just start a process to wrap the sync call and kill >it if
> >you want to cancel.
> 
> Not quite -- the HTTP request remains outstanding and blocks the
> request queue.  I need to clear those requests out, as I need to
> process new requests ASAP.  This should be handled by the HTTP
> client itself, as only it knows how many requests it has actually
> sent to the server.  (The alternative would be for the HTTP client
> to expose a crediting mechanism; though this would complicate the
> application.)

> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions