[erlang-questions] better HTTP client?

[Seth Falcon]

We've taken a similar approach, inspired by Fred's fork of lhttpc, but
built on top of ibrowse. For us as well, the use case is handling many
connections to a small set of servers.

https://github.com/opscode/opscoderl_httpc




On Mon, Sep 9, 2013 at 5:14 AM, Fred Hebert <mononcqc@REDACTED> wrote:

> You might be interested in my fork of lhttpc:
> https://github.com/ferd/lhttpc
>
> It's pretty much regular lhttpc, but I added sharded connection pools to
> it. It has been especially optimized for doing a lot of requests to a
> limited set of endpoints, and was developed during my stay at AdGear for
> real time bidding purposes. It's also used at other real time bidding
> shops now.
>
> It will not do request cancelling the way you mention it. It has a
> central load-balancer per endpoint, where you ask for sockets. It's easy
> to cancel a task once you have the connection yourself, but not during
> the time it's in the queue to the manager.
>
> If you kill the process that made the request, the manager will find out
> once it tries to give away the process and keep going until it hits a
> request that works
> (https://github.com/ferd/lhttpc/blob/master/src/lhttpc_lb.erl#L98)
>
> That might be suboptimal behavior in your case.
>
> I also have a fork of that very load balancer done using dispcount
> called dlhttpc: https://github.com/ferd/dlhttpc
>
> Dispcount is a low-latency pool (https://github.com/ferd/dispcount) that
> basically will tell you right away if you can have a resource or not.
> There will be no such thing as a blocking request using dlhttpc because
> the pooling model doesn't allow for an explicit request queue. It
> instead uses a pool of workers that may or may not be connected (in the
> case of dlhttpc, each worker will only connect once asked for it, but
> processes will be started already), and assumes you will retry until you
> hit one you need. It's been designed with cases of constant overload in
> mind, where you may arbitrarily decide to drop a request from the
> call-site.
>
> Depending on how flexible your requirements are, I'd try at least one of
> these libraries to see how they fare.
>
> lhttpc has been in production for a long while now and has been fairly
> stable as far as I can tell, everywhere it's been deployed. dlhttpc has
> seen no real production use, but dispcount has, so it's not too likely
> to be a terrible thing to deploy.
>
> Regards,
> Fred.
>
> On 09/08, Chris King wrote:
> > On Sun, 08 Sep 2013 19:33:05 -0400, Bob Ippolito <bob@REDACTED> wrote:
> >
> > >Async cancelable and synchronous are basically isomorphic in
> > >Erlang, just start a process to wrap the sync call and kill >it if
> > >you want to cancel.
> >
> > Not quite -- the HTTP request remains outstanding and blocks the
> > request queue.  I need to clear those requests out, as I need to
> > process new requests ASAP.  This should be handled by the HTTP
> > client itself, as only it knows how many requests it has actually
> > sent to the server.  (The alternative would be for the HTTP client
> > to expose a crediting mechanism; though this would complicate the
> > application.)
>
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>



-- 
Seth Falcon | @sfalcon | http://userprimary.net/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130913/ea66799c/attachment.htm>