[erlang-questions] What lib to use for http requests

[Loïc Hoguin]

On 12/13/2014 10:27 PM, Tuncer Ayaz wrote:
> On Fri, Dec 12, 2014 at 9:03 PM, Felix Gallo wrote:
>> httpc has some weird bugs under load and should be retired from the
>> standard distribution.
>
[...]
> I haven't used httpc enough to run into the mentioned issues, but I do
> wonder why instead of fixing httpc we, as a community, write more than
> a few alternative http clients. Is it because it's easy to do like a
> httpd?

The "httpc gets confused under load" bug has popped up for at least two 
years. I am not sure as to why nobody sent a patch for it (other than it 
being quite hard to replicate outside production), but I can tell you 
why some alternatives were written: to get different properties. Hackney 
would be what you use for process-less connections. Gun what you use for 
message-based always-connected connections. Etc.

This is the main reason why alternatives exist. Existing project A has 
different properties than you need; you can't add those properties to 
project A without breaking all existing code (or sometimes, at all); you 
either fork A into project A' or create project B.

In my projects for example, the need for specific properties is why 
Cowboy and erlang.mk exist. I moved from rebar to erlang.mk because of 
two issues that can never be fixed in rebar: "hot" compilation speed, 
and self-contained dependencies (aka don't use top-level rebar to 
compile everything, because some dependencies expect a particular 
version of rebar).

And in projects I use, Gproc would be a great example of replicating 
functionality found in OTP but with different, and imo much better, 
properties.

Sometimes people also write alternatives as a way of learning, too, but 
I do not know many examples like this.

> I think it's very important that the network code in otp.git is fixed
> and improved where necessary to leave external httpd or httpc libs to
> special purpose cases. Other languages don't seem to suffer from this
> and carry http[c,d] libs which are well-maintained and the go to
> implementation for common use cases.

As for httpd being the odd one out, I would guess the main reason would 
be that the documentation is very confusing as to how to make it run 
your Erlang code, and what you can do and how. It's well documented for 
the file serving part, but seem to fall apart after that. The Apache 
oriented design will also leave many people scratching their head I'm sure.

> Also see:
> http://erlang.org/pipermail/erlang-questions/2014-February/077766.html
>
>
> With all that said, I'm confident the OTP team would more than welcome
> improvements or even replacements for inets components.

Perhaps it is fine as it is (though that bug being fixed eventually 
would still be nice). OTP libs don't have to try to be the ultimate 
"everything you'll ever need", just provide the bare minimum for a 
useful system. (I would say they provide way too many libs right now, 
but http[cd] are fine.)

-- 
Loïc Hoguin
http://ninenines.eu