[erlang-questions] What lib to use for http requests

[Drew Varner]

Mark,

If you are communicating to servers via HTTPS, Hackney and other HTTP clients allow you to pass options to the underlying SSL/TLS socket, including some verification of the peer certificate. For example:

https://github.com/talko/httpcbench/blob/master/src/httpcbench_client.erl#L79-L86 <https://github.com/talko/httpcbench/blob/master/src/httpcbench_client.erl#L79-L86>

Gun has an open issue to address this (https://github.com/extend/gun/pull/27 <https://github.com/extend/gun/pull/27>), but it is not implemented. If you use Gun as your HTTPS client, you’re open to man-in-the-middle attacks.

Do not fall into a false sense of security that any Erlang HTTPS clients provide complete protection against man-in-the-middle attacks out of the box. You’ll also want to consider cases where your peer certificate was revoked by a Certificate Authority. You’d want to know how your HTTPS client handles certificates when their revocation data has been published via a CRL or OCSP. CRL verification has made some headway in Erlang (see the ssl module docs and https://github.com/Vagabond/erl_crl_example <http://erlang.org/doc/man/ssl.html>). You’ll also want to look at hostname verification (https://github.com/deadtrickster/ssl_verify_hostname.erl <https://github.com/benoitc/ssl_verify_hostname>). 

HTTPS is as secure as you make it.

Felix mentioned ESL’s lhttpc repo. I’d take a look at Talko’s: https://github.com/talko/lhttpc <https://github.com/talko/lhttpc> It’s the underlying HTTP client for erlcloud.

Hackney also has nice multipart/form functionality.

Cheers,
Drew

> On Dec 12, 2014, at 3:45 PM, Mark Nijhof <mark.nijhof@REDACTED> wrote:
> 
> I meant to mention Hackney. Why would you choose Hackney over Gun and vise verse?
> 
> On Fri, Dec 12, 2014 at 9:27 PM, Alex Shneyderman <a.shneyderman@REDACTED <mailto:a.shneyderman@REDACTED>> wrote:
> nobody mentioned hackney: well written, mature, filled with features. not to mention very active.
> 
> On Fri, Dec 12, 2014 at 3:17 PM, Mark Nijhof <mark.nijhof@REDACTED <mailto:mark.nijhof@REDACTED>> wrote:
> Hi Felix, Iñaki,
> 
> Thanks for your reply, you confirm what I was thinking myself about using Gun but I asked because of: 
> 
> > In general the state of http clients in erlang is a bewildering, overgrown thicket of ancient decaying masonry and beguiling dead ends filled with poisonous invisible gila monsters.
> 
> Cheers,
> 
> -Mark
> 
> 
> On Fri, Dec 12, 2014 at 9:03 PM, Felix Gallo <felixgallo@REDACTED <mailto:felixgallo@REDACTED>> wrote:
> httpc has some weird bugs under load and should be retired from the standard distribution.
> 
> lhttpc is better but is a dead project and has a variety of forks, some of which are buggy or incomplete.  The 'esl' fork seems to be the closest although in my experience it seems to have a broken pooling mechanism.  Additionally it uses 'let it crash' for the common case of timeouts, which can fill up crash.log quickly and impede investigation into real issues.
> 
> dlhttpc is a fork of lhttpc that ferd put together to handle high volume requests to a low number of endpoints.  It's also not actively maintained, but ferd is still alive and kicking and has recently responded to pull requests there.  I intended on using this but rustled up my own nasty pool mechanism on top of my own hacked fork of lhttpc.
> 
> gun appears to be the most actively maintained; I haven't tried it yet but if it's as solid as cowboy, this is probably the right one to use for new projects.
> 
> shotgun is just an SSE convenience wrapper on gun, so if you don't need SSE, sticking with gun is probably your best bet.
> 
> fusco is an alpha quality http client that doesn't appear to be actively maintained.
> 
> In general the state of http clients in erlang is a bewildering, overgrown thicket of ancient decaying masonry and beguiling dead ends filled with poisonous invisible gila monsters.
> 
> F.
> 
> 
> On Fri, Dec 12, 2014 at 11:44 AM, Iñaki Garay <igarai@REDACTED <mailto:igarai@REDACTED>> wrote:
> We love our tool shotgun[1], built on top of gun:
> 
> [1] https://github.com/inaka/shotgun <https://github.com/inaka/shotgun>
> 
> It improves on gun's SSE support. 
> 
> good luck, 
> Iñaki
> 
> On Fri, Dec 12, 2014 at 4:38 PM, Mark Nijhof <mark.nijhof@REDACTED <mailto:mark.nijhof@REDACTED>> wrote:
> Hi,
> 
> I am looking for the atm best solution to make http requests, from downloading a small file till larger (100mb) archives. I know of f.ex. the httpc, ibrowse, gun.
> 
> Any preferences? And why?
> 
> -Mark
> 
> -- 
> Mark Nijhof
> t:   @MarkNijhof <https://twitter.com/MarkNijhof>
> s:  marknijhof
> 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> http://erlang.org/mailman/listinfo/erlang-questions <http://erlang.org/mailman/listinfo/erlang-questions>
> 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> http://erlang.org/mailman/listinfo/erlang-questions <http://erlang.org/mailman/listinfo/erlang-questions>
> 
> 
> 
> 
> -- 
> Mark Nijhof
> t:   @MarkNijhof <https://twitter.com/MarkNijhof>
> s:  marknijhof
> 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> http://erlang.org/mailman/listinfo/erlang-questions <http://erlang.org/mailman/listinfo/erlang-questions>
> 
> 
> 
> 
> 
> -- 
> Mark Nijhof
> t:   @MarkNijhof <https://twitter.com/MarkNijhof>
> s:  marknijhof
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141212/64ac33cb/attachment.htm>