www_tools http client

[Shawn Pearce]

This couldn't have been more timely for me.  I was opening my email
to write a gen_tcp question, as I was trying to port this to use
gen_tcp rather than socket.  Thanks Hal.

My problem is now that I'm trying to make this thing use non-active
sockets, so I can passively poll for TCP data rather than get it via
messages.

My intended application is in a load testing tool where I may be
getting back a few hundred KB of data per process, and having a few
hundred processes running.  I'm wondering if I can get away with active
sockets, or have to use passive.  The process won't do anything once
the HTTP request is sent off other than receive data.

Right now I'm getting from gen_tcp:recv {error,enotsock} rather than
{error, closed} as the docs state, when I connect the socket with
{active, false}.  Here's what I changed in Hal's url.erl:

114c114
<     case catch gen_tcp:connect(IP, Port, [binary, {packet, 0}]) of
---
>     case catch gen_tcp:connect(IP, Port, [binary, {packet, 0}, {active, false} ]) of
129,130c129,130
<     receive
<       {tcp,Socket,B} ->
---
>     case gen_tcp:recv(Socket, 0) of
>       {ok,B} ->
139,143c139,141
<           {error, {tcp_error, Reason}}
<     after
<       Timeout ->
<           gen_tcp:close(Socket),
<           {error, timeout}
---
>           {error, {tcp_error, Reason}};
>       Other ->
>               io:format("What the? ~p~n", [Other])

Its printing:

	42> url:raw_get_url("http://xenon:80/",6000).
	Socket = #Port<0.119>
	What the? {error,enotsock}
	ok
	43>

Huh?  Why is gen_tcp:recv/2 giving me {error,enotsock} rather than
{error, closed} as the docs state?  I get {ok, B} for the data packet,
but that's it.

This is R9B-0.

Hal Snyder <hal@REDACTED> wrote:
> Sean's recent postings on http clients reminded me, there is a nice,
> small http client in Joe Armstrong's contrib,
> 
>   http://www.erlang.org/user.html#www_tools-1.0
> 
> but it doesn't compile with recent releases of OTP. With apologies to
> Joe, I've posted an updated version of url.erl at
> 
>   ftp://ftp.enteract.com/users/hal/url.erl
> 
> The new version makes the jump from socket to gen_tcp, so it compiles
> with R9. Also added are raw_post_url/3 and /4 entries.
> 
> There is a dependency on the url_parse.erl module from Joe's tarball,
> but that compiles unmodified.
> 
> Comments/brickbats welcome. Caveat utilitor.

-- 
Shawn.