http:set_options/1, http:request/4

[Michael McDaniel]

BRIEFLY:
http:request/4 used directly will properly retrieve URL.
http:request/4 through proxy fails.

DETAILS:

Linux fangora 2.6.4-52-default #1 Wed Apr 7 02:08:30 UTC 2004 i686 i686 i386 
GNU/Linux

using erl
Eshell V5.4.3  (abort with ^G)


	 +----------------+
	 | www.erlang.org |
	 +----------------+
	        |
	     INTERNET
		|
	+-------+--------+
	| www.autosys.us |
	+-------+--------+
		|
		|
	+-------+------------+
	| fangora.autosys.us |
	+--------------------+

hd.erl is running on fangora.autosys.us inside private network.
Squid proxy is running on www.autosys.us (aka cougora.autosys.us)
on port 3128.

-module(hd).
-export([hd/2]).

hd(Port, URL) ->
  application:start(inets) ,

  http:set_options([{proxy, { {"cougora.autosys.us", Port},
	    ["www.autosys.us", "autosys.us", "localhost"]} }]) ,

  http:request(get, 
  {URL,  [ {"Referer", URL}, {"Host", "fangora.autosys.us"} ]}, [], []).

%% end hd.erl


1> hd:hd(3128,"http://www.erlang.org").
{ok,{{"HTTP/1.0",400,"Bad Request"}, ...
2> hd:hd(3128,"http://www.autosys.us"). 
{ok,{{"HTTP/1.1",200,"OK"}, ...
3> 

The problem appears to be that the http:request/4 command in both of
the above invocations create the following get:

GET / HTTP/1.1

This is fine when not using proxy.  When using proxy, the proxy server
is expecting, for example, "GET http://www.erlang.org HTTP/1.1",
so the get request fails.  Follows is the headers trace...

GET / HTTP/1.1
Host: fangora.autosys.us
TE: 
referer: http://www.erlang.org

HTTP/1.0 400 Bad Request
Server: squid/3.0-PRE3
Mime-Version: 1.0
Date: Fri, 21 Jan 2005 05:55:23 GMT
Content-Type: text/html
Content-Length: 1436
Expires: Fri, 21 Jan 2005 05:55:23 GMT
X-Squid-Error: ERR_INVALID_URL 0
X-Cache: MISS from cougora.autosys.us
X-Cache-Lookup: NONE from cougora.autosys.us:3128
Via: 1.0 cougora.autosys.us (squid/3.0-PRE3)
Proxy-Connection: close

Follows is headers trace from a succesful browser GET

GET http://www.erlang.org/ HTTP/1.1
Host: www.erlang.org
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8a4) Gecko/20040927
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive

HTTP/1.0 200 OK
Date: Fri, 21 Jan 2005 06:28:51 GMT
Server: Apache/1.3.9 (Unix)
Last-Modified: Wed, 15 Dec 2004 14:24:04 GMT
ETag: "8d715-1efb-41c04904"
Accept-Ranges: bytes
Content-Length: 7931
Content-Type: text/html
X-Cache: MISS from cougora.autosys.us
X-Cache-Lookup: MISS from cougora.autosys.us:3128
Via: 1.0 cougora.autosys.us (squid/3.0-PRE3)
Proxy-Connection: keep-alive

At first I thought the problem was the TE: (transport encoding) header,
but it is not.  I did the tests with adding the other headers.  The
problem is the initial "GET".  The browser is configured to use the
squid proxy on port 3128.

I tried the following:
1> hd:hd(3128,"http://www.erlang.org/http://www.erlang.org").

and the GET is "GET /http://www.erlang.org HTTP/1.1" which is still
wrong.

Any suggestions are welcome.  I want to be able to use the commands
from my inside machine through proxy on Internet connected machine.

Michael McDaniel
Portland, Oregon, USA