[erlang-questions] Question about http client

[mtalyans]

Hi All,

I am a rank beginner, so please bear with me:

I am fetching Web content from a set of URLs using http:request. For most of
URLs everything works fine; however for some of them I see error reports on
the console that look like so:

<0.49.0> spawned pid <0.50.0> (here my main process spawns another one to do
the fetch)

=ERROR REPORT==== 15-Apr-2009::13:29:50 ===
** Generic server <0.51.0> terminating
** Last message in was {tcp_closed,#Port<0.1570>}
** When Server state == {state,
                            {request,#Ref<0.0.0.67>,<0.50.0>,0,http,
                                {"message.real.com",80},
                                "/Msg/7.0/win32",[],get,
                                {http_request_h,undefined,"keep-alive",
                                    undefined,undefined,undefined,undefined,
                                    undefined,undefined,undefined,undefined,
                                    undefined,undefined,undefined,undefined,
                                    undefined,undefined,"message.real.com",
                                    undefined,undefined,undefined,undefined,
                                    undefined,undefined,undefined,undefined,
                                    undefined,[],undefined,undefined,
                                    undefined,undefined,"0",undefined,
                                    undefined,undefined,undefined,undefined,
                                    undefined,[]},
                                {[],[]},
                                {http_options,"HTTP/1.1",5000,true,[],
                                    undefined,true},
                                "http://message.real.com/Msg/7.0/win32",[],
                                none,[]},
                            {tcp_session,
                                {{"message.real.com",80},<0.51.0>},
                                false,http,#Port<0.1570>,1,keep_alive},
                            undefined,undefined,undefined,
                            {httpc_response,parse,[nolimit,true]},
                            {[],[]},
                            {[],[]},
                            new,[],nolimit,nolimit,
                            {options,
                                {undefined,[]},
                                0,2,5,120000,2,disabled,enabled,false},
                            {timers,
                                [{#Ref<0.0.0.67>,#Ref<0.0.0.77>}],
                                undefined},
                            httpc_manager,undefined}
** Reason for termination ==
** session_remotly_closed
<0.50.0>: normal clause "ERLANG: Request error session_remotly_closed on
http://message.real.com/Msg/7.0/win32" (here my fetched process gets
results)

As far as I can tell, session has been remotely closed. I don¹t mind a
failure like this, but I would like to get rid of the error reports, as I am
fetching many, many URLs. I set verbose option to false, and I also catch
everything around the call to http:request (in fact, the last line in the
snippet is my own printout of the normal clause in the try/catch), all to no
avail. It seems that what is terminating is not the process I have spawned
to do the fetch.

In other cases, I get content properly fetched, and then get an error report
with a timeout:

<0.49.0> spawned pid <0.50.0>
<0.50.0>: normal clause "Normal completion"

=ERROR REPORT==== 15-Apr-2009::13:35:05 ===
** Generic server <0.51.0> terminating
** Last message in was {timeout,#Ref<0.0.0.67>}
** When Server state == {state,undefined,
                            {tcp_session,
                                {{"links.preplogic.com",80},<0.51.0>},
                                false,http,#Port<0.1570>,1,keep_alive},
                            undefined,undefined,undefined,undefined,
                            {[],[]},
                            {[],[]},
                            keep_alive,[],nolimit,nolimit,
                            {options,
                                {undefined,[]},
                                0,2,5,120000,2,disabled,enabled,false},
                            {timers,[],#Ref<0.0.0.78>},
                            httpc_manager,undefined}
** Reason for termination ==
** {{badrecord,request},
    [{httpc_handler,handle_info,2},
     {gen_server,handle_msg,5},
     {proc_lib,init_p_do_apply,3}]}

Could someone in the know help me to solve this issue?

Thanks in advance!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20090415/78321ba6/attachment.htm>