[erlang-questions] gen_tcp very slow to fetch data

zabrane Mikael <>
Sat Nov 21 01:02:04 CET 2009


Hi List !

While trying to learn how to write a simple TCP Web Server in Erlang which
only dump what it gets to stdout, I realize that time to time, the HTTP
requests get truncated when reaching the server. My main socket loop looks
like:

-------------------------------------------
-define(TCP_OPTIONS,[binary, {packet, 0}, {active, false}, {reuseaddr,
true}]).
...
loop_recv(Socket)
   case gen_tcp:recv(Socket, 0) of
        {ok, BinData} ->
             %% here, I'm assuming that all the HTTP request (Headers +
Body) is in "BinData". Hope I'm right.
             io:format("BinData: ~p~n", []),
             ok;
        NotOK ->
            error_logger:info_report([{"gen_tcp:recv/2", NotOK}]),
            error
    end.
-------------------------------------------

For some requests, the "io:format" prints a truncated data (in BinData):

<<"
http://www.foo.tv/images/v30/LoaderV3.swf?loop=false&quality=high&request=357&HTTP/1.0\r\nHost:
www.foo.tv\r\nUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5;
fr; rv:1.9.0.2) Gecko/2008090512 Firefox/3.0.2\r\nAccept: text/h">>

As you can see, the request isn't complete "... Accept: text/h".

Am I doing somthing wrong? How can I fix it please?

Regards
Zabrane

2009/11/17 Tony Rogvall <>

> Do not forget about {active, once} mode.
> {active,once} will receive one message (depends on buffer size etc)
> the it will switch to passive mode. To get the next message you use
> inet:setopts(Socket, [{active,once}]) to activate it again. This mode
> enables
> a selective receive at the same time as it enables flow control.
>
> /Tony
>
>
> On 17 nov 2009, at 01.51, Ngoc Dao wrote:
>
> >> From inet's doc:
> >
> http://www1.erlang.org/documentation/doc-4.9.1/lib/kernel-2.4.1/doc/html/inet.html
> >
> > If the active option is true, which is the default, everything
> > received from the socket will be sent as messages to the receiving
> > process. If the active option is set to false (passive mode), the
> > process must explicitly receive incoming data by calling
> > gen_tcp:recv/N or gen_udp:recv/N  (depending on the type of socket).
> > Note: Passive mode provides flow control; the other side will not be
> > able send faster than the receiver can read. Active mode provides no
> > flow control; a fast sender could easily overflow the receiver with
> > incoming messages. Use active mode only if your high-level protocol
> > provides its own flow control (for instance, acknowledging received
> > messages) or the amount of data exchanged is small.
> >
> >
> > On Tue, Nov 17, 2009 at 2:59 AM, ERLANG <> wrote:
> >> Hi Chandru !
> >>
> >> That's fix my problem. Thanks.
> >> While googling a bit, I found two ways to read from the Socket:
> >>
> >> recv(Socket, Bin) ->
> >>    receive
> >>        {tcp, Socket, B} ->
> >>            io:format(".", []),
> >>            recv(Socket, concat_binary([Bin, B]));
> >>        {tcp_closed, Socket} ->
> >>            {ok, Bin};
> >>        Other ->
> >>            {error, {socket, Other}}
> >>        after
> >>            ?TIMEOUT ->
> >>            {error, {socket, timeout}}
> >>    end.
> >>
> >> % version 2 with "gen_tcp:recv"
> >> recv2(Socket, Bin) ->
> >>    case gen_tcp:recv(Socket, 0, ?TIMEOUT) of
> >>         {ok, B} ->
> >>             io:format(".", []),
> >>             recv(Socket, concat_binary([Bin, B]));
> >>         {error, closed} ->
> >>             {ok, Bin};
> >>        {error, timeout} ->
> >>             {error, {socket, timeout}};
> >>         Other ->
> >>             {error, {socket, Other}}
> >>     end.
> >>
> >>
> >> Which one is the best in my case (see below: fetch.erl)?
> >>
> >> Regards
> >> Zabrane
> >>
> >> Le 16 nov. 09 à 18:53, Chandru a écrit :
> >>
> >>> You are expecting the server to indicate end of response by closing the
> >>> connection, but because you specify HTTP/1.1 in the request, the server
> is
> >>> holding up your connection, and you are timing out. Try replacing
> HTTP/1.1
> >>> with HTTP/1.0 in your request, or parse the response to detect end of
> >>> response.
> >>>
> >>> cheers
> >>> Chandru
> >>>
> >>> 2009/11/16 zabrane Mikael <>
> >>>
> >>>> Hi List !
> >>>>
> >>>> New to Erlang, I'm trying to implement a simple URL fetcher.
> >>>> Here's my code (please, feel free to correct it if you find any bug or
> >>>> know
> >>>> a better approach):
> >>>>
> >>>>
> >>>>
> >>>>
> 8-----8-----8-----8-----8-----8-----8-----8-----8-----8-----8-----8-----8----
> >>>> -module(fetch).
> >>>>
> >>>> -export([url/1]).
> >>>>
> >>>> -define(TIMEOUT,    7000).
> >>>> -define(TCP_OPTS,   [binary, {packet, raw}, {nodelay, true},
> >>>>                   {active, true}]).
> >>>>
> >>>> url(Url) ->
> >>>>  {ok, _Tag, Host, Port} = split_url(Url),
> >>>>
> >>>>  Hdrs = [],
> >>>>  Request = ["GET ", Url, " HTTP/1.1\r\n", Hdrs, "\r\n\r\n"],
> >>>>
> >>>>  case catch gen_tcp:connect(Host, Port, ?TCP_OPTS) of
> >>>>    {'EXIT', Why} ->
> >>>>          {error, {socket_exit, Why}};
> >>>>      {error, Why} ->
> >>>>          {error, {socket_error, Why}};
> >>>>      {ok, Socket} ->
> >>>>          gen_tcp:send(Socket, list_to_binary(Request)),
> >>>>          recv(Socket, list_to_binary([]))
> >>>>  end.
> >>>>
> >>>> recv(Socket, Bin) ->
> >>>>  receive
> >>>>      {tcp, Socket, B} ->
> >>>>          io:format(".", []),
> >>>>          recv(Socket, concat_binary([Bin, B]));
> >>>>      {tcp_closed, Socket} ->
> >>>>          {ok, Bin};
> >>>>      Other ->
> >>>>          {error, {socket, Other}}
> >>>> after
> >>>>  ?TIMEOUT ->
> >>>>          {error, {socket, timeout}}
> >>>>  end.
> >>>>
> >>>>
> >>>> split_url([$h,$t,$t,$p,$:,$/,$/|T]) ->  split_url(http, T);
> >>>> split_url(_X)                       ->  {error, split_url}.
> >>>>
> >>>> split_url(Tag, X) ->
> >>>>  case string:chr(X, $:) of
> >>>>      0 ->
> >>>>          Port = 80,
> >>>>          case string:chr(X,$/) of
> >>>>              0 ->
> >>>>                  {ok, Tag, X, Port};
> >>>>              N ->
> >>>>                  Site = string:substr(X,1,N-1),
> >>>>                  {ok, Tag, Site, Port}
> >>>>          end;
> >>>>      N1 ->
> >>>>          case string:chr(X,$/) of
> >>>>              0 ->
> >>>>                  error;
> >>>>              N2 ->
> >>>>                  PortStr = string:substr(X,N1+1, N2-N1-1),
> >>>>                  case catch list_to_integer(PortStr) of
> >>>>                      {'EXIT', _} ->
> >>>>                          {error, port_number};
> >>>>                      Port ->
> >>>>                          Site = string:substr(X,1,N1-1),
> >>>>                          {ok, Tag, Site, Port}
> >>>>                  end
> >>>>          end
> >>>>  end.
> >>>>
> >>>>
> >>>>
> >>>>
> 8-----8-----8-----8-----8-----8-----8-----8-----8-----8-----8-----8-----8------
> >>>>
> >>>> When testing it, the receiving socket gets very very slow:
> >>>> $ erl
> >>>> 1> c(fetch).
> >>>> 2> Bin = fetch:url("http://www.google.com").
> >>>> ......{error,{socket,timeout}}
> >>>>
> >>>> Am I missing something?
> >>>> What I like to get at the end is a very fast fetcher. Any hint?
> >>>>
> >>>> Regards
> >>>> Zabrane
> >>>>
> >>
> >>
> >> ________________________________________________________________
> >> erlang-questions mailing list. See http://www.erlang.org/faq.html
> >> erlang-questions (at) erlang.org
> >>
> >>
> >
> > ________________________________________________________________
> > erlang-questions mailing list. See http://www.erlang.org/faq.html
> > erlang-questions (at) erlang.org
> >
>
>


More information about the erlang-questions mailing list