[erlang-questions] Reading a big text file from a web server.

[Julian Fondren]

On 4/28/07, rtgreg rtgreg <rtgreg@REDACTED> wrote:
> Right now my, very naive, implementation is reading the result of
> http:request("http://example.com/file.txt") and split it by new lines.
> The file is several megabytes long so the memory consumption increases
> considerably.
>
> What do you recommend to implement this functionality?

I recommend you deal with a binary (and give a user agent),

   http:request(get, {"http://example.com/file.txt", [{"User-Agent", "..."}]},
                [], [{body_format, binary}]).

and that you skip the 'split by lines' step, instead generating
your key-value structure directly from the binary.

Aside, I've been playing with this, for good user agents:

agent() ->
    L = [version(?MODULE), version(inets), os:type(), os:version()],
    lists:flatten(io_lib:fwrite("PlanetErlang/~s Erlang/~s ~p/~p", L)).
version(inets) -> key(app_vsn, inets:module_info(attributes));
version(M)     -> key(vsn,         M:module_info(attributes)).

key(K, L) -> case lists:keysearch(K, 1, L) of {value, {_, V}} -> V end.

Cheers,
Julian