[erlang-questions] Reading a big text file from a web server.

[Chandru]

On 28/04/07, rtgreg rtgreg <rtgreg@REDACTED> wrote:
> Hello.
>
> I'm starting to study erlang from the "Concurrent Programming in
> Erlang" book available in the website.
>
> To test my knowledge I've decided to port a small Python application
> of mine to Erlang but I'm having problems with memory consumption and
> performance:
>
> One of the modules has to read a file from a web server in the following format:
>
> Key1 The value for key 1
> Key2 The value for key 2
> Key3
> Begin Key3
> The value for key 3 is
> a lot longer and we
> need several lines
> End Key3
>
> Right now my, very naive, implementation is reading the result of
> http:request("http://example.com/file.txt") and split it by new lines.
> The file is several megabytes long so the memory consumption increases
> considerably.
>
> What do you recommend to implement this functionality?

Check to see if the inets http client has an option to save the
response to a file. If it doesn't, use ibrowse which has this feature.
The command to do it using ibrowse is:

ibrowse:send_req("http://example.com/file.txt", [], get, [], [
{save_response_to_file, true}], 60000).

Once the file is downloaded, you can then parse it in chunks.

cheers
Chandru