[erlang-questions] Reading a big text file from a web

[Ingela Anderton Andin]

erlang-questions-request@REDACTED wrote:
[...]
>> Right now my, very naive, implementation is reading the result of
>> http:request("http://example.com/file.txt") and split it by new lines.
>> The file is several megabytes long so the memory consumption increases
>> considerably.
>>
>> What do you recommend to implement this functionality?
>>     
>
> Check to see if the inets http client has an option to save the
> response to a file. 
It does have an option to save  it to a file.

http:request(get, {URL, []}, [], [{stream, FilePath}]).
> Once the file is downloaded, you can then parse it in chunks.
>   
Better up the inets http-client lets you stream the result to a process 
directly so you do not have to wait for the
whole response to be saved to a file to then read it in chunks.

 {ok, RequestId} =
    http:request(get, {URL, []}, [], [{sync, false}, {stream, self}]).

Then your process can receive the messages

{http, {RequestId, stream_start, Headers},
{http, {RequestId, stream, BinBodyPart},
{http, {RequestId, stream_end, Headers}.


Regards Ingela - OTP team