[erlang-questions] Streaming Data using httpc

[Jesper Louis Andersen]

On Sun, Apr 19, 2015 at 6:14 PM John Duffy <jb_duffy@REDACTED> wrote:

>
>
> -module(streaming).
>
>
[...]

You are pretty close to the goal, but you are confusing the stream/receiver
options I think. In streaming, you will receive the data as a series of
chunks, which is what your code expect, but you don't supply an option
requesting streaming operation. So you don't retrieve an expected tuple.
You can add a catchall tuple to your receieve clause in receive_data/1 to
make sure you have the right format in your match. Also, you can add a
'after' clause to time out after a while. This can make debugging easier
since you "get back" to the the REPL.

The following works on a quick test in my end. Note how the receive clause
is different from yours, and that you get everything in one fell swoop,
rather than having to match on a multitude of clauses.

For more serious work, you might want to check out some of the numerous
other projects for HTTP client requests. I'm partial to Gun and Hackney
myself, but there are also ibrowse, lhttpc and fusco. They have slightly
different semantics and areas at which they excel, so choose wisely :)

For a prototype however, I think httpc is fine.

-module(streaming).

-export([data/0]).

data() ->
    {ok, RequestId} = httpc:request(get, {"example.com", []}, [], [{sync,
false}, {receiver, self()}]),
    receive_data(RequestId).

receive_data(RequestId) ->
    receive
        {http, {RequestId, {StatusLine, Headers, Body}}} ->
            error_logger:info_report(
            #{
              status => StatusLine,
              headers => length(Headers),
              body_size => byte_size(Body)
             }),
            ok
    after 5000 ->
            error_logger:info_report(timeout)
    end.

>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150419/047e7d84/attachment.htm>