[erlang-questions] Streaming Data using httpc

[nx]

Not sure what you're trying to do, but if you're actually processing
the body of the message in a stream you may want to use the stream
option and make your module a process. Otherwise you can stream the
data to a file (i.e. downloading files) using the {stream, filename()}
option.

If you're wanting the former, and you want to do something with each
part of the stream and ask for the next block in the stream, you can
do something like this. This assumes you're module is a gen_server and
you're using the {stream, once} option to step through the stream
response:

handle_call({async_request, Url}, _From, State) ->
  httpc:request(get, {Url, _Headers=[]}, _HttpOptions=[], [{stream,
{self, once}}, {sync, false}]),
  {reply, ok, State}.

%% Start stream, store httpc Pid, continue
handle_info({http, {_RequestId, stream_start, _Headers, Pid}}, State}) ->
  ok = httpc:stream_next(Pid),
  {noreply, State#state{pid=Pid}};
%% Handle a chunk, continue
handle_info({http, {_RequestId, stream, Part}}, #state{pid=Pid}=State}) ->
  handle_part(Part),
  ok = httpc:stream_next(Pid),
  {noreply, State};
%% Handle a 404 error, cancel the request
handle_info({http, {RequestId, {{_HTTPVersion, 404, "Not Found"}, _Headers,
_Body}}}, State) ->
  ok = httpc:cancel_request(RequestId),
  {stop, normal, State};
%% Handle stream error, cancel the request
handle_info({http, {RequestId, {error, Reason}}}, State) ->
  ok = httpc:cancel_request(RequestId),
  {stop, normal, State};
%% Stream end
handle_info({http, {_RequestId, stream_end, _Headers}}, State) ->
  do_something(),
  {stop, normal, State}.

The standard httpc library is flexible enough if you want a precise
level of control over the lifetime of the HTTP stream.


On Sun, Apr 19, 2015 at 2:06 PM, Paul Peregud <paulperegud@REDACTED> wrote:
> I don't have experience using httpc, so one remark only:
>
> receive_data(RequestId, State) ->
>     receive
>         {http, {RequestId, stream_start, Headers}} -> do something...,
> receive_data(RequestId, State);
>         {http, {RequestId, stream, Data}}               - > do
> something..., receive_data(RequestId, State);
>         {http, {RequestId, stream_end, Headers}}   -> do something...
>     end.
>
> Because you want to exit receive_data when no more data is going your way.
>
>
> On Sun, Apr 19, 2015 at 5:30 PM, John Duffy <jb_duffy@REDACTED> wrote:
>> Hi
>>
>> I'm new to Erlang so please forgive my ignorance. I'm trying to stream data
>> from a REST API using httpc, and although I have scoured the internet and
>> the documentation I can't find a good example of how to do this, in
>> particular how the "sync" and "receiver" options interoperate. My
>> unsuccessful module looks like this...
>>
>> -module(streaming).
>>
>> -export([data/0]).
>>
>> data() ->
>>     {ok, RequestId} = httpc:request(get,
>> {"http://my_streaming_data_source.com", []}, [], [{sync, false}, {receiver,
>> self()]),
>>     receive_data(RequestId).
>>
>> receive_data(RequestId) ->
>>     receive
>>         {http, {RequestId, stream_start, Headers}} -> do something...;
>>         {http, {RequestId, stream, Data}}               - > do something...;
>>         {http, {RequestId, stream_end, Headers}}   -> do something...;
>>     end,
>>     receive_data(RequestId).
>>
>>
>> Is the above how I should be structuring my module?
>>
>> Kind regards
>>
>> John Duffy
>>
>>
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
>
>
> --
> Best regards,
> Paul Peregud
> +48602112091
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions