[erlang-questions] Streaming Data using httpc

[John Duffy]

Hi Jesper
Thank you for your reply, very helpful.
I'm still a bit puzzled as to why my test doesn't work, I'm being to wonder if  I need to be sending additional headers, or something, to the server.
If I use 'curl' then everything works, I get a steady stream of data...
curl "http://stream-sandbox.oanda.com/v1/prices?accountId=99999&instruments=EUR_USD"
However, putting the same URL into your example results in a single time-out and the Erlang emulator stalling.
Kind regards
John
----Original message----
>From : jesper.louis.andersen@REDACTED
Date : 19/04/2015 - 22:00 (GMTDT)
To : jb_duffy@REDACTED, erlang-questions@REDACTED
Subject : Re: [erlang-questions] Streaming Data using httpc
On Sun, Apr 19, 2015 at 6:14 PM John Duffy <jb_duffy@REDACTED> wrote:
-module(streaming).
[...]
You are pretty close to the goal, but you are confusing the stream/receiver options I think. In streaming, you will receive the data as a series of chunks, which is what your code expect, but you don't supply an option requesting streaming operation. So you don't retrieve an expected tuple. You can add a catchall tuple to your receieve clause in receive_data/1 to make sure you have the right format in your match. Also, you can add a 'after' clause to time out after a while. This can make debugging easier since you "get back" to the the REPL.
The following works on a quick test in my end. Note how the receive clause is different from yours, and that you get everything in one fell swoop, rather than having to match on a multitude of clauses.
For more serious work, you might want to check out some of the numerous other projects for HTTP client requests. I'm partial to Gun and Hackney myself, but there are also ibrowse, lhttpc and fusco. They have slightly different semantics and areas at which they excel, so choose wisely :)
For a prototype however, I think httpc is fine.
-module(streaming).
-export([data/0]). 
data() ->
    {ok, RequestId} = httpc:request(get, {"example.com", []}, [], [{sync, false}, {receiver, self()}]),
    receive_data(RequestId).
receive_data(RequestId) ->
    receive
        {http, {RequestId, {StatusLine, Headers, Body}}} ->
            error_logger:info_report(
            	#{
            	  status => StatusLine,
            	  headers => length(Headers),
            	  body_size => byte_size(Body)
            	 }),
            ok
    after 5000 ->
            error_logger:info_report(timeout)
    end.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150420/907ed83e/attachment.htm>