[erlang-questions] Streaming Data using httpc

[Antoine Koener]

> Le 20 avr. 2015 à 13:05, John Duffy <jb_duffy@REDACTED> a écrit :
> 
> Hi Jesper
> 
> Thank you for your reply, very helpful.
> 
> I'm still a bit puzzled as to why my test doesn't work, I'm being to wonder if  I need to be sending additional headers, or something, to the server.
> 
> If I use 'curl' then everything works, I get a steady stream of data...
> 
> curl "http://stream-sandbox.oanda.com/v1/prices?accountId=99999&instruments=EUR_USD"

The problem seems to be the server because if you use -v for curl you will observe something like this:

< HTTP/1.1 200 Ok
* Server openresty/1.7.0.1 is not blacklisted
< Server: openresty/1.7.0.1
< Date: Tue, 21 Apr 2015 15:15:16 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: close
< Access-Control-Allow-Origin: *
<
{"tick":{"instrument":"EUR_CHF","time":"2015-04-21T15:13:48.585046Z","bid":1.2041,"ask":1.20435}}
{"heartbeat":{"time":"2015-04-21T15:15:16.632703Z"}}
{"heartbeat":{"time":"2015-04-21T15:15:21.632772Z"}}
{"heartbeat":{"time":"2015-04-21T15:15:24.598956Z"}}

As you can see there's some strange headers:
Connection: close  
The connection is not closed because it's a stream.

Transfer-Encoding: chunked
What I see is absolutely not chunked transfer, it's a bunch of json lines...
Chunks should be preceded by the size (hex encoded) and \r\n

So I think that the erlang code is trying to respect headers, close the connection and search for chunk encoding, but there's none...

It might be interesting to report those problems to the developers of this service :)




> 
> However, putting the same URL into your example results in a single time-out and the Erlang emulator stalling.
> 
> Kind regards
> 
> John
> 
> ----Original message----
> From : jesper.louis.andersen@REDACTED
> Date : 19/04/2015 - 22:00 (GMTDT)
> To : jb_duffy@REDACTED, erlang-questions@REDACTED
> Subject : Re: [erlang-questions] Streaming Data using httpc
> 
> 
> 
> On Sun, Apr 19, 2015 at 6:14 PM John Duffy <jb_duffy@REDACTED <mailto:jb_duffy@REDACTED>> wrote:
> 
> 
> -module(streaming).
> 
> 
> [...]
> 
> You are pretty close to the goal, but you are confusing the stream/receiver options I think. In streaming, you will receive the data as a series of chunks, which is what your code expect, but you don't supply an option requesting streaming operation. So you don't retrieve an expected tuple. You can add a catchall tuple to your receieve clause in receive_data/1 to make sure you have the right format in your match. Also, you can add a 'after' clause to time out after a while. This can make debugging easier since you "get back" to the the REPL.
> 
> The following works on a quick test in my end. Note how the receive clause is different from yours, and that you get everything in one fell swoop, rather than having to match on a multitude of clauses.
> 
> For more serious work, you might want to check out some of the numerous other projects for HTTP client requests. I'm partial to Gun and Hackney myself, but there are also ibrowse, lhttpc and fusco. They have slightly different semantics and areas at which they excel, so choose wisely :)
> 
> For a prototype however, I think httpc is fine.
> 
> -module(streaming).
> 
> -export([data/0]). 
> 
> data() ->
>     {ok, RequestId} = httpc:request(get, {"example.com <http://example.com/>", []}, [], [{sync, false}, {receiver, self()}]),
>     receive_data(RequestId).
> 
> receive_data(RequestId) ->
>     receive
>         {http, {RequestId, {StatusLine, Headers, Body}}} ->
>             error_logger:info_report(
>             	#{
>             	  status => StatusLine,
>             	  headers => length(Headers),
>             	  body_size => byte_size(Body)
>             	 }),
>             ok
>     after 5000 ->
>             error_logger:info_report(timeout)
>     end.
> 
> 
> 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150421/e712e8d9/attachment.htm>